A training method, recognition method and device for speech emotion model

Abstract

The embodiment of the present application provides a training method, recognition method and device for a speech emotion model, wherein the speech emotion model includes an instance-level network framework and a cluster-level network framework. The training method obtains an audio data set of original speech; performs dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set; inputs the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation; inputs the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result; and updates the parameters of the initialized speech emotion model based on the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model. This training method can effectively reduce the computing power resources required for predictive recognition of the speech emotion model and improve the efficiency of speech emotion recognition. The present application relates to the field of speech recognition technology.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a training method, recognition method and device for a speech emotion model. Background Art

[0002] Speech Emotion Recognition (SER) refers to the technology of identifying the speaker's emotional state through the acoustic features of a speech. It can help intelligent customer service better understand the user's voice content and emotional changes and has attracted much attention.

[0003] Currently, speech emotion recognition primarily relies on extracting distinct acoustic and spectral features of speech, then using deep learning networks to analyze and learn the emotional information contained in these features. This approach targets instance-level speech data and requires extracting and learning speech data at each instance. However, due to the large volume of speech data and the high computing power required, speech emotion recognition is inefficient.

[0004] Therefore, the problems existing in the existing technology still need to be solved and optimized. Summary of the Invention

[0005] The object of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] The main purpose of the embodiments of the present application is to propose a training method, recognition method and device for a speech emotion model, wherein the training method can effectively reduce the computing power resources required for predictive recognition of the speech emotion model and improve the efficiency of speech emotion recognition.

[0007] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for training a speech emotion model, wherein the speech emotion model includes an instance-level network framework and a cluster-level network framework, wherein the instance-level network framework and the cluster-level network framework are connected in parallel, and the training method includes:

[0008] Get the audio dataset of the original speech;

[0009] Performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0010] Inputting the dual-branch feature dataset into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used to record multiple audio features of the original speech at the instance level;

[0011] Inputting the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0012] According to the acoustic feature representation and the speech emotion classification result, the parameters of the initialized speech emotion model are updated to obtain a trained speech emotion model.

[0013] In some embodiments, the dual-branch feature dataset includes a first branch feature dataset and a second branch feature dataset, and performing dual-branch feature extraction on the audio dataset to obtain the dual-branch feature dataset includes:

[0014] Based on a preset first enhancement rule, performing a first data enhancement on the audio dataset to obtain a first branch enhanced dataset;

[0015] Based on a preset second enhancement rule, performing a second data enhancement on the audio dataset to obtain a second branch enhanced dataset;

[0016] Performing a first feature extraction on the first branch enhanced data set to obtain the first branch feature data set, and performing a second feature extraction on the second branch enhanced data set to obtain the second branch feature data set;

[0017] The first enhancement rule is different from the second enhancement rule.

[0018] In some embodiments, the first enhancement rule is obtained by the following steps:

[0019] Obtaining a preset data enhancement set, wherein the data enhancement set includes a plurality of different enhancement rules;

[0020] Random number screening is performed on the data enhancement set to obtain the first enhancement rule.

[0021] In some embodiments, the acoustic feature representation includes a first audio feature matrix and a second audio feature matrix, and inputting the dual-branch feature dataset into the instance-level network framework to obtain the acoustic feature representation includes:

[0022] Inputting the first branch feature data set into the instance-level network framework to obtain the first audio feature matrix;

[0023] Inputting the second branch feature data set into the instance-level network framework to obtain the second audio feature matrix;

[0024] The first audio feature matrix includes multiple first audio feature vectors, and the second audio feature matrix includes multiple second audio feature vectors; the number of rows to which each first audio feature vector belongs is the same as the number of rows to which the corresponding second audio feature vector belongs.

[0025] In some embodiments, the speech emotion classification result includes a first emotion category matrix and a second emotion category matrix, and inputting the dual-branch feature dataset into the cluster-level network framework to obtain the speech emotion classification result includes:

[0026] Inputting the first branch feature data set into the cluster-level network framework to obtain the first emotion category matrix;

[0027] Inputting the second branch feature data set into the cluster-level network framework to obtain the second emotion category matrix;

[0028] Among them, the first emotion category matrix includes multiple first emotion category feature vectors, and the second emotion category matrix includes multiple second emotion category feature vectors; the first emotion category feature vectors with the same number of columns correspond to the same emotion prediction category; the second emotion category feature vectors with the same number of columns correspond to the same emotion prediction category.

[0029] In some embodiments, updating the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model includes:

[0030] Performing row-wise screening on the acoustic feature representation to obtain a first audio row set having a positive example relationship and a second audio row set having a non-positive example relationship;

[0031] Performing column screening on the speech emotion classification results to obtain a first category column set having a positive example relationship and a second category column set having a non-positive example relationship;

[0032] Calculating an instance-level loss value by maximizing the cosine similarity of the first audio row set and minimizing the cosine similarity of the second audio row set;

[0033] The cluster-level loss value is calculated by maximizing the cosine similarity of the first category column set and minimizing the cosine similarity of the second category column set;

[0034] Based on the instance-level loss value and the cluster-level loss value, the parameters of the initialized speech emotion model are updated to obtain the trained speech emotion model.

[0035] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application proposes a method for recognizing a speech emotion model, the method comprising the following steps:

[0036] Obtain the speech data set to be recognized;

[0037] The speech data set is input into the above-trained speech emotion model for emotion recognition to obtain a speech emotion recognition result.

[0038] To achieve the above objectives, a third aspect of an embodiment of the present application provides a training device for a speech emotion model, wherein the speech emotion model includes an instance-level network framework and a cluster-level network framework, wherein the instance-level network framework and the cluster-level network framework are connected in parallel, and the training device includes:

[0039] An acquisition module is used to obtain an audio data set of original speech;

[0040] An extraction module, configured to perform dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0041] A first processing module is configured to input the dual-branch feature dataset into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is configured to record a plurality of audio features of the original speech at the instance level;

[0042] A second processing module is used to input the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0043] The updating module is used to update the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

[0044] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0045] To achieve the above-mentioned purpose, the fifth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned method when executed by a processor.

[0046] The present application proposes a training method, recognition method and device for a speech emotion model, wherein the training method obtains an audio dataset of original speech; performs dual-branch feature extraction on the audio dataset to obtain a dual-branch feature dataset; inputs the dual-branch feature dataset into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used to record multiple audio features of the original speech at the instance level; inputs the dual-branch feature dataset into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level; and updates the parameters of the initialized speech emotion model based on the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model. The training method inputs the dual-branch feature dataset into the instance-level network framework and the cluster-level network framework for training respectively, and can learn the audio data at two granularities, namely, the instance level and the cluster level. The speech emotion model is jointly trained by the acoustic feature representation and speech emotion classification results obtained through learning, which can reduce the computing resources required for the speech emotion model to perform speech emotion prediction and recognition, thereby improving the efficiency of speech emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 This is a schematic diagram of the process principle of a method for training a speech emotion model provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Schematic diagram of the process principle of step S102 in FIG.

[0050] Figure 3 This is a schematic diagram of the process principle of the first enhancement rule provided in an embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of the process principle of a method for recognizing a speech emotion model provided in an embodiment of the present application;

[0052] Figure 5 Schematic diagram of a training device for a speech emotion model provided in an embodiment of the present application;

[0053] Figure 6This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0057] Currently, speech emotion recognition primarily relies on extracting distinct acoustic and spectral features of speech, then using deep learning networks to analyze and learn the emotional information contained in these features. This approach targets instance-level speech data and requires extracting and learning speech data at each instance. However, due to the large volume of speech data and the high computing power required, speech emotion recognition is inefficient.

[0058] In view of this, an embodiment of the present application provides a training method, a recognition method and a device for a speech emotion model, wherein the training method inputs a dual-branch feature data set into an instance-level network framework and a cluster-level network framework for training respectively, and can perform feature learning at two granularities, instance level and cluster level, on the audio data. The acoustic feature representation and speech emotion classification results obtained through learning are used to jointly train the speech emotion model, which can reduce the computing resources required for the speech emotion model to perform speech emotion prediction and recognition, and improve the efficiency of speech emotion recognition.

[0059] The present invention provides a method for training, a method for recognizing, and a device for a speech emotion model, which are specifically described in the following embodiments.

[0060] The method provided in the embodiment of the present application can be applied to digital medical application scenarios or digital financial application scenarios. In digital medical application scenarios or digital financial application scenarios, the speech emotion model provided in the embodiment of the present application can be loaded into the digital medical intelligent customer service or digital financial intelligent customer service. The speech emotion model requires low computing resources for speech emotion prediction and recognition, and has high efficiency in speech emotion recognition. The speech emotion model helps the intelligent customer service identify the user's current emotional state, so that the intelligent customer service can respond more reasonably and quickly, thereby improving user satisfaction with the intelligent customer service.

[0061] The training method or recognition method of the speech emotion model provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the above method, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0063] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0064] Figure 1 This is an optional flow chart of the method for training a speech emotion model provided in an embodiment of the present application, wherein the speech emotion model includes an instance-level network framework and a cluster-level network framework, wherein the instance-level network framework and the cluster-level network framework are connected in parallel. Figure 1 The training method may include but is not limited to steps S101 to S105.

[0065] Step S101: Obtain an audio data set of original speech;

[0066] In an embodiment of the present application, the audio data set of the original speech can be the audio data of several voice speakers over a certain length of time, which can be collected based on an audio sensor, or a public audio data set can be used. The specific public audio data set can be any one of the I EMOCAP audio data set, the RAVDESS audio data set, etc., wherein the audio emotion categories in the I EMOCAP audio data set include anger, happiness, sadness, neutrality, frustration, excitement, fear, disgust, etc., and the audio emotion categories in the RAVDESS audio data set include neutrality, calmness, happiness, sadness, anger, fear, disgust, surprise, etc. The number of specific audio emotion categories can be flexibly changed according to actual conditions, and this application will not go into details here.

[0067] Step S102: performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0068] In some embodiments, reference Figure 2 The dual-branch feature dataset includes a first branch feature dataset and a second branch feature dataset. Step S102, performing dual-branch feature extraction on the audio dataset to obtain a dual-branch feature dataset, includes:

[0069] A1. Based on a preset first enhancement rule, perform first data enhancement on the audio dataset to obtain a first branch enhanced dataset;

[0070] A2. Based on a preset second enhancement rule, perform a second data enhancement on the audio dataset to obtain a second branch enhanced dataset;

[0071] A3. Performing a first feature extraction on the first branch enhanced dataset to obtain the first branch feature dataset, and performing a second feature extraction on the second branch enhanced dataset to obtain the second branch feature dataset;

[0072] In an embodiment of the present application, first, data enhancement can be performed on the audio data set based on two different enhancement rules (i.e., the first enhancement rule and the second enhancement rule). Specifically, the data enhancement of the audio data set can be implemented based on audio editing software, and the audio editing software can be open source Audiocity.

[0073] It is understandable that after data enhancement of the audio data set is achieved and the first branch enhanced data set and the second branch enhanced data set are obtained, the audio enhanced data in the first branch enhanced data set can be converted into a Mel spectrogram (Mel spectrogram) or Mel-Frequency Cepstrum Coefficients (MFCC). The Mel spectrogram can be a pre-processing operation such as noise removal, framing, and windowing of the audio enhanced data in the first branch enhanced data set in sequence, so as to better extract the audio features in the audio enhanced data; then, a fast Fourier transform (FFT) is performed on the pre-processed audio enhanced data to convert the time signal in the audio enhanced data into a spectrum to obtain the frequency information of the audio enhanced data; then, based on the Mel filter bank, the obtained linear frequency information is mapped to the Mel scale to obtain the Mel spectrogram.

[0074] It should be noted that the mel-frequency cepstral coefficients can be obtained by performing logarithmic operations and discrete cosine transforms on the basis of the mel-frequency spectrum, which will not be described in detail in this application. It is worth mentioning that after obtaining the mel-frequency spectrum or the mel-frequency cepstral coefficients, the mel-frequency spectrum and the mel-frequency cepstral coefficients corresponding to the first frame enhancement data set can be input into the feature extractor to complete the first feature extraction and obtain the first branch feature data set.

[0075] It is also worth noting that for the second feature extraction, it is similar to the content of the first feature extraction mentioned above, and can be simply deduced. In addition, for the feature extractor, it can be constructed based on networks such as ResNet18, ResNet34, and ResNet50. The embodiment of the present application takes the feature extractor constructed based on the ResNet34 network as an example. When the sampling rate of the Mel spectrum map is 22050Hz, N_FFT=1024, and Melbins is 128, the Mel spectrum map is input to the feature extractor, which passes through the four convolution layers of the ResNet34 network and the two-layer long short-term memory (LSTM) network, and outputs a feature vector with a dimension of 512. Each feature vector with a dimension of 512 corresponds to an audio sample.

[0076] In some embodiments, reference Figure 3 , the first enhancement rule is obtained by the following steps:

[0077] B1. Obtain a preset data enhancement set, where the data enhancement set includes multiple different enhancement rules;

[0078] B2. Perform random number screening on the data enhancement set to obtain the first enhancement rule.

[0079] In an embodiment of the present application, the data enhancement set includes multiple different enhancement rules. The data enhancement set specifically includes at least two of the enhancement rules, such as increasing the audio speed by a certain ratio, slowing down the audio speed by a certain ratio, increasing the audio fundamental frequency by a certain frequency, reducing the audio fundamental frequency by a certain frequency, and enhancing noise. The specific values ​​of the certain ratio and the certain frequency can be set according to actual conditions. For example, the certain ratio can be any one of 1.1, 1.15, 1.3, etc., and the certain frequency can be any one of 20Hz, 50Hz, 100Hz, etc. The examples of this application are for illustration only and do not impose any limitation on this application.

[0080] It can be understood that when the number of random number screening in step B2 is the first time, a random integer can be randomly generated within the number of enhancement rules in the data enhancement set, and each random integer corresponds to an enhancement rule, so as to screen and obtain the first enhancement rule corresponding to the first screening; or, when the number of random number screening in step B2 is the second time, a random integer can be generated for the second time after excluding the random integer generated by the first screening and within the number of enhancement rules in the data enhancement set, and through the mapping relationship that each random integer corresponds to an enhancement rule, the first enhancement rule corresponding to the second screening is obtained by the first screening.

[0081] It should be noted that the method for obtaining the second enhancement rule in the embodiment of the present application is similar to the method for obtaining the first enhancement rule described above and can be simply deduced by analogy. In actual application, when the enhancement rule obtained by the first random number screening is the first enhancement rule, the enhancement rule obtained by the second random number screening can be the second enhancement rule.

[0082] Exemplarily, when the random number screening is the first screening and the enhancement rule corresponding to the first screening is the second enhancement rule, if the data enhancement set includes four enhancement rules: the audio speed is increased by a certain multiple, the audio speed is decreased by a certain multiple, the audio fundamental frequency is increased by a certain frequency, and the audio fundamental frequency is decreased by a certain frequency, when the generated random integer is 3, the enhancement rule corresponding to the random integer is that the audio fundamental frequency is increased by a certain frequency. At this time, increasing the audio fundamental frequency by a certain frequency can be used as the second enhancement rule.

[0083] When the random number screening is the second screening, since the first screening is the second enhancement rule, the enhancement rule corresponding to the second screening at this time is the first enhancement rule. At this time, any one of the numbers 1, 2, and 4 can be used as the generated random integer, and the generated random integer can be used as the first enhancement rule.

[0084] Step S103: inputting the dual-branch feature dataset into the instance-level network framework to obtain an acoustic feature representation, where the acoustic feature representation is used to record multiple audio features of the original speech at the instance level;

[0085] In some embodiments, the acoustic feature representation includes a first audio feature matrix and a second audio feature matrix. Step S103 of inputting the dual-branch feature dataset into the instance-level network framework to obtain the acoustic feature representation includes:

[0086] C1. Inputting the first branch feature dataset into the instance-level network framework to obtain the first audio feature matrix;

[0087] C2. Inputting the second branch feature dataset into the instance-level network framework to obtain the second audio feature matrix;

[0088] The first audio feature matrix includes multiple first audio feature vectors, and the second audio feature matrix includes multiple second audio feature vectors; the number of rows to which each first audio feature vector belongs is the same as the number of rows to which the corresponding second audio feature vector belongs.

[0089] In an embodiment of the present application, an instance-level network framework can be constructed based on a multi-layer perceptron, the number of hidden layers of the multi-layer perceptron can be any one of 512, 256, 128, etc., and the activation function is a Si gmod function. The first branch feature data set is input into the instance-level network framework for instance-level learning to obtain a first audio feature matrix, each matrix element in the first audio feature matrix is ​​used as a first audio feature vector, the audio sample inputs corresponding to the first audio feature vectors with the same number of rows are the same, and the audio sample inputs corresponding to the first audio feature vectors with different numbers of rows are different.

[0090] It can be understood that, for the second audio feature matrix in step C2, second audio feature vectors with the same number of rows correspond to the same audio sample inputs, while second audio feature vectors with different numbers of rows correspond to different audio sample inputs. In addition, the first audio feature vector and the corresponding second audio feature vector correspond to audio sample inputs after data enhancement based on different enhancement rules.

[0091] Step S104: inputting the dual-branch feature dataset into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0092] In some embodiments, the speech emotion classification result includes a first emotion category matrix and a second emotion category matrix. Step S104, inputting the dual-branch feature dataset into the cluster-level network framework to obtain the speech emotion classification result, includes:

[0093] D1. Inputting the first branch feature data set into the cluster-level network framework to obtain the first emotion category matrix;

[0094] D2. Inputting the second branch feature data set into the cluster-level network framework to obtain the second emotion category matrix;

[0095] Among them, the first emotion category matrix includes multiple first emotion category feature vectors, and the second emotion category matrix includes multiple second emotion category feature vectors; the first emotion category feature vectors with the same number of columns correspond to the same emotion prediction category; the second emotion category feature vectors with the same number of columns correspond to the same emotion prediction category.

[0096] In an embodiment of the present application, a cluster-level network framework can be constructed based on a multi-layer perceptron, the number of hidden layers of the multi-layer perceptron can be equal to the number of emotion categories in the audio data set, and the activation function is a Si gmod function. The first branch feature data set is input into the cluster-level network framework for cluster-level learning to obtain a first emotion category matrix, each matrix element in the first emotion category matrix serves as a first emotion category vector, the first emotion category vectors with the same number of columns correspond to the same emotion prediction category, and the first emotion category vectors with different numbers of columns correspond to different emotion prediction categories.

[0097] It can be understood that, for the second emotion category matrix in step C2, second emotion category vectors with the same number of columns correspond to the same emotion prediction category, and second emotion category vectors with different number of columns correspond to different emotion prediction categories.

[0098] Step S105: updating the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

[0099] In some embodiments, the step S105 of updating the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model includes:

[0100] E1. Performing row-wise screening on the acoustic feature representation to obtain a first audio row set having a positive example relationship and a second audio row set having a non-positive example relationship;

[0101] E2. Filter the speech emotion classification results by columns to obtain a first category column set having a positive example relationship and a second category column set having a non-positive example relationship;

[0102] E3. Calculate an instance-level loss value by maximizing the cosine similarity of the first audio row set and minimizing the cosine similarity of the second audio row set;

[0103] E4. Calculate a cluster-level loss value by maximizing the cosine similarity of the first category column set and minimizing the cosine similarity of the second category column set;

[0104] E5. Based on the instance-level loss value and the cluster-level loss value, update the parameters of the initialized speech emotion model to obtain the trained speech emotion model.

[0105] In an embodiment of the present application, the first audio feature vector and the second audio feature vector with the same number of rows in the acoustic feature representation can be integrated to obtain a first audio row set with a positive example relationship; the first audio feature vector and the second audio feature vector with different numbers of rows in the acoustic feature representation can be integrated to obtain an audio row set with a non-positive example relationship.

[0106] It can be understood that for the column screening in step E2, it can be to integrate the first emotion category feature vectors and the second emotion category feature vectors with the same number of columns in the speech emotion classification results to obtain a first category column set with a positive example relationship; and to integrate the first emotion category feature vectors and the second emotion category feature vectors with different numbers of columns in the speech emotion classification results to obtain a second category column set with a non-positive example relationship.

[0107] It should be noted that maximizing the cosine similarity of the data sets with a positive example relationship in steps E3 and E4 is used to shorten the distance between the first audio feature vector and the second audio feature vector with a positive example relationship, or shorten the distance between the first emotion category feature vector and the second emotion category feature vector with a positive example relationship. Minimizing the cosine similarity of the data sets with a non-positive example relationship is used to increase the distance between the first audio feature vector and the second audio feature vector with a non-positive example relationship, or increase the distance between the first emotion category feature vector and the second emotion category feature vector with a non-positive example relationship.

[0108] Exemplarily, the equivalent expression of the instance-level loss value corresponding to the first audio feature vector of a certain matrix row number is:

[0109]

[0110] in, is the instance-level loss value corresponding to the first audio feature matrix a with matrix row number i; s() is the cosine similarity function; is the first audio feature vector in the first audio feature matrix a, the i-th matrix row; is the second audio feature vector of the i-th matrix row in the second audio feature matrix b; τ1 is the temperature coefficient; N is the maximum number of matrix columns; j is the matrix column number; is the first audio feature vector in the first audio feature matrix a, the jth matrix column; is the second audio feature vector in the j-th matrix column of the second audio feature matrix b.

[0111] It is worth mentioning that the instance-level loss values ​​corresponding to the first audio feature vectors of the remaining matrix rows and the instance-level loss values ​​corresponding to the second audio feature vectors of all matrix rows are the same as those mentioned above. The content of the cluster-level loss value is similar to that of the aforementioned instance-level loss value. The difference lies in calculating the cosine similarity between the first emotion category feature vector and the second emotion category feature vector with the same or different number of matrix columns. This application will not go into details here.

[0112] It's worth noting that after obtaining the instance-level loss values ​​corresponding to all identical or different numbers of matrix rows, as well as the cluster-level loss values ​​corresponding to all identical or different numbers of matrix columns, the overall loss value of the speech emotion model can be determined based on these loss values. Based on this trained loss value, the model parameters are updated using the backpropagation algorithm. After several rounds of iteration, a trained speech emotion model is obtained. The specific number of iterations can be pre-set, or training is considered complete when the speech emotion model meets the required accuracy.

[0113] Figure 4 This is an optional flowchart of the method for recognizing a speech emotion model provided in an embodiment of the present application. Figure 4 The identification method may include but is not limited to steps S106 to S107.

[0114] Step S106: obtaining a speech data set to be recognized;

[0115] Step S107: input the speech data set into the trained speech emotion model to perform emotion recognition and obtain a speech emotion recognition result.

[0116] In an embodiment of the present application, the speech data set to be recognized can be input into a trained speech emotion model, and the speech data set is input into a cluster-level network framework through a feature extractor to obtain the prediction probability of the speech data set in each emotion prediction category. The emotion prediction category corresponding to the maximum prediction probability is the speech emotion recognition result obtained by the speech emotion model predicting and recognizing the speech data set.

[0117] See also Figure 5 The present application also provides a speech emotion model training device, which can implement the above-mentioned speech emotion model training method. The training device includes:

[0118] An acquisition module 801 is used to acquire an audio data set of original speech;

[0119] An extraction module 802 is configured to perform dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0120] A first processing module 803 is configured to input the dual-branch feature dataset into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used to record multiple audio features of the original speech at the instance level;

[0121] A second processing module 804 is configured to input the dual-branch feature dataset into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0122] The updating module 805 is used to update the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

[0123] The specific implementation of the speech emotion model training device is basically the same as the specific embodiment of the above-mentioned speech emotion model training method, and will not be repeated here.

[0124] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0125] See also Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0126] The processor 901 may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0127] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the methods of the embodiments of this application.

[0128] Input / output interface 903, used to implement information input and output;

[0129] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0130] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0131] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0132] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above method is implemented.

[0133] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0135] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0137] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0138] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0139] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0144] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for training a speech emotion model, characterized in that: The speech emotion model includes an instance-level network framework and a cluster-level network framework, the instance-level network framework and the cluster-level network framework are connected in parallel, and the training method includes the following steps: Get the audio dataset of the original speech; Performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set; Inputting the dual-branch feature dataset into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used to record multiple audio features of the original speech at the instance level; Inputting the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level; According to the acoustic feature representation and the speech emotion classification result, the parameters of the initialized speech emotion model are updated to obtain a trained speech emotion model.

2. The training method according to claim 1, characterized in that The dual-branch feature dataset includes a first branch feature dataset and a second branch feature dataset, and the dual-branch feature extraction is performed on the audio dataset to obtain the dual-branch feature dataset, including: Based on a preset first enhancement rule, performing a first data enhancement on the audio dataset to obtain a first branch enhanced dataset; Based on a preset second enhancement rule, performing a second data enhancement on the audio dataset to obtain a second branch enhanced dataset; Performing a first feature extraction on the first branch enhanced data set to obtain the first branch feature data set, and performing a second feature extraction on the second branch enhanced data set to obtain the second branch feature data set; The first enhancement rule is different from the second enhancement rule.

3. The training method according to claim 2, characterized in that The first enhancement rule is obtained by the following steps: Obtaining a preset data enhancement set, wherein the data enhancement set includes a plurality of different enhancement rules; Random number screening is performed on the data enhancement set to obtain the first enhancement rule.

4. The training method according to claim 2, characterized in that The acoustic feature representation includes a first audio feature matrix and a second audio feature matrix, and the inputting the dual-branch feature dataset into the instance-level network framework to obtain the acoustic feature representation includes: Inputting the first branch feature data set into the instance-level network framework to obtain the first audio feature matrix; Inputting the second branch feature data set into the instance-level network framework to obtain the second audio feature matrix; The first audio feature matrix includes multiple first audio feature vectors, and the second audio feature matrix includes multiple second audio feature vectors; the number of rows to which each first audio feature vector belongs is the same as the number of rows to which the corresponding second audio feature vector belongs.

5. The training method according to claim 2, characterized in that: The speech emotion classification result includes a first emotion category matrix and a second emotion category matrix, and the dual-branch feature data set is input into the cluster-level network framework to obtain the speech emotion classification result, including: Inputting the first branch feature data set into the cluster-level network framework to obtain the first emotion category matrix; Inputting the second branch feature data set into the cluster-level network framework to obtain the second emotion category matrix; Among them, the first emotion category matrix includes multiple first emotion category feature vectors, and the second emotion category matrix includes multiple second emotion category feature vectors; the first emotion category feature vectors with the same number of columns correspond to the same emotion prediction category; the second emotion category feature vectors with the same number of columns correspond to the same emotion prediction category.

6. The training method according to claim 1, characterized in that: The method of updating parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model includes: Performing row-wise screening on the acoustic feature representation to obtain a first audio row set having a positive example relationship and a second audio row set having a non-positive example relationship; Performing column screening on the speech emotion classification results to obtain a first category column set having a positive example relationship and a second category column set having a non-positive example relationship; Calculating an instance-level loss value by maximizing the cosine similarity of the first audio row set and minimizing the cosine similarity of the second audio row set; The cluster-level loss value is calculated by maximizing the cosine similarity of the first category column set and minimizing the cosine similarity of the second category column set; Based on the instance-level loss value and the cluster-level loss value, the parameters of the initialized speech emotion model are updated to obtain the trained speech emotion model.

7. A method for recognizing a speech emotion model, characterized in that: The identification method comprises the following steps: Obtain the speech data set to be recognized; The speech data set is input into the speech emotion model trained as described in any one of claims 1-6 to perform emotion recognition to obtain a speech emotion recognition result.

8. A training device for a speech emotion model, characterized in that: The speech emotion model includes an instance-level network framework and a cluster-level network framework, the instance-level network framework and the cluster-level network framework are connected in parallel, and the training device includes: An acquisition module is used to obtain an audio data set of original speech; An extraction module, configured to perform dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set; A first processing module is configured to input the dual-branch feature dataset into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is configured to record a plurality of audio features of the original speech at the instance level; A second processing module is used to input the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level; The updating module is used to update the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images