Speech emotion model training method and device and speech emotion model recognition method and device

Through the method of dual-branch feature extraction and joint training of network frameworks, the problem of high computing resources consumption in speech emotion recognition technology is solved, the recognition efficiency is improved, and emotional prediction is suitable for intelligent customer service.

CN119049514AActive Publication Date: 2024-11-29PING AN TECH (SHENZHEN) CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410986655.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-29
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

The existing voice emotion recognition technology requires the extraction and learning of voice data at each instance level, resulting in high computing resources and low efficiency.

Method used

The dual-branch feature extraction method is used to input the audio data set into the instance-level network framework and the cluster-level network framework for training. The speech emotion model is jointly trained through acoustic feature representation and speech emotion classification results to reduce the computing resource requirements.

Benefits of technology

It improves the efficiency of speech emotion recognition, reduces the computing resource requirements of speech emotion models in predicting recognition, and is suitable for application scenarios such as digital medical care and digital finance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049514A_ABST
    Figure CN119049514A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a voice emotion model training method and device and a voice emotion model recognition method and device.A voice emotion model comprises an instance-level network framework and a cluster-level network framework, and the training method comprises the steps that an audio data set of original voice is acquired; performing double-branch feature extraction on the audio data set to obtain a double-branch feature data set; inputting the double-branch feature data set into the instance-level network framework to obtain acoustic feature representation; inputting the double-branch feature data set into the cluster-level network framework to obtain a voice emotion classification result; and according to the acoustic feature representation and the speech emotion classification result, performing parameter updating on the initialized speech emotion model to obtain a trained speech emotion model. The training method can effectively reduce the computing power resources required by the voice emotion model for prediction and recognition, and improves the efficiency of voice emotion recognition. The invention relates to the technical field of speech recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a training method, a recognition method and a device for a speech emotion model. Background Art

[0002] Speech Emotion Recognition (SER) refers to the technology of identifying the speaker's emotional state through the acoustic features of a speech. It can help intelligent customer service better understand the user's voice content and emotional changes and has attracted much attention.

[0003] At present, speech emotion recognition is mainly achieved by extracting different acoustic and spectral features of speech, and then using deep learning networks to analyze and learn the emotional information contained in these features. This method is aimed at instance-level speech data, and it needs to extract and learn speech data at each instance level. However, due to the large amount of speech data and the large amount of computing resources required, the efficiency of speech emotion recognition is not high.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The object of the present invention is to solve one of the technical problems existing in the related art to at least some extent.

[0006] The main purpose of the embodiments of the present application is to propose a training method, a recognition method and a device for a speech emotion model, wherein the training method can effectively reduce the computing power resources required for predictive recognition of the speech emotion model and improve the efficiency of speech emotion recognition.

[0007] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for training a speech emotion model, wherein the speech emotion model includes an instance-level network framework and a cluster-level network framework, wherein the instance-level network framework and the cluster-level network framework are connected in parallel, and the training method includes:

[0008] Get the audio dataset of the original speech;

[0009] Performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0010] Inputting the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used to record multiple audio features of the original speech at the instance level;

[0011] Inputting the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0012] According to the acoustic feature representation and the speech emotion classification result, the parameters of the initialized speech emotion model are updated to obtain a trained speech emotion model.

[0013] In some embodiments, the dual-branch feature data set includes a first branch feature data set and a second branch feature data set, and performing dual-branch feature extraction on the audio data set to obtain the dual-branch feature data set includes:

[0014] Based on a preset first enhancement rule, performing a first data enhancement on the audio data set to obtain a first branch enhanced data set;

[0015] Based on a preset second enhancement rule, performing a second data enhancement on the audio data set to obtain a second branch enhanced data set;

[0016] Performing a first feature extraction on the first branch enhanced data set to obtain the first branch feature data set, and performing a second feature extraction on the second branch enhanced data set to obtain the second branch feature data set;

[0017] The first enhancement rule is different from the second enhancement rule.

[0018] In some embodiments, the first enhancement rule is obtained by the following steps:

[0019] Obtaining a preset data enhancement set, wherein the data enhancement set includes a plurality of different enhancement rules;

[0020] Random number screening is performed on the data enhancement set to obtain the first enhancement rule.

[0021] In some embodiments, the acoustic feature representation includes a first audio feature matrix and a second audio feature matrix, and the step of inputting the dual-branch feature data set into the instance-level network framework to obtain the acoustic feature representation includes:

[0022] Inputting the first branch feature data set into the instance-level network framework to obtain the first audio feature matrix;

[0023] Inputting the second branch feature data set into the instance-level network framework to obtain the second audio feature matrix;

[0024] The first audio feature matrix includes multiple first audio feature vectors, and the second audio feature matrix includes multiple second audio feature vectors; the number of rows to which each first audio feature vector belongs is the same as the number of rows to which the corresponding second audio feature vector belongs.

[0025] In some embodiments, the speech emotion classification result includes a first emotion category matrix and a second emotion category matrix, and the inputting of the dual-branch feature data set into the cluster-level network framework to obtain the speech emotion classification result includes:

[0026] Inputting the first branch feature data set into the cluster-level network framework to obtain the first emotion category matrix;

[0027] Inputting the second branch feature data set into the cluster-level network framework to obtain the second emotion category matrix;

[0028] Among them, the first emotion category matrix includes multiple first emotion category feature vectors, and the second emotion category matrix includes multiple second emotion category feature vectors; the first emotion category feature vectors with the same number of columns correspond to the same emotion prediction categories; the second emotion category feature vectors with the same number of columns correspond to the same emotion prediction categories.

[0029] In some embodiments, the updating of parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model includes:

[0030] Performing row-wise screening on the acoustic feature representation to obtain a first audio row set having a positive example relationship and a second audio row set having a non-positive example relationship;

[0031] Performing column screening on the speech emotion classification results to obtain a first category column set having a positive example relationship and a second category column set having a non-positive example relationship;

[0032] Calculating an instance-level loss value by maximizing the cosine similarity of the first audio row set and minimizing the cosine similarity of the second audio row set;

[0033] By maximizing the cosine similarity of the first category column set and minimizing the cosine similarity of the second category column set, a cluster-level loss value is calculated;

[0034] Based on the instance-level loss value and the cluster-level loss value, the parameters of the initialized speech emotion model are updated to obtain the trained speech emotion model.

[0035] To achieve the above object, a second aspect of an embodiment of the present application proposes a method for recognizing a speech emotion model, the method comprising the following steps:

[0036] Obtain the speech data set to be recognized;

[0037] The speech data set is input into the above-mentioned trained speech emotion model for emotion recognition to obtain a speech emotion recognition result.

[0038] To achieve the above-mentioned purpose, the third aspect of the embodiment of the present application proposes a training device for a speech emotion model, wherein the speech emotion model includes an instance-level network framework and a cluster-level network framework, wherein the instance-level network framework and the cluster-level network framework are connected in parallel, and the training device includes:

[0039] An acquisition module is used to acquire an audio data set of original speech;

[0040] An extraction module, used for performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0041] A first processing module, used for inputting the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used for recording a plurality of audio features of the original speech at the instance level;

[0042] A second processing module is used to input the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0043] The updating module is used to update the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

[0044] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0045] To achieve the above objectives, a fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0046] The present application proposes a training method, a recognition method and a device for a speech emotion model, wherein the training method obtains an audio data set of original speech; performs dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set; inputs the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation, which is used to record multiple audio features of the original speech at the instance level; inputs the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, which is used to record multiple emotion prediction categories of the original speech at the cluster level; and updates the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model. The training method inputs the dual-branch feature data set into the instance-level network framework and the cluster-level network framework for training respectively, and can learn the audio data at two granularities, namely, the instance level and the clustering cluster level. The speech emotion model is jointly trained by using the learned acoustic feature representation and speech emotion classification results, which can reduce the computing resources required for the speech emotion model to perform speech emotion prediction and recognition, and improve the efficiency of speech emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a schematic diagram of the process principle of a method for training a speech emotion model provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Schematic diagram of the process principle of step S102 in FIG.

[0050] Figure 3 is a schematic diagram of the process principle of the first enhancement rule provided in an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of the process principle of a method for recognizing a speech emotion model provided in an embodiment of the present application;

[0052] Figure 5 It is a structural schematic diagram of a speech emotion model training device provided in an embodiment of the present application;

[0053] Figure 6It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] At present, speech emotion recognition is mainly achieved by extracting different acoustic and spectral features of speech, and then using deep learning networks to analyze and learn the emotional information contained in these features. This method is aimed at instance-level speech data, and it needs to extract and learn speech data at each instance level. However, due to the large amount of speech data and the large amount of computing resources required, the efficiency of speech emotion recognition is not high.

[0058] In view of this, an embodiment of the present application provides a training method, a recognition method and a device for a speech emotion model, wherein the training method inputs a dual-branch feature data set into an instance-level network framework and a cluster-level network framework for training respectively, and can perform feature learning at two granularities, namely, instance level and cluster level, on the audio data. The speech emotion model is jointly trained by using the learned acoustic feature representation and speech emotion classification results, which can reduce the computing resources required for the speech emotion model to perform speech emotion prediction and recognition, and improve the efficiency of speech emotion recognition.

[0059] The present application provides a speech emotion model training method, recognition method and device, which are specifically described through the following embodiments.

[0060] The method provided in the embodiment of the present application can be applied to digital medical application scenarios or digital financial application scenarios. In digital medical application scenarios or digital financial application scenarios, the speech emotion model provided in the embodiment of the present application can be loaded in the digital medical intelligent customer service or digital financial intelligent customer service. The speech emotion model requires low computing resources for speech emotion prediction and recognition, and has high efficiency in speech emotion recognition. The speech emotion model helps the intelligent customer service identify the user's current emotional state, so that the intelligent customer service can respond more reasonably and quickly, thereby improving the user's satisfaction with the intelligent customer service;

[0061] The training method or recognition method of the speech emotion model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the above method, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0064] Figure 1 is an optional flow chart of a method for training a speech emotion model provided in an embodiment of the present application, wherein the speech emotion model comprises an instance-level network framework and a cluster-level network framework, wherein the instance-level network framework and the cluster-level network framework are connected in parallel, Figure 1 The training method may include but is not limited to steps S101 to S105.

[0065] Step S101, obtaining an audio data set of original speech;

[0066] In an embodiment of the present application, the audio data set of the original speech can be the audio data of several voice speakers over a certain length of time, which can be collected based on an audio sensor, or a public audio data set can be used. The specific public audio data set can be any one of the I EMOCAP audio data set, the RAVDESS audio data set, etc., wherein the audio emotion categories in the I EMOCAP audio data set include anger, happiness, sadness, neutral, frustration, excitement, fear, disgust, etc., and the audio emotion categories in the RAVDESS audio data set include neutral, calm, happiness, sadness, anger, fear, disgust, surprise, etc. The number of specific audio emotion categories can be flexibly changed according to actual conditions, and this application will not go into details here.

[0067] Step S102, performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0068] In some embodiments, reference Figure 2 The dual-branch feature data set includes a first branch feature data set and a second branch feature data set. The step S102, performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set, includes:

[0069] A1. Based on a preset first enhancement rule, perform a first data enhancement on the audio data set to obtain a first branch enhanced data set;

[0070] A2. Based on a preset second enhancement rule, perform a second data enhancement on the audio data set to obtain a second branch enhanced data set;

[0071] A3. performing a first feature extraction on the first branch enhanced data set to obtain the first branch feature data set, and performing a second feature extraction on the second branch enhanced data set to obtain the second branch feature data set;

[0072] In an embodiment of the present application, first, data enhancement can be performed on the audio data set based on two different enhancement rules (i.e., the first enhancement rule and the second enhancement rule). Specifically, the data enhancement of the audio data set can be implemented based on audio editing software, and the audio editing software can be open source Audiocity.

[0073] It is understandable that after data enhancement of the audio data set is realized and the first branch enhanced data set and the second branch enhanced data set are obtained, the audio enhanced data in the first branch enhanced data set can be converted into a Mel spectrogram or Mel-Frequency Cepstra l Coeff ic ients (MFCC). Wherein, the Mel spectrogram can be the audio enhanced data in the first branch enhanced data set subjected to preprocessing operations such as noise removal, framing, and windowing in sequence, so as to better extract the audio features in the audio enhanced data; then, the preprocessed audio enhanced data is subjected to a fast Fourier transform (FFT), the time signal in the audio enhanced data is converted into a spectrum, and the frequency information of the audio enhanced data is obtained; then, based on the Mel filter bank, the obtained linear frequency information is mapped to the Mel scale, so as to obtain the Mel spectrogram.

[0074] It should be noted that for the Mel-frequency cepstral coefficients, they can be obtained by performing logarithmic operations and discrete cosine transforms in sequence on the basis of the Mel-frequency spectrum, which will not be repeated in this application. It is worth mentioning that after obtaining the Mel-frequency spectrum or the Mel-frequency cepstral coefficients, the Mel-frequency spectrum and the Mel-frequency cepstral coefficients corresponding to the first frame enhancement data set can be input into the feature extractor to complete the first feature extraction and obtain the first branch feature data set.

[0075] It is also worth noting that for the second feature extraction, it is similar to the content of the first feature extraction mentioned above, and can be simply deduced. In addition, for the feature extractor, it can be constructed based on networks such as ResNet18, ResNet34, and ResNet50. The embodiment of the present application takes the feature extractor constructed based on the ResNet34 network as an example. When the sampling rate of the Mel spectrum map is 22050Hz, N_FFT=1024, and Melbins is 128, the Mel spectrum map is input to the feature extractor, which passes through the four convolutional layers of the ResNet34 network and the two-layer long short-term memory (LSTM) network, and outputs a feature vector with a dimension of 512. Each feature vector with a dimension of 512 corresponds to an audio sample.

[0076] In some embodiments, reference Figure 3 , the first enhancement rule is obtained by the following steps:

[0077] B1. Obtain a preset data enhancement set, where the data enhancement set includes a plurality of different enhancement rules;

[0078] B2. Perform random number screening on the data enhancement set to obtain the first enhancement rule.

[0079] In an embodiment of the present application, the data enhancement set includes multiple different enhancement rules, and the data enhancement set specifically includes at least two of the enhancement rules such as increasing the audio speed by a certain multiple, slowing down the audio speed by a certain multiple, increasing the audio fundamental frequency by a certain frequency, reducing the audio fundamental frequency by a certain frequency, and enhancing noise. Among them, the specific values ​​of the certain multiple and the certain frequency can be set according to actual conditions. For example, the certain multiple can be any one of 1.1, 1.15, 1.3, etc., and the certain frequency can be any one of 20Hz, 50Hz, 100Hz, etc. The examples of this application are for illustration only and do not impose any limitation on this application.

[0080] It can be understood that when the number of random number screening in step B2 is the first time, a random integer can be randomly generated within the number of enhancement rules in the data enhancement set, and each random integer corresponds to an enhancement rule, so as to screen and obtain the first enhancement rule corresponding to the first screening; or, when the number of random number screening in step B2 is the second time, a random integer can be generated for the second time within the number of enhancement rules in the data enhancement set after excluding the random integer generated by the first screening, and through the mapping relationship that each random integer corresponds to an enhancement rule, the first screening is performed at the first enhancement rule corresponding to the second screening.

[0081] It should be noted that the method for obtaining the second enhancement rule in the embodiment of the present application is similar to the method for obtaining the first enhancement rule, and can be simply deduced by analogy. In practical applications, when the enhancement rule obtained by the first random number screening is the first enhancement rule, the enhancement rule obtained by the second random number screening can be the second enhancement rule.

[0082] Exemplarily, when the random number screening is the first screening and the enhancement rule corresponding to the first screening is the second enhancement rule, if the data enhancement set includes four enhancement rules, namely, increasing the audio speed by a certain multiple, decreasing the audio speed by a certain multiple, increasing the audio fundamental frequency by a certain frequency, and decreasing the audio fundamental frequency by a certain frequency, when the generated random integer is 3, the enhancement rule corresponding to the random integer is increasing the audio fundamental frequency by a certain frequency. At this time, increasing the audio fundamental frequency by a certain frequency can be used as the second enhancement rule.

[0083] When the random number screening is the second screening, since the first screening is the second enhancement rule, the enhancement rule corresponding to the second screening at this time is the first enhancement rule. At this time, any one of the numbers 1, 2, and 4 can be used as the generated random integer, and the generated random integer can be used as the first enhancement rule.

[0084] Step S103: inputting the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation, where the acoustic feature representation is used to record multiple audio features of the original speech at the instance level;

[0085] In some embodiments, the acoustic feature representation includes a first audio feature matrix and a second audio feature matrix, and the step S103 of inputting the dual-branch feature data set into the instance-level network framework to obtain the acoustic feature representation includes:

[0086] C1. Inputting the first branch feature data set into the instance-level network framework to obtain the first audio feature matrix;

[0087] C2. Inputting the second branch feature data set into the instance-level network framework to obtain the second audio feature matrix;

[0088] The first audio feature matrix includes multiple first audio feature vectors, and the second audio feature matrix includes multiple second audio feature vectors; the number of rows to which each first audio feature vector belongs is the same as the number of rows to which the corresponding second audio feature vector belongs.

[0089] In an embodiment of the present application, an instance-level network framework can be constructed based on a multi-layer perceptron, the number of hidden layers of the multi-layer perceptron can be any one of 512, 256, 128, etc., and the activation function is a Sigmod function. The first branch feature data set is input into the instance-level network framework for instance-level learning to obtain a first audio feature matrix, each matrix element in the first audio feature matrix is ​​used as a first audio feature vector, the audio sample inputs corresponding to the first audio feature vectors with the same number of rows are the same, and the audio sample inputs corresponding to the first audio feature vectors with different numbers of rows are different.

[0090] It can be understood that, for the second audio feature matrix in step C2, the audio sample inputs corresponding to the second audio feature vectors with the same number of rows are the same, and the audio sample inputs corresponding to the second audio feature vectors with different number of rows are different. In addition, for the first audio feature vector and the corresponding second audio feature vector, they correspond to the audio sample input after data enhancement based on different enhancement rules.

[0091] Step S104, inputting the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0092] In some embodiments, the speech emotion classification result includes a first emotion category matrix and a second emotion category matrix, and the step S104, inputting the dual-branch feature data set into the cluster-level network framework to obtain the speech emotion classification result, includes:

[0093] D1. Inputting the first branch feature data set into the cluster-level network framework to obtain the first emotion category matrix;

[0094] D2, inputting the second branch feature data set into the cluster-level network framework to obtain the second emotion category matrix;

[0095] Among them, the first emotion category matrix includes multiple first emotion category feature vectors, and the second emotion category matrix includes multiple second emotion category feature vectors; the first emotion category feature vectors with the same number of columns correspond to the same emotion prediction categories; the second emotion category feature vectors with the same number of columns correspond to the same emotion prediction categories.

[0096] In an embodiment of the present application, a cluster-level network framework can be constructed based on a multi-layer perceptron, the number of hidden layers of the multi-layer perceptron can be equal to the number of emotion categories in the audio data set, and the activation function is a Sigmod function. The first branch feature data set is input into the cluster-level network framework for cluster-level learning to obtain a first emotion category matrix, in which each matrix element in the first emotion category matrix serves as a first emotion category vector, first emotion category vectors with the same number of columns correspond to the same emotion prediction category, and first emotion category vectors with different number of columns correspond to different emotion prediction categories.

[0097] It can be understood that, for the second emotion category matrix in step C2, the emotion prediction categories corresponding to the second emotion category vectors with the same number of columns are the same, and the emotion prediction categories corresponding to the second emotion category vectors with different number of columns are different.

[0098] Step S105: updating the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

[0099] In some embodiments, the step S105, updating the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model, includes:

[0100] E1. Performing row-wise screening on the acoustic feature representation to obtain a first audio row set having a positive example relationship and a second audio row set having a non-positive example relationship;

[0101] E2. Filter the speech emotion classification results by columns to obtain a first category column set having a positive example relationship and a second category column set having a non-positive example relationship;

[0102] E3. Calculate an instance-level loss value by maximizing the cosine similarity of the first audio row set and minimizing the cosine similarity of the second audio row set;

[0103] E4. Calculate a cluster-level loss value by maximizing the cosine similarity of the first category column set and minimizing the cosine similarity of the second category column set;

[0104] E5. Based on the instance-level loss value and the cluster-level loss value, update the parameters of the initialized speech emotion model to obtain the trained speech emotion model.

[0105] In an embodiment of the present application, the first audio feature vector and the second audio feature vector with the same number of rows in the acoustic feature representation can be integrated to obtain a first audio row set with a positive example relationship; the first audio feature vector and the second audio feature vector with different numbers of rows in the acoustic feature representation can be integrated to obtain an audio row set with a non-positive example relationship.

[0106] It can be understood that for the column-based screening in step E2, it can be to integrate the first emotion category feature vectors and the second emotion category feature vectors with the same number of columns in the speech emotion classification results, so as to obtain a first category column set with a positive example relationship; and to integrate the first emotion category feature vectors and the second emotion category feature vectors with different numbers of columns in the speech emotion classification results, so as to obtain a second category column set with a non-positive example relationship.

[0107] It should be noted that the maximization of the cosine similarity of the data set with a positive example relationship in step E3 and step E4 is used to shorten the distance between the first audio feature vector and the second audio feature vector with a positive example relationship, or shorten the distance between the first emotion category feature vector and the second emotion category feature vector with a positive example relationship. The minimization of the cosine similarity of the data set with a non-positive example relationship is used to increase the distance between the first audio feature vector and the second audio feature vector with a non-positive example relationship, or increase the distance between the first emotion category feature vector and the second emotion category feature vector with a non-positive example relationship.

[0108] Exemplarily, an equivalent expression of the instance-level loss value corresponding to the first audio feature vector of a certain number of matrix rows is:

[0109]

[0110] in, is the instance-level loss value corresponding to the first audio feature matrix a with matrix row number i; s() is the cosine similarity function; is the first audio feature vector of the i-th matrix row in the first audio feature matrix a; is the second audio feature vector of the i-th matrix row in the second audio feature matrix b; τ1 is the temperature coefficient; N is the maximum number of matrix columns; j is the number of matrix columns; is the first audio feature vector of the j-th matrix column in the first audio feature matrix a; is the second audio feature vector of the j-th matrix column in the second audio feature matrix b.

[0111] It is worth mentioning that the instance-level loss values ​​corresponding to the first audio feature vectors of the remaining matrix rows and the instance-level loss values ​​corresponding to the second audio feature vectors of all matrix rows are the same as those mentioned above. The content of the cluster-level loss value is similar to that of the aforementioned instance-level loss value, the difference lies in calculating the cosine similarity between the first emotion category feature vector and the second emotion category feature vector with the same or different number of matrix columns, which will not be repeated in this application.

[0112] It is worth noting that after obtaining the instance-level loss values ​​corresponding to all the same or different numbers of matrix rows, and the cluster-level loss values ​​corresponding to all the same or different numbers of matrix columns, the overall loss value of the speech emotion model can be determined based on the above loss values, and based on the loss values ​​obtained from the training, the back propagation algorithm is used to update the parameters of the model, and the trained speech emotion model can be obtained after several rounds of iteration. The specific number of iterations can be set in advance, or the training is considered to be completed when the speech emotion model meets the accuracy requirements.

[0113] Figure 4 is an optional flowchart of the method for recognizing a speech emotion model provided in an embodiment of the present application. Figure 4 The identification method may include but is not limited to steps S106 to S107.

[0114] Step S106, obtaining a speech data set to be recognized;

[0115] Step S107: input the speech data set into the trained speech emotion model to perform emotion recognition and obtain a speech emotion recognition result.

[0116] In an embodiment of the present application, the speech data set to be recognized can be input into a trained speech emotion model, and the speech data set is input into a cluster-level network framework via a feature extractor to obtain the prediction probability of the speech data set in each emotion prediction category, and the emotion prediction category corresponding to the maximum prediction probability is the speech emotion recognition result obtained by the speech emotion model predicting and recognizing the speech data set.

[0117] See also Figure 5 The present application also provides a training device for a speech emotion model, which can implement the training method for the speech emotion model. The training device includes:

[0118] An acquisition module 801 is used to acquire an audio data set of original speech;

[0119] An extraction module 802 is used to perform dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set;

[0120] A first processing module 803 is used to input the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation, where the acoustic feature representation is used to record multiple audio features of the original speech at the instance level;

[0121] The second processing module 804 is used to input the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level;

[0122] The updating module 805 is used to update the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

[0123] The specific implementation of the training device for the speech emotion model is basically the same as the specific implementation of the training method for the speech emotion model mentioned above, and will not be repeated here.

[0124] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0125] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0126] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0127] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 calls and executes the methods of the embodiments of this application;

[0128] Input / output interface 903, used to implement information input and output;

[0129] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0130] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0131] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0132] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0133] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0135] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0136] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0138] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0139] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0144] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for training a speech emotion model, characterized in that: The speech emotion model includes an instance-level network framework and a cluster-level network framework, the instance-level network framework and the cluster-level network framework are connected in parallel, and the training method includes the following steps: Get the audio dataset of the original speech; Performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set; Inputting the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used to record multiple audio features of the original speech at the instance level; Inputting the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level; According to the acoustic feature representation and the speech emotion classification result, the parameters of the initialized speech emotion model are updated to obtain a trained speech emotion model.

2. The training method according to claim 1, characterized in that: The dual-branch feature data set includes a first branch feature data set and a second branch feature data set, and the dual-branch feature extraction is performed on the audio data set to obtain the dual-branch feature data set, including: Based on a preset first enhancement rule, performing a first data enhancement on the audio data set to obtain a first branch enhanced data set; Based on a preset second enhancement rule, performing a second data enhancement on the audio data set to obtain a second branch enhanced data set; Performing a first feature extraction on the first branch enhanced data set to obtain the first branch feature data set, and performing a second feature extraction on the second branch enhanced data set to obtain the second branch feature data set; The first enhancement rule is different from the second enhancement rule.

3. The training method according to claim 2, characterized in that: The first enhancement rule is obtained by the following steps: Obtaining a preset data enhancement set, wherein the data enhancement set includes a plurality of different enhancement rules; Random number screening is performed on the data enhancement set to obtain the first enhancement rule.

4. The training method according to claim 2, characterized in that: The acoustic feature representation includes a first audio feature matrix and a second audio feature matrix, and the inputting of the dual-branch feature data set into the instance-level network framework to obtain the acoustic feature representation includes: Inputting the first branch feature data set into the instance-level network framework to obtain the first audio feature matrix; Inputting the second branch feature data set into the instance-level network framework to obtain the second audio feature matrix; The first audio feature matrix includes multiple first audio feature vectors, and the second audio feature matrix includes multiple second audio feature vectors; the number of rows to which each first audio feature vector belongs is the same as the number of rows to which the corresponding second audio feature vector belongs.

5. The training method according to claim 2, characterized in that: The speech emotion classification result includes a first emotion category matrix and a second emotion category matrix, and the dual-branch feature data set is input into the cluster-level network framework to obtain the speech emotion classification result, including: Inputting the first branch feature data set into the cluster-level network framework to obtain the first emotion category matrix; Inputting the second branch feature data set into the cluster-level network framework to obtain the second emotion category matrix; Among them, the first emotion category matrix includes multiple first emotion category feature vectors, and the second emotion category matrix includes multiple second emotion category feature vectors; the first emotion category feature vectors with the same number of columns correspond to the same emotion prediction categories; the second emotion category feature vectors with the same number of columns correspond to the same emotion prediction categories.

6. The training method according to claim 1, characterized in that: The method of updating the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model comprises: Performing row-wise screening on the acoustic feature representation to obtain a first audio row set having a positive example relationship and a second audio row set having a non-positive example relationship; Performing column screening on the speech emotion classification results to obtain a first category column set having a positive example relationship and a second category column set having a non-positive example relationship; Calculating an instance-level loss value by maximizing the cosine similarity of the first audio row set and minimizing the cosine similarity of the second audio row set; By maximizing the cosine similarity of the first category column set and minimizing the cosine similarity of the second category column set, a cluster-level loss value is calculated; Based on the instance-level loss value and the cluster-level loss value, the parameters of the initialized speech emotion model are updated to obtain the trained speech emotion model.

7. A method for recognizing a speech emotion model, characterized in that: The identification method comprises the following steps: Obtain the speech data set to be recognized; The speech data set is input into the speech emotion model trained as described in any one of claims 1-6 to perform emotion recognition to obtain a speech emotion recognition result.

8. A training device for a speech emotion model, characterized in that: The speech emotion model includes an instance-level network framework and a cluster-level network framework, the instance-level network framework and the cluster-level network framework are connected in parallel, and the training device includes: An acquisition module is used to acquire an audio data set of original speech; An extraction module, used for performing dual-branch feature extraction on the audio data set to obtain a dual-branch feature data set; A first processing module, used for inputting the dual-branch feature data set into the instance-level network framework to obtain an acoustic feature representation, wherein the acoustic feature representation is used for recording a plurality of audio features of the original speech at the instance level; A second processing module is used to input the dual-branch feature data set into the cluster-level network framework to obtain a speech emotion classification result, wherein the speech emotion classification result is used to record multiple emotion prediction categories of the original speech at the cluster level; The updating module is used to update the parameters of the initialized speech emotion model according to the acoustic feature representation and the speech emotion classification result to obtain a trained speech emotion model.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Speech emotion recognition method through fusion of feature assessment and multi-layer perceptron

    CN107393525A

  • Speech emotion recognition method and system based on frame-level emotion state alignment

    CN117649861A

  • Emotional state feature extraction method based on deep speech analysis technology

    CN117953921A

  • Speech emotion recognition method and system, storage medium and electronic equipment

    CN118098285A

  • Model, device and method estimating feeling through determination on specific feeling kind group

    JP2023014916A