Speech emotion recognition method and recognition system based on dual adversarial learning

Abstract

The present invention discloses a speech emotion recognition method and recognition system based on dual adversarial learning, which relates to the field of speech signal processing technology, including: obtaining and preprocessing a speech signal, extracting WavLM features from the preprocessed speech signal using a WavLM pre-training model in an emotion classifier; respectively sending the extracted WavLM features to an emotion encoder, an adversarial phoneme classifier, and an adversarial speaker classifier, removing speaker information and content information from the speech signal to be recognized through dual adversarial learning, and obtaining the emotion category of the speech signal to be recognized through the emotion classifier. The present invention removes speaker information and content information from the speech signal by performing adversarial learning on the speaker classifier and the phoneme classifier respectively, thereby extracting features containing only emotion information for speech emotion recognition, effectively improving the accuracy of speech emotion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and more specifically, to a speech emotion recognition method and recognition system based on dual adversarial learning. Background Art

[0002] In recent years, with the rapid development of speech signal processing technology, speech emotion recognition has garnered widespread attention as a key research area in the field of human-computer interaction. However, speech signals typically contain a wealth of speaker and content information, which can interfere with emotion recognition and reduce the model's recognition capabilities. Therefore, an effective technique is needed to extract speech features that contain only emotional information, thereby improving the accuracy of speech emotion recognition.

[0003] Existing speech emotion recognition methods typically directly use speech signal features for emotion classification, lacking effective mechanisms to remove interfering information related to the speaker and content. Therefore, how to remove speaker and content information to achieve higher-precision speech emotion recognition is an urgent problem that needs to be solved. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a speech emotion recognition method and recognition system based on dual adversarial learning. By training an adversarial speaker classifier and an adversarial phoneme classifier, the speaker information and content information in the speech signal are removed, thereby extracting features containing only emotion information for emotion classification.

[0005] The present invention provides a speech emotion recognition method based on dual adversarial learning, comprising the following steps:

[0006] Acquire and preprocess the speech signal, and use the WavLM pre-trained model in the sentiment classifier to extract WavLM features from the preprocessed speech signal;

[0007] The extracted WavLM features are fed into the emotion encoder, adversarial phoneme classifier, and adversarial speaker classifier respectively, and the cross entropy loss of the emotion classifier, adversarial phoneme classifier, and adversarial speaker classifier is calculated respectively;

[0008] Adding the three calculated cross entropy losses to obtain a total loss function, and using the total loss function to simultaneously train the sentiment classifier, the adversarial phoneme classifier, and the adversarial speaker classifier;

[0009] The speech signal to be recognized is respectively imported into the trained emotion classifier, adversarial phoneme classifier and adversarial speaker classifier. The speaker information and content information in the speech signal to be recognized are removed through double adversarial learning, and the emotion category of the speech signal to be recognized is obtained through the emotion classifier.

[0010] In this solution, the speech signal is acquired and preprocessed, and the WavLM pre-trained model in the sentiment classifier is used to extract WavLM features from the preprocessed speech signal. Specifically:

[0011] Obtain a large amount of speech signals with phoneme annotations, emotion annotations, and speaker annotations, perform frequency domain analysis on the speech signals, obtain the frequency band component distribution corresponding to the speech signals, determine the corresponding frequency segments based on the frequency band component distribution, and configure bandpass filtering based on the frequency segments to remove signals that do not meet the frequency requirements;

[0012] The speech signal after bandpass filtering is downsampled, and digitally filtered using a biorthogonal wavelet basis to obtain a denoised speech signal, and the wavelet entropy of the denoised speech signal is calculated. The interval between the maximum wavelet entropy and the minimum wavelet entropy is obtained to generate a threshold interval;

[0013] Using the threshold interval to perform fuzzy speech discrimination on the noise-reduced speech signal, when the wavelet entropy of the speech signal is not within the threshold interval, it is eliminated, and all speech signals are traversed to obtain the preprocessed speech signal;

[0014] Constructing a WavLM pre-trained model. During the training of the WavLM pre-trained model, the convolutional encoder and the Transformer encoder speech signal are used to perform feature encoding, the input speech signal is randomly transformed, a preset proportion of the speech signal is randomly masked, and the label corresponding to the masked position is predicted;

[0015] After training, the WavLM pre-training model is used to extract the probability distribution of the label sequence corresponding to the preprocessed speech signal, and the probability distribution is used as the WavLM feature. The WavLM feature contains emotion information, phoneme information and speaker information.

[0016] In this solution, the sentiment classifier consists of a WavLM pre-trained model, a sentiment encoder, a fully connected layer, and a softmax classification layer;

[0017] The adversarial phoneme classifier consists of a gradient inversion, a phoneme encoder, a fully connected layer and a softmax classification layer;

[0018] The adversarial speaker classifier is composed of gradient reversal, speaker encoder, fully connected layer and softmax classification layer;

[0019] The obtained WavLM features are used as the input of the emotion encoder, adversarial phoneme classifier and adversarial speaker classifier respectively, and the cross entropy losses of the emotion classifier, adversarial factor classifier and adversarial speaker classifier are calculated respectively.

[0020] In this solution, the adversarial phoneme classifier is specifically:

[0021] The acquired WavLM features are imported into the phoneme encoder, where they are initially convolved through one convolutional layer. Then, two convolutional layers are used for downsampling to reduce the feature size. Three identical residual modules are used to extract phoneme features from the downsampled features. A multi-head self-attention mechanism is introduced into the feature extraction to obtain phoneme encoding.

[0022] The obtained phoneme code is imported into the discriminator, and the back propagation is controlled by multiplying the error by a negative coefficient through the gradient reversal layer, so that the network learning objectives before and after the gradient reversal layer are opposite, thus achieving adversarial learning of phoneme features;

[0023] The fully connected layer and Softmax activation function are used to classify and predict the phoneme information in the WavLM feature.

[0024] In this solution, the parameters of the sentiment classifier and the adversarial speaker classifier are configured by sharing features based on the adversarial phoneme classifier, and the WavLM features corresponding to the annotated speech signal are used to simultaneously perform supervised training on the adversarial phoneme classifier, the sentiment classifier, and the adversarial speaker classifier.

[0025] Import the acquired WavLM features into the configured adversarial speaker classifier to perform classification prediction on the speaker information in the WavLM features;

[0026] The content information and speaker information are marked according to the phoneme information label and speaker information label in the WavLM feature, and the marked content information and speaker information are removed.

[0027] In this solution, the cross entropy loss of the sentiment classifier, adversarial factor classifier, and adversarial speaker classifier is calculated separately, specifically:

[0028] The WavLM features corresponding to the annotated speech signals are divided into training and test sets according to the ratio. The framework parameters and learning rates of the emotion classifier, adversarial factor classifier, and adversarial speaker classifier are initialized. The training samples in the training set are input into the three classifiers for training.

[0029] During the training process of the three classifiers, the sentiment cross entropy loss between the output of the sentiment classifier and the sentiment annotation, the phoneme cross entropy loss between the output of the adversarial phoneme classifier and the phoneme annotation, and the speaker cross entropy loss between the output of the adversarial speaker classifier and the speaker annotation are calculated based on the subordinate relationship between the training samples and the label information;

[0030] The emotion cross entropy loss, phoneme cross entropy loss and speaker cross entropy loss are added together to construct a total loss function to supervise the training of the three classifiers. The network parameters of the three classifiers are iteratively updated according to the total loss in the forward propagation. The classification performance is tested using the test set. When the performance test results meet the preset standards, the training of the three classifiers is completed.

[0031] In this solution, a pre-processed speech signal to be recognized is obtained, and the speech signal to be recognized is respectively introduced into a trained emotion classifier, an adversarial phoneme classifier, and an adversarial speaker classifier;

[0032] Through double adversarial learning, the speaker information and content information in the speech signal to be recognized are removed. The probability distribution of the emotion label corresponding to the speech signal to be recognized is obtained through the fully connected layer and Softmax function in the emotion classifier, and the emotion category of the speech signal to be recognized is output according to the probability distribution.

[0033] The second aspect of the present invention provides a speech emotion recognition system based on dual adversarial learning, the system comprising: a speech signal input module, an emotion classifier module, an adversarial phoneme classifier module, an adversarial speaker classifier module, a classifier training module and a speech emotion output module;

[0034] The voice signal input module is responsible for acquiring the voice signal to be recognized and preprocessing the voice signal to be recognized;

[0035] The emotion classifier module is responsible for extracting the WavLM features of the speech signal to be recognized, and obtaining the emotion category of the speech signal to be recognized according to the WavLM features;

[0036] The adversarial phoneme classifier module is responsible for removing the content information of the speech signal to be recognized;

[0037] The adversarial speaker classifier module is responsible for removing speaker information from the speech signal to be recognized;

[0038] The classifier training module is responsible for training the emotion classifier, the adversarial phoneme classifier and the adversarial speaker classifier using the annotated speech signal;

[0039] The speech output module is responsible for outputting the emotion category information corresponding to the speech signal to be recognized without the content information and the speaker information, and displaying it in a preset manner.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention performs adversarial learning on the speaker classifier and the phoneme classifier respectively, removes the speaker information and content information in the speech signal, thereby extracting features containing only emotional information for speech emotion recognition, effectively improving the accuracy of speech emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments or exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or exemplary descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained according to these drawings without paying any creative work.

[0043] Figure 1 A flow chart of a speech emotion recognition method based on dual adversarial learning is shown;

[0044] Figure 2 The process and structural diagram of speech emotion recognition are shown;

[0045] Figure 3 A schematic diagram of the process of classifier training is shown;

[0046] Figure 4 Shown is a block diagram of a speech emotion recognition system based on dual adversarial learning. DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0049] Figure 1 A flowchart of a speech emotion recognition method based on dual adversarial learning is shown.

[0050] like Figure 1 As shown, a first embodiment of the present invention provides a speech emotion recognition method based on dual adversarial learning, comprising:

[0051] S102, obtaining and preprocessing a speech signal, and extracting WavLM features from the preprocessed speech signal using a WavLM pre-trained model in a sentiment classifier;

[0052] S104, sending the extracted WavLM features to the emotion encoder, the adversarial phoneme classifier, and the adversarial speaker classifier respectively, and calculating the cross entropy loss of the emotion classifier, the adversarial phoneme classifier, and the adversarial speaker classifier respectively;

[0053] S106, adding the three calculated cross entropy losses to obtain a total loss function, and using the total loss function to simultaneously train the sentiment classifier, the adversarial phoneme classifier, and the adversarial speaker classifier;

[0054] S108, importing the speech signal to be recognized into the trained emotion classifier, adversarial phoneme classifier and adversarial speaker classifier respectively, removing the speaker information and content information in the speech signal to be recognized through double adversarial learning, and obtaining the emotion category of the speech signal to be recognized through the emotion classifier.

[0055] It should be noted that a large amount of speech signals with phoneme annotations, emotion annotations, and speaker annotations are obtained, and frequency domain analysis is performed on the speech signals to obtain the frequency band component distribution corresponding to the speech signals. The corresponding frequency segments are determined based on the frequency band component distribution, and bandpass filtering is configured according to the frequency segments to remove signals that do not meet the frequency requirements. To reduce the computational complexity of the model, the speech signals after bandpass filtering are downsampled and digitally filtered using a biorthogonal wavelet basis. The translation and scaling of the wavelet function are controlled by setting the translation amount and scale of the wavelet transform to filter out high-frequency noise and DC noise. Due to environmental factors and speaker factors, ambiguous speech may occur. Wavelet entropy can reflect the time and energy spectrum of the signal. Wavelet entropy is used to distinguish ambiguous speech signals, thereby eliminating them and enhancing the quality of the speech signal. The denoised speech signal is obtained, the wavelet entropy of the denoised speech signal is calculated, and the interval between the maximum wavelet entropy and the minimum wavelet entropy is obtained to generate a threshold interval. The calculation formula for the wavelet entropy S(a) is:

[0056] S(a)=-∫P(a,b)log(P(a,b))db

[0057] Wherein, P(a, b) represents the wavelet energy probability distribution, a represents the total time variable, and b represents the effective time variable; the threshold interval is used to perform fuzzy speech discrimination on the noise-reduced speech signal. When the wavelet entropy of the speech signal is not within the threshold interval, it is eliminated, and the preprocessed speech signal is obtained after traversing all speech signals.

[0058] Build a WavLM pre-trained model. The WavLM pre-trained model consists of a convolutional encoder and a Transformer encoder. The convolutional encoder has seven layers, each consisting of a temporal convolution layer, a layer normalization layer, and a GELU activation function layer. In the Transformer encoder, gated relative position encoding is used to incorporate relative position into the attention network calculation to better model local information. The WavLM pre-trained model converts continuous signals into discrete labels using the Kmeans method and models the discrete labels as targets.

[0059] In the training of the WavLM pre-training model, convolutional encoders and Transformer encoders are used to perform feature encoding on speech signals, randomly transform the input speech signal, and then randomly mask a preset proportion of the speech signal, and predict the label corresponding to the masked position, such as mixing the two input speech signals or adding background noise. After that, about 50% of the audio signal is randomly masked, and the label corresponding to the masked position is predicted at the output end. After training, the WavLM pre-training model is used to extract the probability distribution of the label sequence corresponding to the preprocessed speech signal, and the probability distribution is used as the WavLM feature, which contains emotional information, phoneme information and speaker information.

[0060] It should be noted that if Figure 2 As shown in the figure, the emotion classifier consists of a WavLM pre-training model, an emotion encoder, a fully connected layer and a softmax classification layer; the adversarial phoneme classifier consists of a gradient reversal, a phoneme encoder, a fully connected layer and a softmax classification layer; the adversarial speaker classifier consists of a gradient reversal, a speaker encoder, a fully connected layer and a softmax classification layer; the obtained WavLM features are used as the input of the emotion encoder, the adversarial phoneme classifier and the adversarial speaker classifier respectively, and the cross entropy losses of the emotion classifier, the adversarial factor classifier and the adversarial speaker classifier are calculated respectively.

[0061] The acquired WavLM features are fed into a phoneme encoder, where they undergo an initial convolution with one convolutional layer. Downsampling is then performed using two convolutional layers to reduce the feature size. During the downsampling phase, several convolutional layers are used to gradually reduce the size of the input vector for preliminary phoneme feature extraction. Three identical residual modules are then used to extract phoneme features from the downsampled features. Each residual module consists of two convolutional layers and a direct connection. These residual modules are connected in series to further extract phoneme features. A multi-head self-attention mechanism is introduced into feature extraction to model the relative dependencies between elements at different positions in the WavLM feature sequence and extract more salient features. Phoneme codes are then obtained based on the output of the multi-head attention structure. The obtained phoneme codes are fed into the discriminator, where a gradient reversal layer multiplies the error by a negative coefficient to control backpropagation. This results in opposite learning objectives for the networks before and after the gradient reversal layer, enabling adversarial learning of phoneme features. The gradient reversal layer acts as an identity transformation function during forward propagation. During backward propagation, the gradient is multiplied by a negative coefficient to control backpropagation. The features of adversarial learning are input into the fully connected layer, and the output layer is activated using the Softmax activation function to perform classification prediction on the phoneme information in the WavLM features.

[0062] Based on the adversarial phoneme classifier, the parameters of the sentiment classifier and the adversarial speaker classifier are configured by sharing features, and the WavLM features corresponding to the annotated speech signal are used to simultaneously perform supervised training on the adversarial phoneme classifier, the sentiment classifier and the adversarial speaker classifier; the acquired WavLM features are imported into the configured adversarial speaker classifier, and the speaker information in the WavLM features is classified and predicted; the content information and the speaker information are marked according to the phoneme information labels and the speaker information labels in the WavLM features, and the marked content information and speaker information are eliminated.

[0063] like Figure 3 As shown, the cross entropy losses of the sentiment classifier, adversarial factor classifier, and adversarial speaker classifier are calculated respectively. The WavLM features corresponding to the annotated speech signal are divided into a training set and a test set according to the ratio, the framework parameters and learning rates of the sentiment classifier, adversarial factor classifier, and adversarial speaker classifier are initialized, and the training samples in the training set are input into the three classifiers for training; during the training process of the three classifiers, the sentiment cross entropy loss between the output of the sentiment classifier and the sentiment annotation, the phoneme cross entropy loss between the output of the adversarial phoneme classifier and the phoneme annotation, and the speaker cross entropy loss between the output of the adversarial speaker classifier and the speaker annotation are calculated based on the subordinate relationship between the training samples and the label information. The cross entropy loss L e The formal expression is:

[0064]

[0065] Among them, n represents the total number of training samples, x i represents the i-th training sample, y i Represents the label information corresponding to the training sample, θ k represents the encoder shared feature configuration parameters, θ f Represents the parameters of the corresponding classifier e, e is the sentiment classifier, adversarial phoneme classifier and adversarial speaker classifier, P represents the probability distribution representation;

[0066] The emotion cross entropy loss, phoneme cross entropy loss and speaker cross entropy loss are added together to construct a total loss function to supervise the training of the three classifiers. The network parameters of the three classifiers are iteratively updated according to the total loss in the forward propagation. The classification performance is tested using the test set. When the performance test results meet the preset standards, the training of the three classifiers is completed.

[0067] According to an embodiment of the present invention, a preprocessed speech signal to be recognized is obtained, and the speech signal to be recognized is respectively introduced into a trained emotion classifier, an adversarial phoneme classifier, and an adversarial speaker classifier; speaker information and content information in the speech signal to be recognized are removed through double adversarial learning, and the emotion label probability distribution corresponding to the speech signal to be recognized is obtained through the fully connected layer and the Softmax function in the emotion classifier, and the emotion category of the speech signal to be recognized is output according to the probability distribution.

[0068] Figure 4 Shown is a block diagram of a speech emotion recognition system based on dual adversarial learning.

[0069] The second embodiment of the present invention provides a speech emotion recognition system based on dual adversarial learning, which includes: a speech signal input module 401, an emotion classifier module 402, an adversarial phoneme classifier module 403, an adversarial speaker classifier module 404, a classifier training module 405 and a speech emotion output module 406;

[0070] The voice signal input module is responsible for acquiring the voice signal to be recognized and preprocessing the voice signal to be recognized;

[0071] The emotion classifier module is responsible for extracting the WavLM features of the speech signal to be recognized, and obtaining the emotion category of the speech signal to be recognized according to the WavLM features;

[0072] The adversarial phoneme classifier module is responsible for removing the content information of the speech signal to be recognized;

[0073] The adversarial speaker classifier module is responsible for removing speaker information from the speech signal to be recognized;

[0074] The classifier training module is responsible for training the emotion classifier, the adversarial phoneme classifier and the adversarial speaker classifier using the annotated speech signal;

[0075] The speech output module is responsible for outputting the emotion category information corresponding to the speech signal to be recognized without the content information and the speaker information, and displaying it in a preset manner.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed method can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms. In addition, the functional modules in the various embodiments of the present invention can all be integrated into one processing module, or each module can be a separate module, or two or more modules can be integrated into one module; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0077] If the above-mentioned integrated modules of the present invention are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0078] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A speech emotion recognition method based on dual adversarial learning, characterized in that: The following steps are involved: Acquire and preprocess the speech signal, and use the WavLM pre-trained model in the sentiment classifier to extract WavLM features from the preprocessed speech signal; The extracted WavLM features are fed into the emotion encoder, adversarial phoneme classifier, and adversarial speaker classifier respectively, and the cross entropy loss of the emotion classifier, adversarial phoneme classifier, and adversarial speaker classifier is calculated respectively; Adding the three calculated cross entropy losses to obtain a total loss function, and using the total loss function to simultaneously train the sentiment classifier, the adversarial phoneme classifier, and the adversarial speaker classifier; The speech signal to be recognized is respectively introduced into the trained emotion classifier, adversarial phoneme classifier and adversarial speaker classifier. The speaker information and content information in the speech signal to be recognized are removed through double adversarial learning, and the emotion category of the speech signal to be recognized is obtained through the emotion classifier. The sentiment classifier consists of a WavLM pre-trained model, a sentiment encoder, a fully connected layer, and a softmax classification layer; The adversarial phoneme classifier consists of a gradient inversion, a phoneme encoder, a fully connected layer and a softmax classification layer; The adversarial speaker classifier is composed of gradient reversal, speaker encoder, fully connected layer and softmax classification layer; The obtained WavLM features are used as the input of the emotion encoder, adversarial phoneme classifier and adversarial speaker classifier respectively, and the cross entropy losses of the emotion classifier, adversarial factor classifier and adversarial speaker classifier are calculated respectively.

2. The speech emotion recognition method based on dual adversarial learning according to claim 1, characterized in that Acquire and preprocess the speech signal, and use the WavLM pre-trained model in the sentiment classifier to extract WavLM features from the preprocessed speech signal. Specifically: Obtain a large amount of speech signals with phoneme annotations, emotion annotations, and speaker annotations, perform frequency domain analysis on the speech signals, obtain the frequency band component distribution corresponding to the speech signals, determine the corresponding frequency segments based on the frequency band component distribution, and configure bandpass filtering based on the frequency segments to remove signals that do not meet the frequency requirements; The speech signal after bandpass filtering is downsampled, and digitally filtered using a biorthogonal wavelet basis to obtain a denoised speech signal, and the wavelet entropy of the denoised speech signal is calculated. The interval between the maximum wavelet entropy and the minimum wavelet entropy is obtained to generate a threshold interval; Using the threshold interval to perform fuzzy speech discrimination on the noise-reduced speech signal, when the wavelet entropy of the speech signal is not within the threshold interval, it is eliminated, and all speech signals are traversed to obtain the preprocessed speech signal; Constructing a WavLM pre-trained model. During the training of the WavLM pre-trained model, the convolutional encoder and the Transformer encoder speech signal are used to perform feature encoding, the input speech signal is randomly transformed, a preset proportion of the speech signal is randomly masked, and the label corresponding to the masked position is predicted; After training, the WavLM pre-training model is used to extract the probability distribution of the label sequence corresponding to the preprocessed speech signal, and the probability distribution is used as the WavLM feature. The WavLM feature contains emotion information, phoneme information and speaker information.

3. The speech emotion recognition method based on dual adversarial learning according to claim 1, characterized in that The adversarial phoneme classifier is specifically: The acquired WavLM features are imported into the phoneme encoder, where they are initially convolved through one convolutional layer. Then, two convolutional layers are used for downsampling to reduce the feature size. Three identical residual modules are used to extract phoneme features from the downsampled features. A multi-head self-attention mechanism is introduced into the feature extraction to obtain phoneme encoding. The obtained phoneme code is imported into the discriminator, and the back propagation is controlled by multiplying the error by a negative coefficient through the gradient reversal layer, so that the network learning objectives before and after the gradient reversal layer are opposite, thus achieving adversarial learning of phoneme features; The fully connected layer and Softmax activation function are used to classify and predict the phoneme information in the WavLM feature.

4. The speech emotion recognition method based on dual adversarial learning according to claim 1, characterized in that Based on the adversarial phoneme classifier, parameters of the sentiment classifier and the adversarial speaker classifier are configured by sharing features, and the adversarial phoneme classifier, the sentiment classifier and the adversarial speaker classifier are simultaneously supervised trained using WavLM features corresponding to the annotated speech signal; Import the acquired WavLM features into the configured adversarial speaker classifier to perform classification prediction on the speaker information in the WavLM features; The content information and speaker information are marked according to the phoneme information label and speaker information label in the WavLM feature, and the marked content information and speaker information are removed.

5. The speech emotion recognition method based on dual adversarial learning according to claim 1, characterized in that The cross entropy losses of the sentiment classifier, adversarial factor classifier, and adversarial speaker classifier are calculated separately, specifically: The WavLM features corresponding to the annotated speech signals are divided into training and test sets according to the ratio. The framework parameters and learning rates of the emotion classifier, adversarial factor classifier, and adversarial speaker classifier are initialized. The training samples in the training set are input into the three classifiers for training. During the training process of the three classifiers, the sentiment cross entropy loss between the output of the sentiment classifier and the sentiment annotation, the phoneme cross entropy loss between the output of the adversarial phoneme classifier and the phoneme annotation, and the speaker cross entropy loss between the output of the adversarial speaker classifier and the speaker annotation are calculated based on the subordinate relationship between the training samples and the label information; The emotion cross entropy loss, phoneme cross entropy loss and speaker cross entropy loss are added together to construct a total loss function to supervise the training of the three classifiers. The network parameters of the three classifiers are iteratively updated according to the total loss in the forward propagation. The classification performance is tested using the test set. When the performance test results meet the preset standards, the training of the three classifiers is completed.

6. The speech emotion recognition method based on dual adversarial learning according to claim 1, characterized in that Obtaining a preprocessed speech signal to be recognized, and importing the speech signal to be recognized into a trained emotion classifier, an adversarial phoneme classifier, and an adversarial speaker classifier respectively; Through double adversarial learning, the speaker information and content information in the speech signal to be recognized are removed. The probability distribution of the emotion label corresponding to the speech signal to be recognized is obtained through the fully connected layer and Softmax function in the emotion classifier, and the emotion category of the speech signal to be recognized is output according to the probability distribution.

7. A speech emotion recognition system based on dual adversarial learning, characterized in that: Implementing the speech emotion recognition method based on dual adversarial learning as described in any one of claims 1 to 6, the system comprises: a speech signal input module, an emotion classifier module, an adversarial phoneme classifier module, an adversarial speaker classifier module, a classifier training module and a speech emotion output module; The voice signal input module is responsible for acquiring the voice signal to be recognized and preprocessing the voice signal to be recognized; The emotion classifier module is responsible for extracting the WavLM features of the speech signal to be recognized, and obtaining the emotion category of the speech signal to be recognized according to the WavLM features; The adversarial phoneme classifier module is responsible for removing the content information of the speech signal to be recognized; The adversarial speaker classifier module is responsible for removing speaker information from the speech signal to be recognized; The classifier training module is responsible for training the emotion classifier, the adversarial phoneme classifier and the adversarial speaker classifier using the annotated speech signal; The speech output module is responsible for outputting the emotion category information corresponding to the speech signal to be recognized without the content information and the speaker information, and displaying it in a preset manner.

Images