A sentiment asymmetric speaker recognition system based on a generative adversarial network

Abstract

The application provides a sentiment asymmetry speaker recognition system based on a generative adversarial network, and a use process of the system comprises the following steps: firstly, converting the voice corresponding to a plurality of sentiment texts into pre-learned multi-sentiment spectrograms through pre-emphasis, framing and windowing and fast Fourier transform; secondly, training the generative adversarial network using the pre-learned multi-sentiment spectrograms, and supervising the conversion between the spectrograms of neutrality and other sentiments in the training process of the generative adversarial network through overall loss; then, converting the spectrogram of neutrality of a registered user into other sentiment spectrograms using the generative adversarial network, and obtaining the user registration multi-sentiment spectrogram by combining the spectrogram of neutrality and other sentiment of the registered user; finally, training a speaker recognition network using the user registration multi-sentiment spectrogram, calculating the speaker classification probability of the voice to be detected, and obtaining the final voiceprint recognition result. The system solves the problem of the decline of the performance of speaker recognition caused by the inconsistency between the registration and the voice sentiment in the actual application scene.

Description

Technical Field

[0001] This invention relates to the field of speaker recognition technology, and more specifically, to a speaker recognition system based on deep learning and adversarial generative networks under emotion asymmetry conditions. Background Technology

[0002] In recent years, with the continuous development of science and technology, speaker recognition (SR) technology has also achieved remarkable results. Technologies such as mobile phone voiceprint unlocking and voice authentication payment have greatly facilitated people's lives. However, many factors still affect the performance of speaker recognition technology. Emotion is one of the key factors. The speech distribution of the same speaker in different emotional states is often inconsistent, and these differences can greatly affect the performance of speaker recognition technology. Many existing recognition methods do not take into account the impact of emotion, which has certain disadvantages.

[0003] Emotional speaker recognition (ESR) primarily studies the impact of different emotional states on speaker systems. In ESR, emotional speech mismatch is a significant reason for performance degradation, mainly manifested in the inconsistency between the emotional distribution of training and testing speech. Specifically, it's difficult for a person to maintain a consistently positive or negative emotional state. For instance, when users register for voiceprint recognition applications, they typically send neutral speech data, making it difficult to obtain non-neutral emotional data. However, emotional speech inevitably occurs during product use. Therefore, the network's training data is predominantly neutral speech, with far fewer emotional categories than the testing data. Existing Emotional Speaker Recognition research often employs methods like emotional compensation to address the lack of emotional data. This is achieved by adjusting scores for non-neutral emotional speech or through structured training. However, these methods still have drawbacks; the network still requires the target speaker's emotional speech, which significantly limits the recognition system.

[0004] Generative Adversarial Networks (GANs) are currently popular data generation networks, widely used in data generation, style transfer, and image enhancement. This system applies GANs to the task of emotional speaker recognition, generating richer and more comprehensive emotional spectrograms of registered users. This addresses the performance degradation issue caused by a lack of emotional corpus during training, leading to inconsistencies in the emotional content of registered and test speech. Compared to methods like emotion compensation, GANs can generate additional emotional speech data for registered users based on only available neutral speech. This method significantly enriches the emotional training corpus and improves the overall recognition performance of the network. Our proposed speaker emotion transfer adversarial learning system adds a new loss term and introduces a new discriminator to the GAN to ensure the preservation of individual information and the transfer of emotional information during data generation, resulting in more realistic emotional corpus data and improved speaker recognition performance. Summary of the Invention

[0005] This invention provides an emotion-asymmetric speaker recognition system based on generative adversarial networks (GANs). In practical applications, voiceprint recognition systems often register only the user's neutral speech, but need to recognize the voiceprint information of the user's speech under different emotional conditions, leading to poor recognition performance. To solve this problem, this invention proposes using a generative adversarial network to learn a speaker emotion-transformation GAN that combines emotion transformation and voiceprint information preservation. This generates other emotion spectrograms corresponding to the registered user's neutral spectrogram, expanding the sample size of the registered user's spectrogram, thereby training the speaker recognition network and improving the final speaker recognition performance.

[0006] The user process of the emotion-asymmetric speaker recognition system based on generative adversarial networks provided by this invention includes the following steps: First, the speech corresponding to various emotional texts is extracted into pre-learned multi-emotion spectrograms through pre-emphasis, frame-segmentation, windowing, and Fast Fourier Transform. The emotional categories of the speech corresponding to the various emotional texts are neutral, angry, excited, panicked, and sad. Second, the pre-learned multi-emotion spectrograms are used to train a generative adversarial network (GAN) to transform four neutral spectrograms into four target emotional spectrograms. The mutual transformation between neutral and other emotional spectrograms is supervised by the overall loss during GAN training. Next, neutral speech registered by users in the system is extracted into neutral spectrograms through pre-emphasis, frame-segmentation, windowing, and Fast Fourier Transform. Then, the obtained user-registered neutral spectrograms are sequentially input into the four neutral spectrograms to transform them into GANs for the other four target emotions, generating four target emotional spectrograms respectively. These four target emotional spectrograms are then merged with the user-registered neutral spectrograms to expand the user-registered speech data, resulting in a user-registered multi-emotion spectrogram. Finally, the obtained user registration multi-emotion spectrogram is used to train the speaker recognition network, and the speaker classification probability is calculated for the speech to be detected to obtain the final voiceprint recognition result. The technical solution adopted in this invention can be further refined. The overall loss L mentioned above... all Divided into cycle consistency L cycle Counteracting loss L(G,D) Yi ) and L(F,D X Speaker identity consistency loss L identity With sentiment classification loss L emotion These are used to supervise the learning processes of generator style transfer, adversarial learning, speaker identity preservation, and emotion transfer, respectively. The overall loss function L... all This can be summarized as follows: L all =L(G,D Yi )+L(F,D X )+αL cycle +βL identity +γL emotion (1) Where α is the weight coefficient of the cycle consistency loss term, β is the weight coefficient of the adversarial loss term, and γ are the weight coefficients of the sentiment classification loss term, which control the correlation between their respective loss terms and the overall task. G is a generator that generates other sentiment spectrograms based on neutral spectrograms, and D... Yi To identify the authenticity of the spectrogram generated by G with the i-th target emotion among the four target emotions of anger, excitement, panic, and sadness, F is a generator that generates a neutral spectrogram based on the target emotion spectrogram, and D is a generator that generates a neutral spectrogram based on the target emotion spectrogram. X A discriminator for identifying the authenticity of neutral spectrograms generated by F. Adversarial generative networks introduce cycle consistency loss L cycleThe learning process of converting a neutral sentiment spectrogram to a target sentiment spectrogram and then back to a neutral sentiment spectrogram was supervised. Additionally, the learning process of converting a target sentiment spectrogram to a neutral sentiment spectrogram and then back to a target sentiment spectrogram was also supervised. cycle Details of the loss term are shown in the formula: Where E represents mathematical expectation, x is a neutral spectrogram, and y i Let p(x) be the spectrogram of the i-th target emotion among anger, excitement, panic, and sadness, and p(y) be the probability distribution of the neutral spectrogram. i Let y be the probability distribution of the i-th type of target sentiment spectrogram. The adversarial generative network, supervised by this loss term, transforms the neutral sentiment spectrogram x into a generated target sentiment spectrogram G(x) through generator G, and then transforms it into a spectrogram F(G(x)) with neutral sentiment through generator F. The i-th type of target sentiment spectrogram y... i Transformed by generator F into a spectrogram F(y) with neutral sentiment. i After that, it is converted into a spectrogram G(F(y) with other emotions by the generator G. i )). Generative adversarial networks introduce adversarial loss terms L(G,D) Yi ) and L(F,D X The supervised generator and discriminator adversarial training process, L(G,D) Yi ) and L(F,D X The details of the loss item are as follows: Where X represents the neutral affective domain, which contains several neutral verbal spectra x and Y. i Represents the i-th target sentiment domain, which contains several i-th target sentiment spectrograms y. i The loss term supervises the generators F and G and the discriminator Dx and D. Yi The adversarial training process iteratively improves the realism of the generated images by the generator and the discriminator's ability to distinguish between real and synthetic images, thereby generating spectrogram samples that conform to the distribution of real data. Generative adversarial networks introduce this loss term L identity Voiceprint identity is preserved during the emotion transformation process. The adversarial generative network computes the neutral spectrogram x, the generated target emotion spectrogram G(x), and the i-th type of target emotion spectrogram y. i With the generated neutral spectrogram F(y) i The difference in loss term L. identity Defined as: In the above formula, x, yi They respectively conform to the neutral spectrogram distribution p(x) and the i-th target sentiment spectrogram distribution p(y). i ). Introducing an Emotion Discriminator D in Generative Adversarial Networks emotionyi With D emotionx This method identifies the sentiment categories of the spectrograms generated by generators G and F, respectively, and supervises the sentiment identification process using a sentiment classification loss. The loss term L used to supervise the training of these sentiment discriminators is... emotion The definition is as follows: Where D emotionx D is a neutral sentiment discriminator. emotionyi For the i-th target sentiment discriminator, label x The label is a neutral sentiment label. yi Let i be the target sentiment label.

[0007] The beneficial effects of this invention are: compared with existing speaker recognition methods, the proposed adversarial generative network can make full use of the neutral spectrogram of the target speaker to generate high-quality emotional spectrograms of anger, excitement, panic and sadness while retaining the original speaker identity information, greatly enriching the emotional categories of the training data, effectively maintaining the balance of emotional categories between training and testing data, and solving the problem that the inconsistency of speech distribution under different emotional states of the same speaker will greatly reduce the robustness of the network. Attached Figure Description

[0008] Figure 1 This is a flowchart of an emotion-asymmetric speaker recognition system based on generative adversarial networks. Detailed Implementation

[0009] To more clearly describe the content of this invention, further explanation is provided below with reference to examples and accompanying drawings. The embodiments described below are not intended to limit the scope of this invention. The speaker recognition method based on adversarial generative networks for emotion asymmetry conditions of this invention includes the following steps:

[0010] Step 1: Extract spectrograms from all speech data through pre-emphasis, frame segmentation, windowing, and Fast Fourier Transform. The spectrogram extraction process is illustrated below using a single speech data point x as an example.

[0011] 1.1 Pre-emphasis is achieved by using a series of high-pass filters to enhance the high-frequency components of the speech signal: H(x) = 1 - μ*x -1 (7)

[0012] Where μ represents the pre-emphasis coefficient, which is usually taken as 0.97, and H(x) is the pre-emphasis result.

[0013] 1.2 Utilizing the short-time smoothing characteristics of the signal, the pre-emphasis result is processed into frames. The H(x) obtained by pre-emphasis of a segment of speech data with a sampling rate of fkHz can be processed into a matrix of size M*N, H = (H1, H2...H...). M ) T Here, T represents the transpose of the matrix. The dimension of a matrix is ​​calculated as follows: N = f*t (9)

[0014] In the above formula, M represents the number of frames in the speech, N represents the length of each frame of data, the frame size is t, and the frame sliding size is s.

[0015] 1.3 A windowing operation is performed on the matrix H(x) obtained in 1.2, and the edges of the i-th frame signal are smoothed using a Hamming window to obtain ω[H]. i ]:

[0016] 1.4 FFT is used to process ω[H] i The one-dimensional frequency signal of the i-th frame is obtained:

[0017] In the above formula, FFT stands for Fast Fourier Transform, and P i The i-th frame is a one-dimensional frequency signal. A complete two-dimensional spectrogram can be obtained by splicing the one-dimensional frequency signals of M frames.

[0018] Step 2: Train the adversarial generative network using a pre-learned multi-emotion spectrogram with five emotions: neutral, angry, excited, panicked and sad. Train the parameters of the four adversarial generative networks according to the transition process between neutral and the other four emotions.

[0019] 2.1 After determining that the target sentiment of the adversarial generative network is to be the i-th target sentiment, the sentiment labels of the neutral spectrogram and the i-th target sentiment spectrogram in the pre-learned multi-sentiment spectrogram are respectively set as labels. x With label yi The adversarial generative network is trained for 2000 rounds. In each round, a fixed number of neutral spectrograms x and the i-th target sentiment spectrogram y are randomly selected from the pre-learned multi-sentiment spectrograms. i Input network training.

[0020] 2.2 Combine the neutral spectrogram x from 2.1 with the i-th target sentiment spectrogram y. i Input generators G and F respectively, and use labels respectively. yi With label xData is generated for the target sentiment label to obtain the generated target sentiment spectrogram G(x) and the neutral sentiment spectrogram F(y). i The generation process is subject to cycle consistency loss L. cycle Loss of speaker identity L identity The limitations, cyclical loss, and identity loss are as follows:

[0021] 2.3 Using the neutral spectrogram x from 2.1 and the i-th target sentiment spectrogram y i Training the authenticity discriminator D X With D Yi To enhance the ability to distinguish the authenticity of neutral spectrograms and the i-th target sentiment spectrogram, the sentiment discriminator D is then retrained. emotionx D emotionyi The ability to distinguish sentiment categories between neutral spectrograms and the i-th target sentiment spectrogram. The F(y) generated in section 2.2... i Input to the authenticity detector D X Emotion discriminator D emotionx The generated sample G(x) is then input into the authenticity discriminator D for identification. Yi The i-th target emotion discriminator D emotionyi The discrimination is performed during the process. Discriminator D X With D Yi The true or false sample can be determined by binary classification, and the discriminator D emotionx D emotionyi Sentiment classification attributes of samples are obtained through multi-classification results. This process is subject to adversarial loss L(G,D). Yi ) and L(F,D X ) and emotional loss L emotion Limitation, counteracting loss L(G,D) Yi ) and L(F,D X ) and emotional loss L emotion Details are as follows:

[0022] 2.4 Set different target emotions and repeat steps 2.1 to 2.3 until four adversarial generative models are obtained based on neutral spectrograms to generate spectrograms of four target emotions: anger, excitement, panic, and sadness.

[0023] Step 3: Using the four adversarial generative networks trained in Step 2, generate spectrograms of four target sentiment styles based on the neutral spectrogram of user registration in the actual application scenario, thus expanding the neutral spectrogram of user registration into a multi-sentiment spectrogram of user registration.

[0024] 3.1 Input the neutral spectrogram of user registration in the actual application scenario into the four adversarial generative networks trained in step 2, and use the generator G to obtain the four target sentiment spectrograms G(x) corresponding to each neutral spectrogram.

[0025] 3.2 Collect four target sentiment spectrograms G(x) generated by four networks, merge them with the user registration neutral spectrogram, and expand the user registration neutral spectrogram into a user registration multi-sentiment spectrogram.

[0026] Step 4: Train the speaker recognition network based on the user's multi-emotion spectrogram obtained in Step 3, and use the network to perform speaker recognition on the speech to be detected.

[0027] 4.1 Input the user's multi-sentiment spectrogram into the CNN network, extract deep abstract features through convolutional layers, calculate the classification probability p of each speaker label class through fully connected layers, take the speaker label with the largest p as the classification result, calculate the difference between the classification result and the real speaker label using cross-entropy loss, reduce the error using gradient backpropagation, and finally obtain the speaker recognition network parameters.

[0028] 4.2 Extract the speech to be detected into a spectrogram to be detected. Use the speaker recognition network parameters trained in step 4.1 to recognize the spectrogram to be detected. Use a fully connected layer to calculate the classification probability p of each speaker label class. Take the speaker label corresponding to the maximum value of p as the final classification result. Experimental Design

[0029] Experimental Dataset and Partitioning: The dataset used in the experiment is the Mandarin Affective Speech Corpus (MASC), which contains speech data from 68 speakers across five emotion categories: neutral, angry, happy, panicked, and sad. Each speaker's speech data contains 60 sentences for each emotion, and each speech data is labeled with the speaker's ID. There are 45 males and 23 females, and the participants' ages range from 20 to 40 years old. The data from the first 50 speakers was used to train the adversarial generative network (AGN) data generation module for neutral speech against the other four emotions. The remaining 18 speakers' neutral speech data (top 70%) was used for generating emotional speech data, resulting in spectrograms corresponding to the neutral speech of the 18 target speakers for anger, happiness, panic, and sadness. In the SR speaker recognition module, a classification and recognition task will be performed on the remaining 18 people. In the experiment, the first 70% of the neutral speech data of the 18 people was mixed with the generated data to form the training set, which corresponds to the situation in real-world scenarios where only the neutral speech data of the target speaker is available. Then, an adversarial generative network was used to amplify the emotion categories and data volume of the training data. Finally, the last 30% of the fully emotional speech of the 18 people was used for testing, to correspond to the situation in real-world voiceprint recognition scenarios where various emotions may appear. Experimental results

[0030] Experimental Results under Different Speaker Recognition Networks: The adversarial generative network proposed in this invention was compared with other CNN-based speaker recognition networks, embedding methods, and adversarial generative network-based speaker recognition networks. The results are shown in Table 1. Among the comparison methods, the CNN-based speaker recognition networks included VGG, ResNet18, and Densent121, with embedding methods X-vector and ECAPA-TDNN. The adversarial generative network used StarGAN and EC-StarGAN to generate emotional speaker data during the data generation stage, and used a VGG-based speaker recognition network for performance verification during the speaker recognition stage. The adversarial generative network proposed in this invention significantly improves the speaker recognition results for angry, happy, panicked, and sad speakers by combining emotion generation and speaker identity preservation. Due to emotion mismatch, the recognition rate of the state-of-the-art embedding method ECAPA-TDNN drops significantly, thus verifying that the performance of speaker recognition networks is easily affected by emotional conditions. Among the CNN-based speaker recognition methods, VGG outperforms methods such as DenseNet121 and ResNet18. Comparison with other GAN methods such as StarGAN and EC-StarGAN shows that the proposed adversarial generative network achieves the best recognition rate for all non-neutral emotions except sadness, and has the second-highest recognition accuracy for sadness. When the test speech emotion is neutral, the recognition rates of VGG, CycleGAN, and the proposed adversarial generative network all exceed 97%. Compared with the baseline VGG speaker recognition network, the proposed adversarial generative network improves the recognition rate by approximately 8.27%. This improvement demonstrates the enhanced speaker recognition capability of the proposed adversarial generative network under different emotional conditions, reducing the negative impact of emotion on speaker recognition. Table 1. Speaker recognition accuracy (ACC) of Generative Adversarial Network and various contrast algorithms on MASC.

Claims

1. An emotion-asymmetric speaker recognition system based on generative adversarial networks, the usage process of which includes the following steps: S1: The speech corresponding to multiple emotional texts is extracted into a pre-learned multi-emotion spectrogram through pre-emphasis, frame-by-frame windowing and fast Fourier transform. The emotional categories of the multiple emotional speech are neutral, angry, excited, panicked and sad. S2: Using the pre-learned multi-sentiment spectrograms obtained in S1, a generative adversarial network (GAN) is trained to transform four neutral spectrograms into four target sentiment spectrograms. The mutual transformation between neutral spectrograms and the other sentiment spectrograms is supervised during GAN training using an overall loss, where the overall loss L... all Divided into cycle consistency loss L cycle Counteracting loss L(G,D) Yi ) and L(F,D X Speaker identity consistency loss L identity With sentiment classification loss L emotion These are used to supervise the learning process of generator style transfer, adversarial learning, speaker identity preservation, and emotion transfer, respectively, with the overall loss function L. all This can be summarized as follows: in, The weighting coefficients for the cycle consistency loss term. To counteract the weighting coefficients of the loss term, , , are the weight coefficients of the sentiment classification loss term, which control the correlation between their respective loss terms and the overall task; G is the generator that generates other sentiment spectrograms based on the neutral spectrogram; D... Yi To identify the authenticity of the spectrogram generated by G for the i-th target emotion among four target emotions (anger, excitement, panic, and sadness), F is a generator that generates a neutral spectrogram based on the target emotion spectrogram, and D... X A discriminator for identifying the authenticity of neutral spectrograms generated by F; S3: Extract neutral speech from user registrations in the system into neutral speech spectrograms through pre-emphasis, frame-by-frame windowing, and fast Fourier transform; S4: Input the neutral spectrogram of user registration obtained in step S3 into the four neutral spectrograms trained in step S2 to transform them into generative adversarial networks for the other four target emotions, generate four target emotion spectrograms respectively, and merge the four target emotion spectrograms with the neutral spectrogram of user registration to expand the user registration speech data to obtain the multi-emotion spectrogram of user registration. S5: Train the speaker recognition network based on the user registration multi-emotion spectrogram obtained in step S4, calculate the speaker classification probability of the speech to be detected, and obtain the final voiceprint recognition result.

2. The emotion-asymmetric speaker recognition system based on generative adversarial networks according to claim 1, characterized in that, Adversarial generative networks introduce cycle consistency loss L cycle The learning process of converting a neutral sentiment spectrogram to a target sentiment spectrogram and then back to a neutral sentiment spectrogram is supervised. Furthermore, the learning process of converting a target sentiment spectrogram to a neutral sentiment spectrogram and then back to a target sentiment spectrogram is also supervised. The cycle consistency loss L... cycle The loss term is shown in the formula: Where E represents mathematical expectation, x is a neutral spectrogram, and y i Let p(x) be the spectrogram of the i-th target emotion among anger, excitement, panic, and sadness, and p(y) be the probability distribution of the neutral spectrogram. i Let y be the probability distribution of the i-th type of target sentiment spectrogram. The adversarial generative network, supervised by this loss term, transforms the neutral sentiment spectrogram x into a generated target sentiment spectrogram G(x) through generator G, and then into a spectrogram F(G(x)) with neutral sentiment through generator F. The i-th type of target sentiment spectrogram y... i Transformed into a neutral sentiment spectrogram F(y) by generator F. i After that, it is converted into a spectrogram G(F(y) with other emotions by the generator G. i )).

3. The emotion-asymmetric speaker recognition system based on generative adversarial networks according to claim 2, characterized in that, The adversarial training process between the generator and discriminator is supervised, and the adversarial loss term L(G,D) is mentioned. Yi ) and L(F,D X The details of the loss item are as follows: Where X represents the neutral affective domain, which contains several neutral verbal spectra x and Y. i Represents the i-th target sentiment domain, which contains several i-th target sentiment spectrograms y. i The loss term supervises the generators F and G and the discriminator Dx and D. Yi The adversarial training process iteratively improves the realism of the generated images by the generator and the discriminator's ability to distinguish between real and synthetic images, thereby generating spectrogram samples that conform to the distribution of real data.

4. The emotion-asymmetric speaker recognition system based on generative adversarial networks according to claim 2, characterized in that, Adversarial generative networks introduce speaker identity consistency loss L identity Voiceprint identity is preserved during the emotion transformation process. The adversarial generative network computes the neutral spectrogram x and the generated target emotion spectrogram G(x) and the i-th type of target emotion spectrogram y. i With the generated neutral spectrogram F(y) i The difference in identity, the loss of L mentioned above. identity Defined as: In the above formula, x, y i They respectively conform to the neutral spectrogram distribution p(x) and the i-th target sentiment spectrogram distribution p(y). i ).

5. The emotion-asymmetric speaker recognition system based on generative adversarial networks according to claim 2, characterized in that, Introducing an Emotion Discriminator D in Generative Adversarial Networks emotionyi With D emotionx To identify the sentiment categories of the spectrograms generated by generators G and F respectively, and to supervise the sentiment identification process using sentiment classification loss, the sentiment classification loss L, which is used to supervise the training of the aforementioned sentiment discriminators, is used as the training loss term. emotion The definition is as follows: Where D emotionx D is a neutral sentiment discriminator. emotionyi For the i-th target sentiment discriminator, label x The label is a neutral sentiment label. yi Let i be the target sentiment label.

Images