Speaker anonymization method based on emotion migration and feature interpolation

Abstract

The invention discloses a speaker anonymization method based on emotion migration and feature interpolation, which comprises a feature extraction stage, a feature interpolation stage and a speech synthesis stage, and comprises the following steps: firstly, extracting speech features from an original speech signal by adopting a WavLM model; the feature representation captures a plurality of attributes in the input data, such as voice and text features in the voice signal; and then, in an emotion feature extraction stage, a wav2vec2-large-robust-12-ft-emotional-msp-dim model is adopted to extract an emotion feature. The method comprises the following steps of: extracting the emotion feature by using a wav2vec2-large-robust-12-ft model; in the feature interpolation stage, features representing speaker information are recombined. And finally, in a speech synthesis stage, anonymized speech keeping emotion information is synthesized by using the modified HIFI-GAN model. According to the method, the capturing capability of the emotion information is improved by utilizing the self-supervised features, so that the emotion features can be better maintained while the voice content is not damaged in the anonymization process, and the high-quality speaker anonymization process is realized.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and in particular relates to a speaker anonymization method based on emotion transfer and feature interpolation. Background Art

[0002] Speaker anonymization is a technology designed to protect the privacy of speakers. It is mainly used in the process of voice data processing and transmission. With the rapid development of speech technologies such as speech recognition and speech synthesis, more and more voice data is collected, stored and analyzed. Voice data contains a lot of sensitive information, including but not limited to the speaker's identity, emotions, health status, etc. If this information is obtained by criminals, it may be used for malicious purposes, such as identity theft, social engineering attacks, etc. Therefore, how to protect the privacy of speakers while ensuring the availability of voice data has become an important task.

[0003] The main goal of speaker anonymization technology is to change or hide the speaker's identity information in the voice signal while keeping the voice content as natural and understandable as possible. The application of this technology can not only protect user privacy, but also provide security in fields such as medical care, justice and smart assistants.

[0004] Current speaker anonymization methods are mainly divided into two categories: speech conversion-based methods and speech synthesis-based methods. Speech conversion-based methods convert input speech into the voice of a different speaker, thereby hiding the identity of the original speaker. Speech synthesis-based methods synthesize speech based on input text, completely replacing the original sound.

[0005] Although these technologies have achieved certain results in experiments, there are still some challenges. First, after anonymization, the naturalness and comprehensibility of speech signals are often affected. Second, the robustness of existing anonymization technologies in the face of multiple languages, multiple sound ranges, and complex environments still needs to be improved. Finally, how to ensure the effectiveness of speech data in specific tasks (such as sentiment analysis and speech recognition) while anonymizing is also a problem that needs to be solved urgently.

[0006] Therefore, it is of great practical significance to research and develop more efficient and robust speaker anonymization technology, which can not only effectively protect user privacy, but also promote the safe application of voice technology in more fields. Summary of the invention

[0007] The technical problem to be solved by the present invention is that after the speaker anonymization system in the prior art anonymizes the speech, the intelligibility and emotional information of the speech signal will be affected. The present invention provides a speaker anonymization method based on emotion transfer and feature interpolation, which can better maintain the emotional state and language content of the speech.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: a speaker anonymization method based on emotion transfer and feature interpolation, comprising training and conversion stages, wherein the training stage comprises the following steps:

[0009] (1.1) constructing a training corpus, wherein the training corpus consists of speech from multiple speakers, and the corpus of the speakers contains different emotional states;

[0010] (1.2) Traverse each audio file in the dataset, use the WavLM model to extract self-supervised features, build matching and synthesis pools, and calculate the matching results;

[0011] (1.3) Use the wav2vec2-large-robust-12-ft-emotion-msp-dim model to extract the emotional feature vector of the speech;

[0012] (1.4) Construct a vocoder network based on HIFI-GAN;

[0013] (1.5) Input the speaker's pre-matching features and emotional features into the vocoder network for training;

[0014] The conversion phase includes the following steps:

[0015] (2.1) Use the pre-trained WavLM model as a feature extractor to extract language content and speaker information;

[0016] (2.2) Construct an anonymous speaker pool by generating virtual speakers and selecting and interpolating features according to specific rules;

[0017] (2.3) The features extracted in step (2.1) are anonymized through the anonymity pool constructed in step (2.2) and converted into pseudo speaker features.

[0018] (2.4) Use the wav2vec2-large-robust-12-ft-emotion-msp-dim model to extract the emotional feature vector of the speech;

[0019] (2.5) The feature vectors generated in steps (2.3) and (2.4) are fed into the trained model for speech synthesis to obtain the anonymized speaker's speech.

[0020] Further, including:

[0021] In step (1.2), we use the features extracted from the 6th layer of WavLM-Large, which generates a vector for every 20ms of 16kHz audio. According to the speaker of the current audio file, a matching pool and a synthesis pool are constructed from other audio files of the same speaker. And the fast cosine distance calculation method is used to evaluate the similarity between the current audio features and the features in the matching pool. Finally, the best matching features are selected and synthesized into the final feature representation.

[0022] Further, including:

[0023] In step (1.3), the wav2vec2-large-robust-12-ft-emotion-msp-dim model is a variant of the Wav2Vec2.0 framework, which is pre-trained with a large amount of unlabeled speech data and fine-tuned on the labeled data to improve the performance of the emotion recognition task. The emotion features are extracted through the pre-trained model, and the extraction process includes the following steps:

[0024] (1) Construct a regression head to map features to sentiment labels. The regression head successfully transforms the features extracted by the Wav2Vec 2.0 model into sentiment labels through a series of linear transformations, nonlinear activations, and Dropout layers.

[0025] (2) Build the emotion recognition model, load the pre-trained Wav2Vec 2.0 model and add the regression head.

[0026] (3) The input audio signal is inferred through the model to generate emotional feature embedding.

[0027] Further, including:

[0028] In step (1.5), the HIFIGAN-V1 vocoder is used for training, and self-supervised features and emotion features are used as input.

[0029] Further, including:

[0030] Step (2.2) The construction of the anonymous pool and the generation of pseudo speakers are key steps in the anonymization process. The construction process includes the following steps:

[0031] (1) Randomly select N speakers from the corpus, and use the WavLM model to extract features from multiple audios of the speakers, and then concatenate the individual audio features together to form reference features in the conversion process.

[0032] (2) Randomly select m speakers from the reference anonymous pool to provide speech embeddings, where

[0033] (3) For each speaker, use the k-nearest neighbor (KNN) algorithm to obtain the nearest representation from the target reference anonymous pool. This step can be described as:

[0034]

[0035] KNN(x,Y,k) means finding the k vectors closest to vector x in set Y. T represents the source frame length of speaker spk.

[0036] (7) After obtaining the nearest representation of each speaker, we need to perform feature interpolation on the vectors of these target speakers to generate pseudo speakers. Assume that the randomly generated speaker weight vector w = (w1, w2, ..., w k ), the calculation formula of the weight vector is:

[0037]

[0038] Where spk represents the number of speakers randomly selected from the reference anonymity pool, and N(0,1) represents the weight vector w between 0 and 1.

[0039] (8) The sum of weights is constrained to 1 through the softmax function.

[0040] (9) Finally, the target pseudo speaker representation D is obtained by weighted summing the representations of each speaker, and its calculation formula is:

[0041] D=∑w spk D spk

[0042] Further, including:

[0043] In the speech synthesis stage, in addition to utilizing the fused input of speech and emotion features, we also adjusted the loss function. We input the generated speech into an emotion feature extractor, a network designed to capture the emotional characteristics of the input, to obtain the emotional features. We compare the synthesized speech emotion vector with the emotion vector of the original speech, thus forcing the model to produce the same emotion in its response to the input. The structure of the loss function we adopted is as follows:

[0044] Loss = L HIFI-GAN +λcos_sim(emo orig ,emo gen )

[0045] Among them, emo orig and emo gen They represent the emotional features of the input speech and the features of the synthesized speech respectively. λ controls the weight of the emotional loss function.

[0046] Further, including:

[0047] The aforementioned step (2.2) constructs an anonymous pool with a number of speakers N = 30, and randomly selects m = 4 speakers to provide speech embeddings.

[0048] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0049] 1. Use emotion transfer technology to convert the emotional features in the original speech signal so that the converted speech signal is similar to the target speech in emotional expression.

[0050] 2. Interpolate the speaker features of the original speech signal in the feature space to generate new speaker features, thereby hiding the identity information of the original speaker.

[0051] 3. This method uses the self-supervised features of the pre-trained model as input, which can well preserve the language information of the speech, so that the anonymized language can meet the needs of subsequent use, such as speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a diagram of a system structure for speaker anonymization in an embodiment of the present invention;

[0053] Figure 2 This is a diagram of the feature fusion method for speech synthesis of the present invention. DETAILED DESCRIPTION

[0054] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0055] Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

[0056] refer to Figure 1 The present invention proposes a speaker anonymization method based on emotion transfer and feature interpolation, which includes a training stage and a conversion stage. The training stage is used to obtain a generator required for speech conversion, and the conversion stage is used to realize the conversion of the source speaker's speech into a pseudo speaker's speech.

[0057] To further explain, the training phase includes the following steps:

[0058] Step 1: Construct training corpus, which consists of speech from multiple speakers. The speech corpus required for training is taken from the train-clean-360 in the LibriSpeech corpus and the English-speaking part of the ESD corpus. The train-clean-360 corpus contains 921 speakers, each speaker has 25 minutes of speech, and the total length is 363.6 hours; the ESD corpus contains speech from 10 speakers with 5 emotions, with a total length of 13.4 hours.

[0059] Step 2: Traverse each audio file in the dataset, extract self-supervised features using the WavLM model, build matching and synthesis pools, and calculate the matching results. We use the features extracted from the 6th layer of WavLM-Large, which generates a vector for 16kHz audio every 20ms. According to the speaker of the current audio file, build matching pools and synthesis pools from other audio files of the same speaker. And use the fast cosine distance calculation method to evaluate the similarity between the current audio features and the features in the matching pool. Finally, select the best matching features and synthesize them into the final feature representation.

[0060] Step 3. Build a vocoder network based on HIFI-GAN, and input the speaker's pre-matched features and emotional features into the vocoder network for training. And in the speech synthesis stage, in addition to using the fusion input of speech and emotional features, we also adjusted the loss function. We input the generated speech into an emotional feature extractor, a network designed to capture the emotional characteristics of the input, to obtain emotional features. We compare the synthesized speech emotion vector with the emotion vector of the original speech, forcing the model to produce the same emotion in its response to the input. The structure of the loss function we adopted is as follows:

[0061] Loss = L HIFI-GAN +λcos_sim(emo orig ,emo gen )

[0062] Among them, emo orig and emo gen They represent the emotional features of the input speech and the features of the synthesized speech respectively. λ controls the weight of the emotional loss function.

[0063] To further explain, the conversion phase includes the following steps:

[0064] Step 1: Use the pre-trained WavLM model as a feature extractor to extract the language content and speaker information of the source audio.

[0065] Step 2: Use the wav2vec2-large-robust-12-ft-emotion-msp-dim model to extract the emotion feature vector of the speech;

[0066] Step 3: Build an anonymous pool. Randomly select N speakers from the corpus, and use the WavLM model to extract features from multiple audios of the speakers, and splice the features of each audio together to form reference features in the conversion process.

[0067] Step 4: Randomly select m speakers from the reference anonymous pool to provide speech embeddings, where For each speaker, the k-nearest neighbor (KNN) algorithm is used to obtain the nearest representation from the target reference anonymity pool. This step can be described as:

[0068]

[0069] KNN(x,Y,k) means finding the k vectors closest to vector x in set Y. T represents the source frame length of speaker spk.

[0070] Step 5. After obtaining the nearest representation of each speaker, we need to perform feature interpolation on the vectors of these target speakers to generate pseudo speakers. Assume that the randomly generated speaker weight vector w = (w1, w2, ..., w k ), the calculation formula of the weight vector is:

[0071]

[0072] Where spk represents the number of speakers randomly selected from the reference anonymous pool, and N(0,1) represents the weight vector w between 0 and 1. The sum of the weights is constrained to 1 through the softmax function.

[0073] Step 6: Perform weighted summation on the representation of each speaker to obtain the target pseudo speaker representation D, which is calculated as:

[0074] D=∑w spk D spk

[0075] Step 7: Send the emotion vector and target pseudo speaker representation generated in steps 2 and 6 to the trained model for anonymization to obtain anonymized speech.

[0076] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the claims. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A speaker anonymization method based on emotion transfer and feature interpolation, characterized in that: The method comprises a training phase and a conversion phase, The training phase is used to obtain the generator required for speech conversion, using the self-supervised features of the pre-trained model as input. The conversion stage is used to realize the conversion of the source speaker's voice into the pseudo speaker's voice, and the emotional features in the original voice signal are converted through the emotion transfer technology, so that the converted voice signal is similar to the target voice in terms of emotional expression.

2. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 1, characterized in that: The training phase includes the following steps: (1.1) Construct training corpus, which consists of the speaker's speech, and the speaker's corpus contains different emotional states; (1.2) Traverse each audio file in the dataset, use the WavLM model to extract self-supervised features, build matching and synthesis pools, and calculate the matching results; (1.3) Use the wav2vec2-large-robust-12-ft-emotion-msp-dim model to extract the emotional feature vector of the speech; (1.4) Construct a vocoder network based on HIFI-GAN; (1.5) The speaker’s pre-matching features and emotional features are input into the vocoder network for training.

3. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 1, characterized in that: The conversion phase includes the following steps: (2.1) Use the pre-trained WavLM model as a feature extractor to extract language content and speaker information; (2.2) Construct an anonymous speaker pool by generating virtual speakers and selecting and interpolating features according to specific rules; (2.3) The features extracted in step (2.1) are anonymized through the anonymity pool constructed in step (2.2) and converted into pseudo speaker features. (2.4) Use the wav2vec2-large-robust-12-ft-emotion-msp-dim model to extract the emotional feature vector of the speech; (2.5) The feature vectors generated in steps (2.3) and (2.4) are fed into the trained model for speech synthesis to obtain the anonymized speaker's speech.

4. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 2, characterized in that: In step (1.2), the features extracted from the 6th layer of WavLM-Large are used to generate a vector for 16kHz audio every 20ms. According to the speaker of the current audio file, a matching pool and a synthesis pool are constructed from other audio files of the same speaker, and the fast cosine distance calculation method is used to evaluate the similarity between the current audio features and the features in the matching pool. Finally, the best matching features are selected and synthesized into the final feature representation.

5. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 2, characterized in that: In step (1.3), the wav2vec2-large-robust-12-ft-emotion-msp-dim model is a variant based on the Wav2Vec2.0 framework. It is pre-trained with a large amount of unlabeled speech data and fine-tuned on the labeled data to improve the performance of the emotion recognition task. The emotion features are extracted through the pre-trained model, and the extraction process includes the following steps: (1) Construct a regression head to map features to sentiment labels. The regression head successfully converts the features extracted by the Wav2Vec 2.0 model into sentiment labels through a series of linear transformations, nonlinear activations, and Dropout layers. (2) Build an emotion recognition model, load the pre-trained Wav2Vec 2.0 model and add a regression head; (3) The input audio signal is inferred through the model to generate emotional feature embedding.

6. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 2, characterized in that: In step (1.5), the HIFIGAN-V1 vocoder is used for training, and self-supervised features and emotion features are used as input.

7. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 3, characterized in that: Step (2.2) The construction of the anonymous pool and the generation of pseudo speakers are key steps in the anonymization process. The construction process includes the following steps: (1) Randomly select N speakers from the corpus, and use the WavLM model to extract features from multiple audios of the speakers, and splice the features of each audio together to form reference features in the conversion process; (2) Randomly select m speakers from the reference anonymous pool to provide speech embeddings, where (3) For each speaker, use the k-nearest neighbor (KNN) algorithm to obtain the nearest representation from the target reference anonymous pool; this step can be described as: Where KNN(x,Y,k) means finding the k vectors closest to vector x in set Y, and T represents the source frame length of speaker spk; (4) After obtaining the most recent representation of each speaker, it is necessary to perform feature interpolation on the vectors of these target speakers to generate pseudo speakers. Assume that the randomly generated speaker weight vector w = (w1, w2, ..., w k ), the calculation formula of the weight vector is: Where spk represents the number of speakers randomly selected from the reference anonymous pool, and N(0,1) represents the weight vector w between 0 and 1; (5) Constrain the sum of weights to 1 through the softmax function; (6) Finally, the target pseudo speaker representation D is obtained by weighted summing the representations of each speaker, and its calculation formula is: D=2WspkDspk.

8. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 6, characterized in that: In the speech synthesis stage, in addition to using the fusion input of speech and emotional features, the loss function is also adjusted. The generated speech is input into an emotional feature extractor, which is a network designed to capture the emotional characteristics of the input to obtain the emotional features. The synthesized speech emotion vector is compared with the emotion vector of the original speech, thus forcing the model to produce the same emotion in its response to the input. The structure of the loss function adopted is as follows: Loss=L HIFI-GAN +λcos_sim(emo orig ,emo gen ) Among them, emo orig and emo gen They represent the emotional features of the input speech and the features of the synthesized speech respectively, and λ controls the weight of the emotional loss function.

9. The speaker anonymization method based on emotion transfer and feature interpolation according to claim 7, characterized in that: The number of speakers N = 30 to build the anonymous pool, and m = 4 speakers are randomly selected to provide speech embeddings.

Images