Speaker anonymization-oriented stylegan2-based F0 cloning method

Abstract

The invention relates to the technical field of voice processing, in particular to a speaker anonymization-oriented stylegan2-based F0 cloning method, which mainly realizes source speaker anonymization protection and comprises the following steps of: preprocessing an original fundamental frequency sequence, and converting a random noise vector into a representation vector of a style space; and a StyleGAN2 architecture is introduced to carry out layer-by-layer feature processing and fusion. A probabilistic fusion strategy is adopted, content features and constant features are added, then layer-by-layer stylization processing is carried out, feature modulation is carried out on each layer by using a style vector, the time sequence resolution is gradually recovered, and the anonymization target of the identity features of the speaker is effectively changed. And finally, combining the generative fundamental frequency (F0), the text information and the anonymized identity features to synthesize anonymized voice so as to hide the identity information of the speaker. And the effectiveness of the speaker anonymization-oriented StyleGAN2-based F0 cloning method is verified through an automatic speaker verification model and an automatic speech recognition system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, in particular to a F0 cloning method for speaker anonymization based on StyleGAN2. BACKGROUND

[0002] Speech anonymization technology is a speech signal processing technology that processes speech data through specific algorithms and methods to make it difficult to identify the identity features of the original speaker, thereby effectively protecting the privacy of users. The key to this technology is to hide the identity of the speaker while preserving the original semantic information and speech quality, that is, the speech after anonymization processing should retain the semantic information of the original speech, that is, the listener can understand the content expressed by the speech, and the processed speech should maintain high speech quality to avoid distortion or difficulty in understanding due to the processing process. The processed speech should not be easily identified as the identity features of the original speaker, including gender, age, accent, etc. The core purpose of speech anonymization technology is to protect the privacy of the speaker and prevent speech data from being used for unauthorized purposes.

[0003] Speaker anonymization system B3 is a system that uses a Wasserstein generative adversarial network (WGAN-QC) with quadratic transmission cost to generate artificial pseudo-speaker embeddings to replace the original speaker embeddings to anonymize the identity of the speaker, which can effectively hide the identity of the original speaker to a certain extent, making it difficult for attackers to accurately identify. However, although this system uses random values to adjust the fundamental frequency (F0) and energy values of each phoneme sequence, it can enhance privacy protection, but it may also cause problems such as a decrease in speech naturalness and a decrease in speech intelligibility.

[0004] Furthermore, the fundamental frequency (F0) distribution of different speakers usually has significant differences. In speech anonymization, if the fundamental frequency is not changed or the change method is inappropriate, the attacker may identify the original speaker through the identity information retained in the fundamental frequency (F0) features, thereby causing privacy leakage. Therefore, how to overcome the defects of existing technologies that achieve fundamental frequency transformation through deterministic mapping to cause privacy leakage has become a problem to be solved. SUMMARY

[0005] The purpose of the present application is to provide a F0 cloning method for speaker anonymization based on StyleGAN2. This method aims to overcome the defects of existing F0 transformation through deterministic mapping, establish a nonlinear mapping relationship of F0 features, achieve F0 conversion based on style understanding, and generate more natural and more difficult to reverse anonymization results, while better maintaining the expressiveness of speech.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] The F0 cloning method based on StyleGAN2 for speaker anonymization includes two stages.

[0008] The first stage is to anonymize the fundamental frequency F0 by constructing a StyleGAN2 model, and train the model through a specific data set and training strategy.

[0009] The second stage is to replace the anonymization method of the fundamental frequency F0 in the B3 system by inputting the trained model into the speaker anonymization B3 system, and verifying the anonymization performance of the system.

[0010] Preferably, the first stage includes the following steps:

[0011] S100, input preprocessing: the system inputs the fundamental frequency F0 sequence, the dimension of which is 1, then performs dimension expansion preprocessing to convert the one-dimensional F0 sequence into the three-dimensional tensor format expected by the StyleGAN2 model: [batch_size, sequence_length, feature_dimension], and then transpose it to [batch_size, feature_dimension, sequence_length] to adapt to 1D convolution operation;

[0012] S200, convert the randomly generated latent vector into a style vector through a mapping network, which is used as the style vector of the target speaker to generate the identity feature of the target speaker;

[0013] S300, encode the original fundamental frequency through an encoder to obtain the encoded feature, initialize the constant to obtain the constant feature, and then conditionally fuse the encoded feature and the constant feature of the original fundamental frequency to obtain the mixed feature;

[0014] S400, perform progressive stylization processing on the mixed feature to obtain high-resolution fine F0 details;

[0015] S500, convert the output feature after stylization processing back to the original input format to ensure that the length of the generated feature sequence is completely consistent with the input;

[0016] S600, generate the final output and convert the internal feature to the fundamental frequency value.

[0017] Preferably, the second stage includes:

[0018] Input the trained model into the B3 system to replace the F0 anonymization method of the B3 system;

[0019] Use the modified model to anonymize the provided test data set;

[0020] The error rate, the word error rate, and the unweighted average recall are calculated as performance metrics to evaluate the privacy and usability of the anonymization system.

[0021] Preferably, S200 comprises:

[0022] First, randomly generate latent vectors , a tensor of shape [batch_size, latent_dim] is generated by randomly sampling from the standard normal distribution; where latent_dim = 128;

[0023] The vector is converted to a style vector by a mapping network ; the generated Apply the truncation technique to improve the generation quality and avoid extreme unnatural samples;

[0024] Take = 0.8, use 80% of the original features and 20% of the average features, and the average features are general speaker voice features learned from the training data, and the specific formula is:

[0025] ;

[0026] Where w is the original style vector, is the average of the style vector, is the truncation parameter (range 0-1), is the truncated style vector, and the further truncated style vector is taken as the style vector of the target speaker to generate the target speaker identity feature.

[0027] Preferably, S300 comprises:

[0028] Encoding and feature fusion: encode the original fundamental frequency into a feature representation by an encoder to preserve the basic prosodic information of the speech: ;

[0029] At the same time, the constant is initialized to create a learnable constant feature As a starting point for generation, this constant feature is the optimal generation starting point learned in the model training stage;

[0030] Subsequently, the encoded features of the original fundamental frequency F0 and the constant features are conditionally fused, and the two different source features are selectively combined. 90% probability will align the size of the encoded features and the constant features after mixing to obtain the mixed features x, and the specific formula is: ;

[0031] Where 10% probability is generated completely from constants, maximum anonymization, when only constant features are taken .

[0032] Preferably, S400 comprises:

[0033] Obtain mixed features , features Input style convolutional blocks, where each convolutional block is modulated using a style vector w, then inject the F0 features of the target speaker according to w modulating its convolutional weights, the specific formula is: ;

[0034] Where each sample uses a personalized weight At the same time, the style block will be 2 times up-sampled after processing to gradually restore the sequence length, the specific formula is: ;

[0035] The whole process starts from the low-resolution global pitch F0 trend, goes through the medium-resolution syllable-level changes, and finally gets the high-resolution fine F0 details, keeping the style consistency of the whole F0 generation process while gradually enriching the detail information.

[0036] Preferably, S500 comprises:

[0037] Convert the output back to the original input format, ensure that the generated feature sequence length is completely consistent with the input, the specific formula is: ;

[0038] Where, is the target sequence length required for adjustment of the output pitch feature.

[0039] Preferably, S600 comprises:

[0040] Generate the final output, convert the internal features to pitch values, the specific formula is: ;

[0041] Where bias represents a bias value obtained by model training, which limits the value output by the model within a reasonable numerical range;

[0042] Then convert the output to the original dimension [batch_size, sequence_length, feature_dimension], finally only apply the anonymization result to the voiced part, and keep the original value for the unvoiced part, the specific formula is: ;

[0043] Where the voiced mask is applied to the voiced part = 1, in the silent part = 0, so that the system can maintain the naturalness and intelligibility of the voice as much as possible while protecting the speaker's identity.

[0044] Compared with the prior art, the present application has the beneficial effects that:

[0045] The method proposed in the present application is based on the StyleGAN2 architecture, and a fundamental frequency (F0) anonymization system is constructed, that is, the performance test is performed on the speaker anonymization system (B3) which uses the phoneme transcription and the generative adversarial network to perform fundamental frequency anonymization.

[0046] The method first pre-processes the original fundamental frequency sequence, and converts a random noise vector into a representation vector w in the style space through a mapping network, as a new acoustic feature representation of the fundamental frequency. Subsequently, the system introduces the StyleGAN2 architecture for layer-by-layer feature processing and fusion. The system uses a probabilistic fusion strategy (90% fusion probability) to add the content features and the constant features, and then performs layer-by-layer stylization processing through a plurality of style convolution blocks. Each layer uses a style vector for feature modulation, and gradually recovers the temporal resolution through upsampling, so as to achieve the anonymization goal of effectively changing the speaker identity features on the premise of maintaining the speech content semantics. Finally, the generative fundamental frequency (F0), the text information and the anonymized identity features are combined to synthesize an anonymized speech, so as to realize the hiding of the speaker identity information. The effectiveness of the StyleGAN2-based F0 cloning method for speaker anonymization is verified through an automatic speaker verification model (ASV) and an automatic speech recognition (ASR) system. The present method uses the StyleGAN2 to hide the speaker identity information contained in the source speaker fundamental frequency information, so as to improve the privacy performance and utility of the anonymization system. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application, and are used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0048] Fig. 1 is a system schematic diagram of the present application;

[0049] Fig. 2 is a B3 system schematic diagram in the embodiment of the present application;

[0050] Fig. 3 is a system schematic diagram after the B3 system in the access embodiment of the present application. DETAILED DESCRIPTION

[0051] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0052] Please refer to Figs. 1-3 The present application provides technical solutions:

[0053] Embodiment 1: The present application is a F0 cloning method based on StyleGAN2 for speaker anonymization. The present application verifies the performance by accessing the B3 system, and replaces the original model to generate the processing mode of the anonymization fundamental frequency (F0) in the speech anonymization process of the B3 system. The implementation process of the present application includes two stages. The first stage is the pre-training stage of the model. Through specific data set and training strategy, the model is trained to have feature learning and expression ability. The second stage is to access the pre-trained model to the B3 system to replace the anonymization mode of the fundamental frequency (F0) in the B3 system. The performance of the model is measured by the synthesized anonymized language of the B3 system.

[0054] The pre-training of the model includes the following steps:

[0055] Step 1, prepare the training corpus. The training set of the model is the train-clean-100 subset of the LibriSpeech corpus. The train-clean-100 data contains about 100 hours of "clean" speech data, which comes from 251 speakers, with a relatively balanced male-female ratio, with an average of about 24 minutes of audio contributed by each speaker. The data set has relatively clear speech, less background noise, and relatively standard accent. Based on the characteristics of the train-clean-100 data set, it is therefore selected as the training set for model development.

[0056] Step 2, encode the unique speaker ID. In the LibriSpeech data set, since the speaker ID is usually a non-continuous integer, it needs to be processed and encoded into a continuous unique speaker ID (0~250) suitable for model processing, corresponding to the 251 speakers in the data set.

[0057] Step 3: Resample the audio file to 16kHz, extract the fundamental frequency using the DIO (Distributed Inline-filter Operation) algorithm of the WORLD vocoder, and then use the StoneMask algorithm to improve the accuracy of the F0 estimation. Use the obtained F0 to create a sound mask; an F0 greater than zero indicates a frame with sound. Subsequently, perform a logarithmic transformation on F0 to convert the fundamental frequency value to a logarithmic scale. This conforms to the logarithmic perception characteristics of human hearing and transforms the originally exponentially distributed F0 values ​​into a form closer to a normal distribution, reducing the influence of extreme values, improving numerical stability, and making the data more suitable for neural network processing. That is:

[0058] ;

[0059] in .

[0060] Again Standardization, achieved by subtracting the mean and dividing by the standard deviation, unifies the fundamental frequency data from different speakers to the same numerical range (mean 0, standard deviation 1). This eliminates differences in fundamental frequency range between individuals, allowing the model to focus on learning the relative variation patterns and contour features of the fundamental frequency, rather than being affected by the absolute fundamental frequency value. This improves training stability and generalization ability. ;

[0061] in for The mean, for The variance.

[0062] Step 3: Model Training. The main objective of the model is to modify the fundamental frequency (F0) speaker features without losing linguistic content. The system employs adversarial training, comprising two core components: a generator and a discriminator. The generator converts the original F0 into an anonymized F0, while the discriminator determines the authenticity of the generated anonymized F0 and identifies the speaker. During training, the generator randomly selects another speaker ID from the dataset that differs from the original speaker ID as the target speaker, and uses an embedding layer to encode the target speaker ID into a 256-dimensional speaker embedding. ,Right now: ;

[0063] Speaker embedding Generating the target speaker's style vector using a mapping network. The acoustic features of the target speaker are represented as follows: ;

[0064] The generator controls the generation process by the style vector and generates the anonymized fundamental frequency (F0) combining the input original fundamental frequency content features and the learnable constant features. The discriminator is a two-headed classifier, which is the speaker classifier and the authenticity classifier. The speaker classification and authenticity classification use the cross-entropy loss function and the Softplus function, respectively. The loss calculation formula is as follows:

[0065]

[0066]

[0067] is the predicted output value of the discriminator for the real sample, is the predicted output value of the discriminator for the generated sample (fake sample). is the cross-entropy loss function, is the R1 regularization loss, is the gradient accumulation step number, which is a hyperparameter, and here = 4. The loss of the generator includes the content preservation loss, the adversarial loss, and the identity transformation loss. The mean square error loss function, the Softplus function, and the cross-entropy loss function are used, respectively. The loss calculation formula is as follows:

[0068]

[0069] are weight parameters of the loss functions, and the adversarial loss and the identity transformation loss are the same as the generated pseudo-speaker loss and the classification loss.

[0070]

[0071] N is the total number of voiced frames, is the original fundamental frequency value, is the anonymized fundamental frequency value, which is constrained by the content loss function to ensure that the anonymized fundamental frequency contour is similar to the original fundamental frequency in terms of language content, thereby maintaining the intelligibility of the speech. Each round of training includes two main steps: first, training the discriminator, and then training the generator. In the process of adversarial training, the discriminator learns to recognize different speaker features, and the generator learns to extract and separate identity and content features. The model is trained for a total of 30 rounds, and the learning rate of the generator and the discriminator is 0.0001, the truncation technique is truncation=0.8, and the R1 regularization strength is 0.1.

[0072] Step 4, save the trained model. ​​​​​​​​​​​

[0073] The second stage after the model training is completed includes the following steps:

[0074] Step A, the trained model is connected to the B3 system, replacing the B3 system's way of anonymizing F0.

[0075] Step B, using the modified model to anonymize the provided test data set.

[0076] Step C, through enough anonymized samples and unanonymized original language, calculate the error rate (EER), word error rate (WER), and unweighted average recall (UAR) as performance indicators to evaluate the privacy and usability of the anonymization system.

[0077] Finally, it should be noted that the above only for the preferred embodiments of the present application, and not for the purpose of limiting the present application, although the foregoing embodiments of the present application have been described in detail, for those skilled in the art, it still can be modified, or part of the technical features of the equivalent replacement of the technical solutions described in the foregoing embodiments. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A stylegan2-based F0 cloning method for speaker anonymization, characterized by: The method comprises two stages; The first stage involves anonymizing the fundamental frequency F0 by constructing a StyleGAN2 model and training the model using a specific dataset and training strategy. The second stage involves connecting the trained model to the speaker anonymization B3 system, replacing the B3 system's anonymization method for the fundamental frequency F0, and verifying the anonymization performance of this system.

2. The F0 cloning method based on StyleGAN2 for speaker anonymization as described in claim 1, characterized in that, The first stage includes the following steps: S100. The system inputs a fundamental frequency F0 sequence. The sequence has a dimension of 1. Then, it performs dimension expansion preprocessing to convert the one-dimensional F0 sequence into the three-dimensional tensor format expected by the StyleGAN2 model: [batch_size, sequence_length, feature_dimension]. Then, it is transposed to [batch_size, feature_dimension, sequence_length] to adapt to 1D convolution operations. S200. The randomly generated latent vectors are converted into style vectors through a mapping network, which are used as the style vectors of the target speaker to generate the target speaker's identity features. S300: The encoder encodes the original baseband to obtain the encoding features, initializes the constant to obtain constant features, and then conditionally fuses the encoding features of the original baseband and the constant features to obtain the hybrid features. S400: Progressive stylization of mixed features yields high-resolution, fine F0 details. S500: Convert the stylized output features back to the original input format to ensure that the length of the generated feature sequence is exactly the same as the input. S600 generates the final output and converts the internal features into the fundamental frequency value.

3. The F0 cloning method based on StyleGAN2 for speaker anonymization as described in claim 2, characterized in that, The S200 includes: First, generate potential vectors randomly. A tensor z of shape [batch_size,latent_dim] is generated by randomly sampling from a standard normal distribution; where latent_dim=128; Will Vectors are converted into style vectors through a mapping network. ; For the generated By applying truncation techniques, the quality of generated samples can be improved, and extremely unnatural samples can be avoided. Pick = 0.8, using 80% of the original features and 20% of the average features. The average features are general speaker voice features learned from the training data, and the specific formula is: ； Where w is the original style vector. The average value of the style vector. For the truncation parameter, The truncated style vector is further truncated. The style vector of the target speaker is used to generate the target speaker identity features.

4. The F0 cloning method based on StyleGAN2 for speaker anonymization as described in claim 2, characterized in that, The S300 includes: The original fundamental frequency is encoded into a feature representation by an encoder. To preserve the basic prosodic information of speech: ; Simultaneously, constants are initialized, creating learnable constant features. As the starting point for generation, this constant feature is the optimal starting point for generation learned during the model training phase; Subsequently, the encoded features and constant features of the original fundamental frequency F0 are conditionally fused. Features from two different sources are selectively combined, and the encoded features and constant features are aligned in size with a 90% probability before being mixed to obtain the mixed feature x. The specific formula is as follows: ; Of these, there is a 10% probability that the data is generated entirely from constants, resulting in maximum anonymization; in this case, only constant features are taken. .

5. The F0 cloning method based on stylegan2 for speaker anonymization as described in claim 1, characterized in that, The S400 includes: Obtaining blended features , will feature Given input style convolutional blocks, where each convolutional block is modulated using a style vector w, the F0 features of the target speaker are injected by modulating the convolutional weights according to w, as shown in the specific formula: ; Each sample uses a personalized weight. Meanwhile, the style blocks are upsampled by a factor of 2 after processing to gradually restore the sequence length. The specific formula is as follows: ; The entire process begins with a low-resolution global fundamental frequency (F0) trend, progresses through medium-resolution syllable-level changes, and finally yields high-resolution, fine F0 details, maintaining stylistic consistency throughout the F0 generation process while gradually enriching the detailed information.

6. The F0 cloning method based on StyleGAN2 for speaker anonymization as described in claim 2, characterized in that, The S500 includes: Convert the output back to the original input format to ensure that the length of the generated feature sequence is exactly the same as the input. The specific formula is as follows: ; in, The target sequence length that needs to be adjusted for the output fundamental frequency characteristics.

7. The F0 cloning method based on stylegan2 for speaker anonymization as described in claim 2, characterized in that, The S600 includes: To generate the final output, the internal features are converted into fundamental frequency values, using the following formula: ; Here, bias represents a bias value obtained during model training, which limits the output value of the model to a reasonable range. Then output Convert to the original dimensions [batch_size, sequence_length, feature_dimension], and finally apply the anonymization result only to the audio portion, while keeping the silent portion unchanged. The specific formula is as follows: ; Among them, audio mask In the audio part =1, in the silent part =0, which allows the system to protect the speaker's identity while maintaining the naturalness and comprehensibility of the speech as much as possible.

8. The F0 cloning method based on stylegan2 for speaker anonymization as described in claim 1, characterized in that, The second stage includes: The trained model is then connected to the B3 system, replacing the B3 system's method of anonymizing F0. Anonymize the provided test dataset using the modified model; The privacy and usability of anonymization systems are evaluated by using error rates, word error rates, and unweighted average recall rates as performance metrics, calculated using a sufficient number of anonymized samples and the original, unanonymized language.

Images