A voice data-oriented identity privacy protection method and system

Abstract

This invention relates to the field of identity privacy protection technology, specifically to a method and system for protecting the identity privacy of voice data. The system includes: a voice feature extraction module, which extracts the F0, BN vector, and x-vector of the user's voice signal using an F0 extractor, a BN vector extractor, and an x-vector extractor; an x-vector modification module, which uses the obtained x-vector to generate an anonymized x-vector through three independently trained adversarial example generators; and a voice synthesis module, which uses the F0, BN vector, and anonymized x-vector to generate an MFCC using an MFCC generator, and inputs the generated MFCC, F0, and anonymized x-vector together with a vocoder to generate anonymized speech. This invention ensures security while preserving data usability as much as possible, avoiding unnecessary information loss, and better preserving the sound quality and intelligibility of the original speech.

Description

Technical Field

[0001] This invention relates to the field of identity privacy protection technology, specifically a method and system for protecting identity privacy for voice data. Background Technology

[0002] Voice privacy protection technology refers to hiding the speaker's identity while preserving the original voice quality and intelligibility, and it has a very important application in the field of personal privacy protection.

[0003] Currently, voice privacy protection methods typically decouple speaker representation from semantic information and then achieve anonymization by modifying the speaker representation. One approach is to modify the speaker representation x-vector. This method uses an x-vector generated from an external set of speakers to replace the target speaker's x-vector, thus anonymizing and protecting identity privacy. For a given target speaker, this method selects a set of candidate x-vectors from the external x-vector set according to predetermined rules, calculates their mean, and uses this mean vector as the anonymized x-vector, replacing the original speech x-vector. The resulting anonymized speech is then resynthesized. However, this method often results in a significant difference between the anonymized x-vector and the original speech x-vector, greatly impacting the sound quality and intelligibility of the synthesized anonymized speech. Furthermore, the randomness of the x-vector modification stage is poor, making it vulnerable to targeted attacks and negatively affecting the security of the anonymized speech. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention proposes a method and system for protecting the identity privacy of voice data. Anonymization is achieved by modifying the x-vector by adding adversarial perturbations. This method can increase the randomness of the x-vector modification stage and reduce the difference between the anonymized x-vector and the original voice x-vector during the modification process, thereby improving security while reducing the impact on the sound quality and intelligibility of the anonymized voice.

[0005] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows: An identity privacy protection system for voice data includes: The speech feature extraction module extracts the F0, BN and x-vector of the user's speech signal through the F0 (fundamental frequency) extractor, BN (bottleneck vector) extractor and x-vector extractor, respectively. Modify the x-vector module to generate an anonymized x-vector by passing the obtained x-vector through three independently trained adversarial example generators; The speech synthesis module is used to generate MFCCs from F0, BN vectors and anonymized x-vectors through the MFCC generator, and then input the generated MFCCs, F0 and anonymized x-vectors into the vocoder to synthesize anonymized speech.

[0006] Furthermore, the F0 extractor utilizes the time-domain information of speech to determine the pitch period by calculating the autocorrelation function and the variable-length average amplitude difference function, thereby determining the F0 parameters; the MFCC extractor uses a Mel frequency filter bank to extract the MFCC, and the obtained MFCC is used by an i-vector estimator and an x-vector extractor to estimate the i-vector and extract the x-vector, respectively. Then, the obtained MFCC and i-vector are used by a BN vector extractor to extract the BN vector.

[0007] Furthermore, the three adversarial example generators establish different loss functions, calculate the gradient of their respective loss functions with respect to the x-vector, add adversarial perturbations based on the gradient and perturbation coefficients to the x-vector, causing the x-vector to be updated along the negative gradient direction, and iteratively update the x-vector until it converges to generate adversarial examples. The outputs of the three adversarial example generators are then fused using a parallel ensemble method as an anonymized x-vector. This anonymized x-vector can cause the attacker's speaker classifier to misclassify the speaker, making it unable to distinguish the original speaker's identity, thus achieving the purpose of anonymization.

[0008] Furthermore, during the training of the i-vector estimator, arbitrary vectors are extracted from the speech database. The speech data of each speaker is used to model the probability distribution density of the feature parameter space of each speaker's speech using a Gaussian Mixture Model (GMM). The mean vector of each Gaussian component of the GMM is concatenated and projected onto the global difference space to calculate its corresponding i-vector, ultimately yielding a result... A GMM model and An i-vector estimator composed of corresponding i-vectors; Furthermore, the x-vector extractor consists of a frame-level extraction module, a statistical pooling layer, a sentence-level extraction module, and an output layer. During training, it processes data from the speech database... The speech data of each speaker is used to extract the MFCC of each frame of speech. The MFCC parameters of each frame of a sentence are used as input, and the corresponding speaker identity is used as the label. An automatic speaker verifier with a TDNN structure is trained as the x-vector extractor. The output of a fully connected layer of the TDNN network is used as the x-vector.

[0009] Furthermore, the BN vector extractor consists of a TDNN-F linear layer and an output layer. During training, it is trained on the speech database... The speech data of each speaker is used to extract the MFCC of each frame. The MFCC of each frame of each speaker and the speaker's i-vector are concatenated and used as the input of the network. The phoneme sequence of each frame is used as the label. An automatic speech recognizer with a TDNN-F structure is trained as the BN vector extractor. The output of a linear layer of the network is used as the BN vector.

[0010] Furthermore, the MFCC generator consists of a fully connected layer, a bidirectional long short-term memory (Bi-LSTM) network layer, an autoregressive layer, a long short-term memory (LSTM) network layer, and a Highway-PostNet layer. The F0, BN vectors, and x-vector are concatenated and used as input to the fully connected layer. The fully connected layer performs a linear transformation on the input to extract features and uses the ReLU function to perform a non-linear transformation on the features. The Bi-LSTM layer models the feature sequence and captures the contextual information within it. The autoregressive layer uses a recursive method to generate the output of the current time step using the output of the previous time step, thus generating the output sequence. The LSTM layer captures and transmits information from the sequence data through a combination of input gates, forget gates, cell states, and output gates. The Highway-PostNet layer controls the transmission of information through the gating mechanism of the Highway layer and uses the PostNet layer to post-process the generated MFCCs, finally outputting the generated MFCCs.

[0011] Furthermore, the voice database The speech data of a speaker is used to extract MFCC, x-vector, and F0 as inputs to a vocoder. The speech signal time series is used as a label to train the vocoder. The vocoder is a neural source filter consisting of a conditional module, a source module, and a filter module. It converts the input acoustic features into an output speech waveform. The conditional module consists of a Bi-LSTM layer, a convolutional layer, and an upsampling module. It directly upsamples F0. After passing the MFCC and x-vector sequences through the Bi-LSTM and convolutional layers, it further upsamples them to become a speech feature sequence of length T. The source module receives the upsampled F0. After sequencing, a sinusoidal excitation signal is generated and sent to the filter module.

[0012] This invention also provides a method for protecting the identity privacy of voice data. This method uses the aforementioned system for protecting the identity privacy of voice data to achieve identity privacy protection. When a user uses the system, they only need to input voice data. The system first decouples the speaker representation and semantic information of the voice signal. Then, through three independently trained adversarial example generators, adversarial perturbations based on different norm constraints are added to the x-vector to modify the speaker representation x-vector. The outputs of the three adversarial example generators are then fused using a parallel integration method to form an anonymized x-vector. This anonymized x-vector replaces the original voice x-vector, and the data is resynthesized, thus providing the user with anonymized voice data.

[0013] The present invention provides a method for protecting identity privacy for voice data, comprising the following steps: 1) For any user, inputting a speech signal, the speech signal is first pre-emphasized in the high-frequency part by passing through a high-pass filter to obtain a speech signal with a relatively flat spectral amplitude. Then, the system uses a sliding window of fixed length to divide the pre-emphasized signal into short time frames. After dividing the signal into short time frames, each frame signal is multiplied by a Hamming window function, and each frame signal is then subjected to a Fast Fourier Transform (FFT) to obtain the spectrum of each frame signal. 2) The preprocessed signal is passed through the F0 extractor to extract the fundamental frequency of each frame; the F0 extractor first calculates the... Frame signal The short-time autocorrelation function is then calculated, followed by the calculation of the first... Frame signal The variable-length average amplitude difference function is used to calculate the quasi-periodic function. ,when When the maximum value is taken within the set fundamental frequency range, the sampling frequency is... In addition to the time delay at this time That is, the first F0 of the frame signal; 3) The preprocessed signal is passed through an MFCC extractor, which uses a Mel-frequency filter bank to extract the MFCCs. The obtained MFCCs are then used by an i-vector estimator to estimate the user's i-vector. The i-vector estimator first calculates the MFCC sequence in... The probabilities under each GMM model are obtained. After finding the probabilities, The speaker corresponding to the maximum of the probabilities Speaker The corresponding i-vector is the estimated user i-vector; 4) The obtained MFCC uses the output of the first fully connected layer of the sentence-level feature extraction module in the network as the speaker feature x-vector through the x-vector extractor to extract the x-vector; 5) The obtained MFCC and estimated i-vector are passed through the BN vector extractor, which uses the output of the last linear layer in the network as the BN vector to extract the BN vector; 6) The extracted x-vector is processed by three adversarial example generators based on different norm constraints. By adding adversarial perturbations based on different norm constraints to the x-vector, anonymized x-vectors are generated. Specifically, pseudo-speaker labels are first randomly selected for the anonymized x-vector. (Pseudo-speaker tag) (Cannot be labeled as a real speaker), then Norm adversarial sample generators generate samples by feeding inputs to the input. Add based on Adversarial noise with norm constraints, generating adversarial examples ,Right now: (1) Specifically The norm adversarial sample generator first calculates the loss function. right The gradient and loss function are: (2) The first term of the loss function is minimized. and Euclidean distance to limit and The difference, the second term encourages the classifier to use adversarial examples. Classified as a pseudo-speaker To ensure the perturbation effect, among which, For classifier adversarial examples Classified as speaker The probability, To eliminate false speakers In addition, the maximum value of the classification probability of other speakers; It is an exponential function. The base of the loss function is 2, and its value is 1; when the second term of the loss function is less than or equal to 0, the classifier is considered to have handled adversarial examples. Misclassified as a pseudo-speaker ; Hyperparameters are used to adjust The weights of norm constraints and perturbation effects; Calculate the loss function pair After obtaining the gradient, it is propagated back to... Add based on Adversarial perturbations of norm constraints make Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: (3) (4) in, This indicates that, for classifiers The pseudo-speaker label is The adversarial examples generated by the perturbation are The loss function; For differential vector operators, Indicates the input vector For each element of the loss function Find the partial derivative; This is the step size for each iteration; set it to 0.001. It is a signed function that takes a real value as its parameter. If the input is greater than 0, it returns 1; if the input is equal to 0, it returns 0; if the input is less than 0, it returns -1. 7) Norm adversarial sample generators generate samples by sending... Add based on Adversarial noise with norm constraints, generating adversarial examples ,Right now: (5) Specifically Norm adversarial sample generators limit the number of modified elements. and differences The norm adversarial sample generator first estimates the input through a privacy risk value estimator. The privacy risk value of each element is calculated, and the privacy risk values ​​of each element are sorted in descending order. When the sum of the privacy risk values ​​of the elements with the largest privacy risk values ​​at the top of the sort is k% of the sum of the privacy risk values ​​of all elements, the set of elements with the largest privacy risk values ​​is defined as the privacy risk region. The number of elements that can be modified is limited by allowing modification only to elements within the privacy risk region. The norm adversarial example generator then calculates the loss function. The loss function is: (The loss function is used to calculate the gradient of elements within the privacy-risk region.) (6) The loss function encourages the classifier to... Classified as pseudo-speaker tag To ensure the perturbation effect; after calculating the gradient of the loss function with respect to elements in the privacy risk region, it is then propagated back to... Adding adversarial perturbations to make Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: (7) (8) 8) Norm adversarial sample generators generate samples by sending... Add based on After applying adversarial noise with norm constraints, maximum values ​​are clipped to generate adversarial examples. ,Right now: (9) Specifically Norm adversarial sample generators limit the maximum magnitude of modifications to elements. and differences The norm adversarial sample generator first calculates the loss function. right The gradient and loss function are: (10) The first term of the loss function is appropriately constrained by minimizing the Euclidean distance. and The difference, the second term encourages the classifier to Classified as pseudo-speaker tag ; Hyperparameters are used to adjust Norm constraints and weights of perturbation effects; calculation of the loss function. After obtaining the gradient, it is propagated back to... Add based on After adversarial perturbation of the norm constraint, maximum pruning is performed, such that... Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: (11) (12) 9) Then, the outputs of the three adversarial example generators are fused using a parallel ensemble method, i.e.: (13) Determined by gradient descent method , , The value of , specifically, is calculated for the loss function. , , The gradient and loss function are: (14) The loss function encourages the classifier to... Classified as pseudo-speaker tag To ensure the perturbation effect; calculate the loss function for... , , After obtaining the gradient, it is backpropagated to make , , Update along the negative gradient direction, and after several iterations, make... , , It tends to converge, that is: (15) (16) (17) 10) Finally, F0, BN vector, and anonymized x-vector are passed through the MFCC generator to generate MFCC, and then input together with F0 and anonymized x-vector into the vocoder to synthesize anonymized speech.

[0014] This invention has the following characteristics and beneficial effects: 1) By adding adversarial perturbations, the problems of significant differences between the anonymized x-vector and the original speech x-vector, as well as poor randomness in the anonymization process, are addressed in traditional x-vector modification algorithms. During x-vector modification, the magnitude of the adversarial perturbation is constrained by the norm. While ensuring security, this approach maximizes data usability, avoids unnecessary information loss, and better preserves the sound quality and intelligibility of the original speech.

[0015] 2) Add based on When dealing with adversarial perturbations of the norm, the privacy risk region is estimated using a privacy risk value estimator, which provides a more accurate estimate of the privacy risk region of the x-vector compared to traditional estimation methods.

[0016] 3) When adding adversarial perturbations, adversarial perturbations based on different norm constraints were used and fused using a parallel integration method. This increased the robustness of anonymization, making it more difficult for attackers to eliminate the added perturbations using distillation defenses or other means, thus increasing the security of anonymization. Attached Figure Description

[0017] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a system block diagram of an identity privacy protection system for voice data according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the i-vector estimator in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the x-vector extractor in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of the BN vector extractor in an embodiment of the present invention.

[0020] Figure 5 This is a schematic diagram of the MFCC generator network structure in an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of a vocoder in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0023] The identity privacy protection system for voice data proposed in this embodiment of the invention requires training before it can be used. During the training phase, arbitrary data is extracted from the voice database. The speech data of each speaker is used to model the probability distribution density of the feature parameter space of each speaker's speech using a Gaussian Mixture Model (GMM). The mean vectors of the Gaussian components of each GMM are concatenated and projected onto the global difference space to calculate their corresponding i-vectors, ultimately yielding a result... A GMM model and An i-vector estimator composed of corresponding i-vectors. Simultaneously, Using the speech data of several speakers as the training set, an automatic speech recognizer based on a Time Delay Neural Network with Factorized (TDNN-F) architecture was trained, serving as a bottleneck (BN) vector extractor, with the output of a linear layer of this network used as the BN vector. Separately, an automatic speaker verifier based on a Time Delay Neural Network (TDNN) architecture was trained, serving as an x-vector extractor, with the output of a fully connected layer of this network used as the x-vector. An x-vector classifier and a privacy risk value estimator were then trained to construct... The loss function of the adversarial example generator is calculated by adding an adversarial perturbation to the x-vector by calculating the gradient of the loss function with respect to the x-vector, causing the x-vector to be updated along the negative gradient direction, thereby generating adversarial examples. Adversarial examples. Train an acoustic model with a neural network architecture as a Mel-Frequency Cepstral Coefficients (MFCC) generator, outputting the generated MFCCs. Train a Neural Source Filter (NSF) as a vocoder, outputting anonymized speech.

[0024] When a user uses the system, they only need to input voice data. The system first decouples the speaker representation and semantic information from the voice signal. Then, through three independently trained adversarial example generators, adversarial perturbations based on different norm constraints are added to the x-vector, modifying the speaker representation x-vector. The outputs of the three adversarial example generators are then fused using a parallel ensemble method to create an anonymized x-vector. This anonymized x-vector replaces the original voice x-vector, and the system is resynthesized, providing the user with anonymized voice data. The system block diagram of this invention is shown below. Figure 1 As shown.

[0025] The entire identity privacy protection system consists of three parts. The first is the speech feature extraction module. The user's speech signal is processed by a fundamental frequency (F0) extractor and an MFCC extractor to extract F0 and MFCC respectively. The F0 extractor uses the time-domain information of the speech to determine the fundamental period by calculating the autocorrelation function and the variable-length average amplitude difference function, thereby determining the F0 parameters. The MFCC extractor uses a Mel-frequency filter bank to extract the MFCC. The obtained MFCC is then used by an i-vector estimator and an x-vector extractor to estimate the i-vector and extract the x-vector respectively. Finally, the obtained MFCC and i-vector are used by a BN vector extractor to extract the BN vector.

[0026] Secondly, the x-vector module is modified so that the obtained x-vector is used by three independently trained adversarial example generators to generate anonymized x-vectors. The three adversarial example generators establish different loss functions, calculate the gradient of their respective loss functions with respect to the x-vector, and add adversarial perturbations based on the gradient and perturbation coefficients to the x-vector, causing it to update along the negative gradient direction. This process is iteratively updated until convergence, generating adversarial examples. The outputs of the three adversarial example generators are then fused using a parallel ensemble method to create the anonymized x-vector. This anonymized x-vector can cause the attacker's speaker classifier to misclassify the speaker, making it unable to distinguish the original speaker's identity, thus achieving anonymization.

[0027] Third is the speech synthesis module, which generates MFCCs by using the F0, BN vectors and anonymized x-vectors through the MFCC generator, and then inputs the generated MFCCs, F0, and anonymized x-vectors into the vocoder to synthesize anonymized speech.

[0028] The following sections will introduce the structure and training process of each module in detail, and elaborate on the specific steps of the voice identity privacy protection method.

[0029] 1. Training the i-vector estimator For the speech library Speech data from each speaker is processed using Mel-frequency filter banks to extract the MFCCs (Multi-Frequency Computational Vectors) for each frame. Based on the MFCC vectors of each speaker's features, a Gaussian Mixture Model (GMM) is constructed in the feature parameter vector space for each speaker. The GMM model parameters are then obtained using the Expectation Maximization (EM) algorithm. A GMM model is generated. Then, the mean vectors of the Gaussian components of each GMM are concatenated to obtain a supervector. Projecting onto the global difference space, calculate its corresponding i-vector, i.e.: in, Indicates the first The supervector of a GMM; for hypervectors The mean vector, The global difference space matrix, This represents the global difference space factor. Global difference space matrix. The parameters are obtained using the EM algorithm. , , Given a certain value, calculate the global dissimilarity space factor using the maximum a posteriori probability. , That is, the first Each GMM corresponds to an i-vector. The above operations are performed on the speech of each speaker to obtain... A GMM model and its corresponding There are i-vectors. The above process is as follows: Figure 2 As shown.

[0030] 2. Training the x-vector extractor For the speech library The speech data of each speaker is used to extract the MFCC (Multiple Mean Cross-Counting) of each frame. Using the MFCC parameters of each frame of a sentence as input and the corresponding speaker identity as a label, an automatic speaker verifier with a TDNN (Digital Transmission Neural Network) structure is trained. This verifier serves as the x-vector extractor, with the output of a fully connected layer of the TDNN network as the x-vector. The x-vector extractor consists of a frame-level extraction module, a statistical pooling layer, a sentence-level extraction module, and an output layer. The network structure is as follows: Figure 3 As shown.

[0031] The frame-level feature extraction module, consisting of five linear layers, extracts frame-level speech features. (MFCC of each frame in a sentence) (T represents the frame number) is used as input. The TDNN linear layer extracts frame-level feature representations by performing a non-linear transformation within a sliding window on the MFCC of each input frame, and learns the correlation between frames in the speech information to capture short-term contextual information. For the MFCC of each frame of the input sentence, the output of the linear layer of the frame-level feature extraction module is: in, For the first The first frame Linear layer output, For the activation function, the ReLU function is used here. and These are the weight coefficients and biases for this layer.

[0032] The statistical pooling layer aggregates the output frame-level speech features and converts the variable-length speech features into fixed-length speech features by calculating their mean and standard deviation.

[0033] The sentence-level feature extraction module consists of two fully connected layers. It further performs nonlinear transformation on the output of the statistical pooling layer to extract higher-order speech features.

[0034] The output layer consists of a fully connected layer and a softmax classifier, which transforms the speech features output by the sentence-level feature extraction module into probabilities for each speaker's personality. The x-vector extractor uses the output of the first fully connected layer of the sentence-level feature extraction module as the speaker feature x-vector.

[0035] During the training phase, the cross-entropy between the predicted probability distribution of other human speakers and the true labels is used as the loss function. The network parameters are updated using gradient descent. The gradient of the loss function with respect to the network parameters is calculated, and the parameters are updated along the negative gradient direction through backpropagation. After several iterations, the network parameters tend to converge.

[0036] 3. Train the Batch Normalization (BN) vector extractor For the speech library The speech data from each speaker is processed, and the MFCC (Multiple Mean Cross-Counting) of each frame is extracted. The MFCC of each speaker in each frame is concatenated with the speaker's i-vector and used as the network input. The phoneme sequence of each frame is used as the label. A TDNN-F structured automatic speech recognizer is trained, which serves as the BN (Browser Normalization) vector extractor. The output of a linear layer of the network is used as the BN vector. The BN vector extractor consists of a TDNN-F linear layer and an output layer, as shown in the network structure below. Figure 4 As shown.

[0037] The MFCC and i-vector are concatenated and used as the input to the first TDNN-F linear layer. Each TDNN-F linear layer performs a non-linear transformation within a sliding window on the input to extract speech features and capture contextual information. Specifically, the linear layer first performs a linear transformation on the input, then passes it through an activation function to obtain the layer's output. The linear transformation extracts speech features, and the activation function introduces non-linear characteristics. The output of the linear layer is represented as: in, For the first Linear layer output, For the activation function, the ReLU function is used here. and These are the weights and biases for this layer. Each linear layer takes the output of the previous linear layer as input; as the number of layers increases, the linear layers can learn a wider range of contextual temporal relationships. The output layer transforms the speech features extracted by the linear layers into probabilities for each phoneme category. The BN vector extractor uses the output of the last linear layer as the BN vector.

[0038] During network training, the cross-entropy between the phoneme prediction distribution and the true labels is used as the loss function. The network parameters are updated using gradient descent. The gradient of the loss function with respect to the network parameters is calculated, and the parameters are updated along the negative gradient direction through backpropagation. After several iterations, the network parameters tend to converge.

[0039] 4. Training the adversarial example generator For the speech library The speech data of each speaker is processed by an x-vector extractor to extract the x-vector of each sentence. Using the x-vector as input and the corresponding speaker identity as a label, an x-vector classifier is trained, i.e.: in, Let x be the input x-vector, and y be the predicted probability distribution of each speaker's class. The training process uses the cross-entropy between the predicted distribution and the actual speaker labels as the loss function. The network parameters are updated using gradient descent. The gradient of the loss function with respect to the network parameters is calculated, and the parameters are updated along the negative gradient direction through backpropagation. After several iterations, the network parameters tend to converge.

[0040] Based on the speech database An x-vector extracted from the speech data of each speaker is used to train a privacy risk estimator to estimate the privacy risk value of each element of the x-vector. First, a small amount of noise is added to each x-vector to generate explanatory samples. For each explanatory sample, an x-vector classifier is used for classification. Using the x-vector and the elements of each dimension of the explanatory sample as input, and the classification result of the x-vector classifier as the label, a random forest model is trained to reclassify the x-vector and the explanatory sample. Training uses the cross-entropy of the predicted distribution and the label as the loss function, and gradient descent is used to obtain the parameters of the random forest model. Finally, the mean of the feature importance of each element of the x-vector in each decision tree of the random forest is calculated; this mean is the privacy risk value estimated by the privacy risk estimator for each element of the x-vector.

[0041] 5. Train the MFCC generator For the speech library Speech data from one speaker is processed using an F0 extractor, a BN vector extractor, and an x-vector extractor to extract F0, BN, and x-vectors, respectively; MFCCs are extracted using an MFCC extractor. The extracted F0, BN, and x-vectors are concatenated as input, and MFCCs are used as labels to train a neural network-based MFCC generator to generate MFCCs. The MFCC generator consists of fully connected layers, a bidirectional long short-term memory (Bi-LSTM) network layer, an autoregressive layer, a long short-term memory (LSTM) network layer, and a Highway-Postnet layer. The MFCC generator network structure is as follows: Figure 5 As shown.

[0042] The F0, BN vectors, and x-vector are concatenated and used as input to the fully connected layer. The fully connected layer performs a linear transformation on the input to extract features and then uses the ReLU function to perform a non-linear transformation on the features. The Bi-LSTM layer models the feature sequence, capturing contextual information within it. The autoregressive layer recursively uses the output from the previous time step to generate the output for the current time step, thus generating the output sequence. The LSTM layer, through a combination of input gates, forget gates, cell state gates, and output gates, captures and transmits information from the sequence data, further modeling the feature sequence and addressing the vanishing and exploding gradient problems. The Highway-PostNet layer uses the gating mechanism of the Highway layer to control information transmission and uses the PostNet layer to post-process the generated MFCCs, improving synthesis performance and ultimately outputting the generated MFCCs.

[0043] During training, the minimum mean square error of the generated MFCC and the actual MFCC is used as the loss function. The network parameters are updated using gradient descent. The gradient of the loss function with respect to the network parameters is calculated. Through backpropagation, the parameters are updated along the negative gradient direction. After several iterations, the network parameters tend to converge.

[0044] 6. Training vocoder For the speech library The speech data of each speaker is used to extract MFCC, x-vector, and F0 as input to the vocoder. The speech signal time series, i.e., the speech waveform, is used as a label to train the vocoder. The vocoder is a neural source filter, consisting of a conditional module, a source module, and a filter module, which converts the input acoustic features into the output speech waveform. The structure is as follows: Figure 6 As shown.

[0045] The conditional module consists of a Bi-LSTM layer, a convolutional layer, and an upsampling module. It upsamples the input speech feature sequence into a speech feature sequence with a waveform sampling point length. Assume the length of the input speech feature sequence is B, and the waveform sampling point length is T. sequence By copying each frame (T / B) times, a fundamental frequency sequence of length T is obtained. After passing the MFCC and x-vector sequences through Bi-LSTM layers and convolutional layers, they are upsampled into speech feature sequences of length T. The source module receives the upsampled sequence. After sequencing, a sinusoidal excitation signal is generated and sent to the filter module. That is: in, obtained by upsampling sequence, The initial phase is random. The sampling frequency of the output waveform. It is Gaussian noise. The hyperparameter for controlling the amplitude is set to 0.1.

[0046] The filter module consists of six convolutional and affine transform modules, which convert the received excitation signal into the output speech waveform. The entire process uses minimizing the Euclidean distance between the generated waveform sequence and the actual waveform sequence as the loss function. The network parameters are updated using gradient descent. The gradient of the loss function with respect to the network parameters is calculated, and the parameters are updated along the negative gradient direction through backpropagation. After several iterations, the network parameters tend to converge.

[0047] 7. Identity Privacy Protection Algorithms and Procedures 1) Establish a system containing A speech database of individual speaker speech data is used as the training set; 2) According to Speech data from each speaker was used to build a GMM model for each speaker, and the GMM model parameters were obtained using the EM algorithm. A GMM model is generated. The mean vectors of the Gaussian components of each GMM are concatenated and projected onto the global difference space to calculate the corresponding i-vector, ultimately yielding a result. A GMM and An i-vector estimator is constructed from corresponding i-vectors. An automatic speech recognizer with a TDNN-F structure is trained as the BN vector extractor. An automatic speaker verifier with a TDNN structure is trained as the x-vector extractor. An x-vector classifier and a privacy risk value estimator are trained to construct... Loss function for adversarial example generator. Training an acoustic model with a neural network architecture as an MFCC synthesizer. Training a neural source filter as a vocoder.

[0048] 3) For any user, only a voice input is required. The voice signal is first pre-emphasized in the high-frequency portion by a high-pass filter, resulting in a voice signal with a relatively flat spectral amplitude. Then, the system uses a fixed-length sliding window to divide the pre-emphasized signal into short frames. After dividing the signal into short frames, each frame is multiplied by a Hamming window function to increase the continuity at both ends of each frame and reduce spectral energy leakage. Finally, each frame is subjected to a Fast Fourier Transform (FFT) to obtain the spectrum of each frame.

[0049] 4) The preprocessed signal is passed through the F0 extractor, which extracts the fundamental frequency of each frame. The F0 extractor first calculates the... Frame signal The short-time autocorrelation function, i.e.: in, The length of each frame of signal, For time delay, The maximum time delay is set. Then the calculation of the... Frame signal The variable-length average amplitude difference function, i.e.: In the formula, For the length of each frame, For time delay, The maximum time delay is calculated. Finally, the quasi-periodic function is calculated. : in, The hyperparameter is set to 10. The lower limit of the fundamental frequency is set to 60Hz. The upper limit of the fundamental frequency is set to 500Hz. When... When the maximum value is taken within the set fundamental frequency range, the sampling frequency is... In addition to the time delay at this time That is, the first F0 of the frame signal.

[0050] 5) The preprocessed signal is passed through an MFCC extractor, which uses a Mel-frequency filter bank to extract the MFCCs. The obtained MFCCs are then passed through an i-vector estimator to estimate the user's i-vector. The i-vector estimator first calculates the MFCC sequence in... The probabilities under each GMM model, i.e.: in, The length of the MFCC vector sequence. The MFCC feature vector; For the first The parameters of the GMM model are obtained. After finding the probabilities, The speaker corresponding to the maximum of the probabilities ,Right now: speaker The corresponding i-vector is the estimated user i-vector.

[0051] 6) The obtained MFCC is processed by the x-vector extractor, which extracts the x-vector by taking the output of the first fully connected layer of the sentence-level feature extraction module in the network as the speaker feature x-vector.

[0052] 7) The obtained MFCC and estimated i-vector are passed through the BN vector extractor, which uses the output of the last linear layer in the network as the BN vector to extract the BN vector.

[0053] 8) The extracted x-vector is processed by three adversarial example generators based on different norm constraints. Anonymized x-vectors are generated by adding adversarial perturbations based on different norm constraints to the x-vector. First, pseudo-speaker labels are randomly selected for the generator's anonymized x-vector. (Pseudo-speaker tag) (Cannot be labeled as a real speaker), that is: Norm adversarial sample generator, by feeding input Add based on Adversarial noise with norm constraints, generating adversarial examples ,Right now: Specifically, the generator first calculates the loss function. right The gradient and loss function are: The first term of the loss function is minimized. and Euclidean distance to limit and The difference, the second term encourages the classifier to use adversarial examples. Classified as a pseudo-speaker To ensure the perturbation effect, among which, For classifier adversarial examples Classified as speaker The probability, To eliminate false speakers In addition, the maximum value of the classification probability of other speakers; It is an exponential function. The base of the loss function is 2, and its value is 1; when the second term of the loss function is less than or equal to 0, the classifier is considered to have handled adversarial examples. Misclassified as a pseudo-speaker ; Hyperparameters are used to adjust Norm constraints and the weights of perturbation effects.

[0054] Calculate the loss function pair After obtaining the gradient, it is propagated back to... Add based on Adversarial perturbations of norm constraints make Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: in This is the step size for each iteration. Set it to 0.001. It is a signed function. It takes a real value as its parameter, and returns 1 if the input is greater than 0, 0 if the input is equal to 0, and -1 if the input is less than 0.

[0055] 9) Norm adversarial sample generator, by sending... Add based on Adversarial noise with norm constraints, generating adversarial examples ,Right now: Specifically, the generator limits the number of elements that can be modified. and The difference lies in the fact that the generator first estimates the input using a privacy risk value estimator. The privacy risk value of each element is calculated, and these values ​​are sorted in descending order. The set of elements with the highest privacy risk values ​​at the top of the sorted list is defined as a privacy risk region when the sum of their privacy risk values ​​accounts for k% of the sum of all element privacy risk values. Modifications are limited to elements within this privacy risk region, thus restricting the number of elements that can be modified.

[0056] The generator then calculates the loss function. The loss function is: (The loss function is used to calculate the gradient of elements within the privacy-risk region.) The loss function encourages the classifier to... Classified as pseudo-speaker tag To ensure the perturbation effect, the gradient of the loss function with respect to elements within the privacy risk region is calculated, and then backpropagated to... Adding adversarial perturbations to make Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: 10) Norm adversarial sample generator, by sending... Add based on After applying adversarial noise with norm constraints, maximum values ​​are clipped to generate adversarial examples. ,Right now: Specifically, the generator limits the maximum extent to which elements can be modified. and The difference lies in the generator's first calculation of the loss function. right The gradient and loss function are: The first term of the loss function is appropriately constrained by minimizing the Euclidean distance. and The difference, the second term encourages the classifier to Classified as pseudo-speaker tag . Hyperparameters are used to adjust The weights for norm constraints and perturbation effects should be appropriately increased here. Calculate the loss function... After obtaining the gradient, it is propagated back to... Add based on After adversarial perturbation of the norm constraint, maximum pruning is performed, such that... Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: 11) Then, the outputs of the three adversarial example generators are fused using a parallel ensemble method, i.e.: Determined by gradient descent method , , The value of . Specifically, the loss function is calculated for . , , The gradient. The loss function is: The loss function encourages the classifier to... Classified as pseudo-speaker tag This is to ensure the perturbation effect. The loss function is calculated for... , , After obtaining the gradient, it is backpropagated to make , , Update along the negative gradient direction, and after several iterations, make... , , It tends to converge, that is: 12) Finally, F0, BN vector, and anonymized x-vector are passed through the MFCC generator to generate MFCC, and then input together with F0 and anonymized x-vector into the vocoder to synthesize anonymized speech.

[0057] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A system for protecting identity privacy for voice data, characterized in that: include: The speech feature extraction module extracts the F0, MFCC, BN vector and x-vector of the user's speech signal through the F0 extractor, MFCC extractor, BN vector extractor and x-vector extractor respectively; Modify the x-vector module to generate an anonymized x-vector by passing the obtained x-vector through three independently trained adversarial example generators; The three adversarial example generators generate adversarial examples by establishing different loss functions, calculating the gradient of each loss function with respect to the x-vector, adding adversarial perturbations based on the gradient and perturbation coefficients to the x-vector, causing the x-vector to be updated along the negative gradient direction, iterating the x-vector until it converges, and then using a parallel ensemble method to fuse the outputs of the three adversarial example generators as an anonymized x-vector. This anonymized x-vector can cause the attacker's speaker classifier to misclassify the speaker, making it impossible for the attacker's speaker classifier to distinguish the original speaker's identity, thus achieving the purpose of anonymization. The extracted x-vector is anonymized by adding adversarial perturbations based on different norm constraints to the x-vector through three adversarial example generators based on different norm constraints. Norm adversarial sample generator Norm adversarial sample generator and Norm adversarial sample generator, the The norm adversarial sample generator first estimates the input through a privacy risk value estimator. The privacy risk value of each element is calculated, and the privacy risk values ​​of each element are sorted in descending order. This refers to the input x-vector; The speech synthesis module is used to generate MFCCs from F0, BN vectors and anonymized x-vectors through the MFCC generator, and then input the generated MFCCs, F0 and anonymized x-vectors into the vocoder to synthesize anonymized speech.

2. The identity privacy protection system for voice data as described in claim 1, characterized in that: The F0 extractor utilizes the time-domain information of speech to determine the pitch period by calculating the autocorrelation function and the variable-length average amplitude difference function, thereby determining the F0 parameters. The MFCC extractor uses a Mel frequency filter bank to extract the MFCC. The obtained MFCC is used to estimate the i-vector by an i-vector estimator and to extract the x-vector by an x-vector extractor. Then, the obtained MFCC and i-vector are used to extract the BN vector by a BN vector extractor.

3. The identity privacy protection system for voice data as described in claim 2, characterized in that: During the training of the i-vector estimator, arbitrary vectors are extracted from the speech database. The speech data of each speaker is used to model the probability distribution density of the feature parameter space of each speaker's speech using a Gaussian mixture model. The mean vectors of the Gaussian components of each Gaussian mixture model are concatenated and projected onto the global difference space to calculate their corresponding i-vectors. Finally, the resulting data is... A GMM model and An i-vector estimator consisting of 1 corresponding i-vector.

4. The identity privacy protection system for voice data as described in claim 2, characterized in that: The x-vector extractor consists of a frame-level extraction module, a statistical pooling layer, a sentence-level extraction module, and an output layer. During training, it extracts data from the speech database. The speech data of each speaker is used to extract the MFCC of each frame of speech. The MFCC parameters of each frame of a sentence are used as input, and the corresponding speaker identity is used as the label. An automatic speaker verifier with a TDNN structure is trained as the x-vector extractor. The output of a fully connected layer of the TDNN network is used as the x-vector.

5. The identity privacy protection system for voice data as described in claim 1, characterized in that: The BN vector extractor consists of a TDNN-F linear layer and an output layer. During training, it is trained on the speech database. The speech data of each speaker is used to extract the MFCC of each frame. The MFCC of each frame of each speaker and the speaker's i-vector are concatenated and used as the input of the network. The phoneme sequence of each frame is used as the label. An automatic speech recognizer with a TDNN-F structure is trained as the BN vector extractor. The output of a linear layer of the network is used as the BN vector.

6. The identity privacy protection system for voice data as described in claim 1, characterized in that: The MFCC generator consists of a fully connected layer, a bidirectional long short-term memory (LSTM) network layer, an autoregressive layer, another LSM network layer, and a Highway-PostNet layer. The F0, BN vectors, and x-vectors are concatenated and used as input to the fully connected layer. The fully connected layer performs a linear transformation on the input to extract features and then uses the ReLU function to perform a non-linear transformation on the features. The Bi-LSTM layer models the feature sequence and captures the contextual information within it. The autoregressive layer uses a recursive method to generate the output of the current time step using the output of the previous time step, thus generating the output sequence. The LSTM layer captures and transmits information from the sequence data through a combination of input gates, forget gates, cell states, and output gates. The Highway-PostNet layer controls the transmission of information through the gating mechanism of the Highway layer and uses the PostNet layer to post-process the generated MFCCs, finally outputting the generated MFCCs.

7. The identity privacy protection system for voice data as described in claim 1, characterized in that: For the speech library The speech data of a speaker is used to extract MFCC, x-vector, and F0 as inputs to a vocoder. The speech signal time series is used as labels to train the vocoder. The vocoder is a neural source filter consisting of a conditional module, a source module, and a filter module. It converts the input acoustic features into an output speech waveform. The conditional module consists of a Bi-LSTM layer, a convolutional layer, and an upsampling module. F0 is directly upsampled. The MFCC and x-vector sequences are passed through the Bi-LSTM layer and convolutional layer, and then upsampled again to become a speech feature sequence of length T. The source module receives the feature obtained by upsampling F0. After sequencing, a sinusoidal excitation signal is generated and sent to the filter module.

8. A method for protecting identity privacy for voice data, characterized in that: Identity privacy is protected by the identity privacy protection system for voice data as described in any one of claims 1-7.

9. A method for protecting identity privacy for voice data as described in claim 8, characterized in that: When a user uses the system, they only need to input voice data. The system first decouples the speaker representation and semantic information of the voice signal. Then, through three independently trained adversarial example generators, adversarial perturbations based on different norm constraints are added to the x-vector to modify the speaker representation x-vector. The outputs of the three adversarial example generators are then fused together using a parallel ensemble method as an anonymized x-vector. This anonymized x-vector replaces the original voice x-vector and is then resynthesized, so the user can obtain anonymized voice data.

10. A method for protecting identity privacy for voice data as described in claim 9, characterized in that: Includes the following steps: 1) For any user, inputting a speech signal, the speech signal is first pre-emphasized in the high-frequency part by passing through a high-pass filter to obtain a speech signal with a relatively flat spectral amplitude. Then, the system uses a sliding window of fixed length to divide the pre-emphasized signal into short time frames. After dividing the signal into short time frames, each frame signal is multiplied by a Hamming window function, and each frame signal is then subjected to a fast Fourier transform to obtain the spectrum of each frame signal. 2) The preprocessed signal is passed through the F0 extractor to extract the fundamental frequency of each frame; the F0 extractor first calculates the... Frame signal The short-time autocorrelation function is then calculated, followed by the calculation of the first... Frame signal The variable-length average amplitude difference function is used to calculate the quasi-periodic function. ,when When the maximum value is taken within the set fundamental frequency range, the sampling frequency is... In addition to the time delay at this time That is, the first F0 of the frame signal; 3) The preprocessed signal is passed through an MFCC extractor, which uses a Mel-frequency filter bank to extract the MFCCs. The obtained MFCCs are then used by an i-vector estimator to estimate the user's i-vector. The i-vector estimator first calculates the MFCC sequence in... The probabilities under each GMM model are obtained. After finding the probabilities, The speaker corresponding to the maximum of the probabilities Speaker The corresponding i-vector is the estimated user i-vector; 4) The obtained MFCC uses the output of the first fully connected layer of the sentence-level feature extraction module in the network as the speaker feature x-vector through the x-vector extractor to extract the x-vector; 5) The obtained MFCC and estimated i-vector are passed through the BN vector extractor, which uses the output of the last linear layer in the network as the BN vector to extract the BN vector; 6) The extracted x-vector is processed by three adversarial example generators based on different norm constraints. By adding adversarial perturbations based on different norm constraints to the x-vector, anonymized x-vectors are generated. Specifically, pseudo-speaker labels are first randomly selected for the anonymized x-vector. ,Then Norm adversarial sample generators generate samples by feeding inputs to the input. Add based on Adversarial noise with norm constraints, generating adversarial examples ,Right now: （1） Specifically The norm adversarial sample generator first calculates the loss function. right The gradient and loss function are: （2） The first term of the loss function is minimized. and Euclidean distance to limit and The difference, the second term encourages the classifier to use adversarial examples. Classified as a pseudo-speaker To ensure the perturbation effect, among which, For classifier adversarial examples Classified as speaker The probability, To eliminate false speakers In addition, the maximum value of the classification probability of other speakers; It is an exponential function. The base of the loss function is 2, and its value is 1; when the second term of the loss function is less than or equal to 0, the classifier is considered to have handled adversarial examples. Misclassified as a pseudo-speaker ; Hyperparameters are used to adjust The weights of norm constraints and perturbation effects; Calculate the loss function pair After obtaining the gradient, it is propagated back to... Add based on Adversarial perturbations of norm constraints make Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: （3） （4） in, This indicates that, for classifiers The pseudo-speaker label is The adversarial examples generated by the perturbation are The loss function; For differential vector operators, Indicates the input vector For each element of the loss function Find the partial derivative; This is the step size for each iteration; set it to 0.

001. It is a signed function that takes a real value as its parameter. If the input is greater than 0, it returns 1; if the input is equal to 0, it returns 0; if the input is less than 0, it returns -1. 7) Norm adversarial sample generators generate samples by sending... Add based on Adversarial noise with norm constraints, generating adversarial examples ,Right now: （5） Specifically Norm adversarial sample generators limit the number of modified elements. and differences The norm adversarial sample generator first estimates the input through a privacy risk value estimator. The privacy risk value of each element is calculated, and the privacy risk values ​​of each element are sorted in descending order. When the sum of the privacy risk values ​​of the elements with the largest privacy risk values ​​at the top of the sort is k% of the sum of the privacy risk values ​​of all elements, the set of elements with the largest privacy risk values ​​is defined as the privacy risk region. The number of elements that can be modified is limited by allowing modification only to elements within the privacy risk region. The norm adversarial example generator then calculates the loss function. The loss function is: (The loss function is used to calculate the gradient of elements within the privacy-risk region.) （6） The loss function encourages the classifier to... Classified as a pseudo-speaker To ensure the perturbation effect; after calculating the gradient of the loss function with respect to elements in the privacy risk region, it is then propagated back to... Adding adversarial perturbations to make Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: （7） （8） 8) Norm adversarial sample generators generate samples by sending... Add based on After applying adversarial noise with norm constraints, maximum values ​​are clipped to generate adversarial examples. ,Right now: （9） Specifically Norm adversarial sample generators limit the maximum magnitude of modifications to elements. and differences The norm adversarial sample generator first calculates the loss function. right The gradient and loss function are: （10） The first term of the loss function is appropriately constrained by minimizing the Euclidean distance. and The difference, the second term encourages the classifier to Classified as pseudo-speaker tag ; Hyperparameters are used to adjust Norm constraints and weights of perturbation effects; calculation of the loss function. After obtaining the gradient, it is propagated back to... Add based on After adversarial perturbation of the norm constraint, maximum pruning is performed, such that... Update along the negative gradient direction to generate adversarial examples. And after several rounds of iteration, it became possible to achieve this. It tends to converge, that is: （11） （12） 9) Then, the outputs of the three adversarial example generators are fused using a parallel ensemble method, i.e.: （13） Determined by gradient descent method , , The value of , specifically, is calculated for the loss function. , , The gradient and loss function are: （14） The loss function encourages the classifier to... Classified as pseudo-speaker tag To ensure the perturbation effect; calculate the loss function for... , , After obtaining the gradient, it is backpropagated to make , , Update along the negative gradient direction, and after several iterations, make... , , It tends to converge, that is: （15） （16） （17） 10) Finally, F0, BN vector, and anonymized x-vector are passed through the MFCC generator to generate MFCC, and then input together with F0 and anonymized x-vector into the vocoder to synthesize anonymized speech.

Images