A multi-defense method and system for voiceprint identity based on anonymization

Abstract

This invention discloses a multi-defense method and system for voiceprint identity based on anonymization. The method first extracts voiceprint features from each speaker's original voiceprint data, converts the extracted voiceprint feature vectors of each speaker into a probability distribution using the Softmax function, and forms an initial voiceprint feature library. A Gaussian mixture model is then used to cluster the voiceprint feature library, selecting the cluster center with the greatest difference from the original voiceprint data. For each cluster center, the KL divergence between its probability distribution and the probability distribution of the original voiceprint data is calculated. Finally, for each cluster center, a weight is calculated based on its KL divergence with the original voiceprint data, and a new voiceprint feature vector is synthesized using a weighted average method. This invention utilizes a differentiated defense mechanism based on KL divergence, an advanced initialization weighting method, and an emphasis on privacy protection, all of which significantly improve the system's defense effectiveness and practicality.

Description

Technical Field

[0001] The present invention belongs to the field of voiceprint recognition and security defense, and relates to a multi-defense method and system for voiceprint identity based on anonymization, and in particular to a multi-defense method and system for voiceprint identity that enhances privacy protection and prevents voice forgery through advanced voiceprint feature processing and analysis technology. Background Art

[0002] The rapid development of deep neural networks and generative artificial intelligence (AI) has led to the rise of realistic voice conversion technology. While these advances have driven innovation in a variety of applications, such as intelligent assistants and personalized user interfaces, they have also posed significant security threats. Spoofed speech poses a serious threat to social security and personal privacy.

[0003] Fake speech generation technologies, especially deep learning-based text-to-speech (TTS) and voice conversion (VC) technologies, have reached a highly realistic level and can imitate the voice of a specific speaker without being detected by the human ear.

[0004] Despite the existence of numerous detection and defense technologies, existing defense methods appear to be powerless in the face of complex and diverse attack methods. Current defense strategies are mainly divided into two types: passive defense and active defense. Passive defense mainly focuses on feature-based detection and data-based detection methods. Feature-based detection mainly analyzes the differences in biometric information in forged speech, such as voice signals, voiceprint features, and spectral distribution, and uses these differences to distinguish authenticity. For example, a detector that combines cochlear filter inverse spectral coefficients (CFCC) and instantaneous frequency variation (IF) can be used to distinguish natural speech from deceptive speech. However, this frame-by-frame feature extraction method has difficulty in grasping the temporal dimension connection between data in complex forgery situations.

[0005] In contrast to passive defense, active defense strategies aim to directly intervene in the forgery process, such as by embedding digital watermarks or applying adversarial perturbations to protect sound data from misuse. Digital watermarking techniques, such as those based on discrete cosine transform (DCT) and singular value decomposition (SVD), can generate two new features: piecewise singular value sum (SSVS) and piecewise singular value difference (SSVD) to achieve robust stereo signal watermarking. Although this approach can achieve a certain balance between fidelity and robustness, it is less adaptable to cropping and time scaling and has difficulty coping with complex sound cloning processes.

[0006] To more effectively address these challenges, the "Timbre Watermarking" method was proposed. This technology embeds watermark information into the timbre of the target individual and implements it through an end-to-end anti-cloning detection framework, thereby enhancing resistance to voice conversion attacks.

[0007] While these existing methods have shown promise in increasing the difficulty of forgery and improving system security, their efficiency and feasibility in practical applications remain challenging. In particular, most defense strategies, while focusing on forged speech, can inadvertently make the voice characteristics resemble those of a genuine, but unknown, third-party speaker, potentially leaking information about the third-party speaker's voice. This presents not only a new challenge for audio security but also highlights the need for more precise control over voice feature adjustments to avoid inadvertent privacy leaks and new security risks. Summary of the Invention

[0008] To solve the above technical problems, the present invention proposes a multi-defense method and system for voiceprint identity based on anonymization, which is used to protect the voiceprint characteristics of an individual during voice interaction and prevent the voiceprint data from being maliciously imitated or tampered with.

[0009] The technical solution adopted by the method of the present invention is: a multi-defense method for voiceprint identity based on anonymization, characterized by comprising the following steps:

[0010] Step 1: First, extract the voiceprint features of each speaker's original voiceprint data, and use the Softmax function to convert the extracted voiceprint feature vectors of each speaker into a probability distribution to form an initial voiceprint feature library;

[0011] Step 2: Use the Gaussian mixture model to perform cluster analysis on the voiceprint feature library and select the cluster center with the largest difference from the original voiceprint data;

[0012] Step 3: For each cluster center, calculate the KL divergence between its probability distribution and the probability distribution of the original voiceprint data;

[0013] Step 4: For each cluster center, calculate the weight based on its KL divergence with the original voiceprint data, and synthesize a new voiceprint feature vector through the weighted average method.

[0014] As a preference, in step 1, the Softmax function is used to transform the feature vectors V of N speakers into k Convert to probability distribution P k (x ki );

[0015]

[0016] Among them, x ki is the feature vector V of the kth speakerk The i-th element in n k is the eigenvector V k The dimension of , T is the temperature parameter.

[0017] As a preference, in step 3, the probability distribution P(x0) of the original voiceprint data x0 and the probability distribution Q(μ k ) between the KL divergence D KL (P(x0)||Q(μ k )) to measure the difference between the original voiceprint and each cluster center;

[0018]

[0019]

[0020] Among them, μ k is the mean vector of each cluster center; x 0i is the feature vector V of the protected speaker k The i-th element in μ ki is the feature vector V of the selected speaker k The i-th element in n k is the eigenvector V k The dimension of .

[0021] Preferably, in step 4, a new voiceprint feature vector is synthesized by a weighted average method, and the weight is continuously adjusted through an iterative update strategy to find the optimal weight configuration with the largest difference from the original voiceprint feature;

[0022] The specific implementation includes the following steps:

[0023] Step 4.1: Assign an initial weight to each cluster center using random initialization And regularize it so that its sum is 1;

[0024]

[0025] Among them, v k is a randomly generated positive value from a uniform or normal distribution, and K is the number of cluster centers;

[0026] Step 4.2: Calculate the weighted average voiceprint feature vector based on the weight and the KL divergence obtained in step 3

[0027]

[0028] in, is the new voiceprint feature vector in the tth iteration, μ k is the mean vector of the k-th cluster center, is the weight in the tth iteration;

[0029] Step 4.3: Calculate the new voiceprint feature vector KL divergence D with the original voiceprint data x0 KL , and accordingly update the weights of each cluster center to maximize the difference between the synthesized features and the original voiceprint until the preset convergence condition is reached;

[0030]

[0031] Among them, x 0i is the feature vector V of the protected speaker k The i-th element in is the new voiceprint feature vector in the tth iteration, n k is the eigenvector V k , P(x0) is the probability distribution of the original voiceprint data x0, is the new voiceprint feature vector The probability distribution of

[0032] According to the calculation results of KL divergence, adjust the weight w k , to maximize the KL divergence between the original voiceprint data and the cluster center;

[0033]

[0034] in: is the weight of the kth cluster center in the tth iteration; η is the learning rate; represents the KL divergence with respect to weight w k The partial derivative of Q(μ ki ) is the probability distribution of the k-th cluster center;

[0035] Step 4.4: Check the change in KL divergence Is it less than the set threshold ∈, to determine whether the algorithm has converged: if it is not less than the set threshold, return to step 4.2; if it is less than the set threshold, return to the new voiceprint feature vector calculated at this time.

[0036] The technical solution adopted by the system of the present invention is: a multi-defense system based on anonymized voiceprint identity, including the following modules:

[0037] The feature extraction and probability distribution conversion module is used to first extract the voiceprint features of each speaker's original voiceprint data, and use the Softmax function to convert the extracted voiceprint feature vectors of each speaker into probability distribution to form an initial voiceprint feature library;

[0038] The feature selection and clustering module is used to perform cluster analysis on the voiceprint feature library using a Gaussian mixture model and select the cluster center with the greatest difference from the original voiceprint data;

[0039] The KL divergence calculation module is used to calculate the KL divergence between the probability distribution of each cluster center and the probability distribution of the original voiceprint data;

[0040] The voiceprint feature vector synthesis module is used to calculate the weight of each cluster center based on its KL divergence with the original voiceprint data, and synthesize a new voiceprint feature vector through the weighted average method.

[0041] As an advantage, the feature extraction and probability distribution conversion module uses the Softmax function to convert the feature vectors V of N speakers into k Convert to probability distribution P k (x ki );

[0042]

[0043] Among them, x ki is the feature vector V of the kth speaker k The i-th element in n k is the eigenvector V k The dimension of , T is the temperature parameter.

[0044] As an example, the KL divergence calculation module calculates the probability distribution P(x0) of the original voiceprint data x0 and the probability distribution Q(μ k ) between the KL divergence D KL (P(x0)||Q(μ k )) to measure the difference between the original voiceprint and each cluster center;

[0045]

[0046] Among them, μ k is the mean vector of each cluster center; x 0i is the feature vector V of the protected speaker k The i-th element in μ ki is the feature vector V of the selected speaker k The i-th element in n k is the eigenvector V k The dimension of .

[0047] Preferably, the voiceprint feature vector synthesis module synthesizes a new voiceprint feature vector by a weighted average method, and continuously adjusts the weights through an iterative update strategy to find the optimal weight configuration with the largest difference from the original voiceprint feature;

[0048] Specifically includes the following sub-modules:

[0049] Submodule 1 is used to assign an initialization weight to each cluster center using random initialization And regularize it so that its sum is 1;

[0050]

[0051] Among them, v k is a randomly generated positive value from a uniform or normal distribution, and K is the number of cluster centers;

[0052] Submodule 2 is used to calculate the weighted average voiceprint feature vector based on the weight and KL divergence obtained in the KL divergence calculation module

[0053]

[0054] in, is the new voiceprint feature vector in the tth iteration, μ k is the mean vector of the k-th cluster center, is the weight in the tth iteration;

[0055] Submodule 3, used to calculate the new voiceprint feature vector KL divergence D with the original voiceprint data x0 KL , and accordingly update the weights of each cluster center to maximize the difference between the synthesized features and the original voiceprint until the preset convergence condition is reached;

[0056]

[0057] Among them, x 0i is the feature vector V of the protected speaker k The i-th element in is the new voiceprint feature vector in the tth iteration, n k is the eigenvector V k , P(x0) is the probability distribution of the original voiceprint data x0, is the new voiceprint feature vector The probability distribution of

[0058] According to the calculation results of KL divergence, adjust the weight w k , to maximize the KL divergence between the original voiceprint data and the cluster center;

[0059]

[0060] in: is the weight of the kth cluster center in the tth iteration; η is the learning rate, represents the KL divergence with respect to weight w k The partial derivative of Q(μ ki ) is the probability distribution of the k-th cluster center;

[0061] Submodule 4, used to check the change of KL divergence Whether it is less than the set threshold ∈ is used to determine whether the algorithm has converged: if it is not less than the set threshold, it returns to the execution submodule 2; if it is less than the set threshold, it returns the new voiceprint feature vector calculated at this time.

[0062] Compared with the prior art, the beneficial effects of the present invention include:

[0063] 1. This invention enhances the security of voice systems by calculating and maximizing the KL divergence between the original and forged voiceprints. KL divergence is an effective measure of the difference in probability distributions, quantifying the information difference between two distributions. By maximizing KL divergence, it ensures that the forged voiceprint is significantly different from the original, making it difficult for attackers to imitate or tamper with the voiceprint data, significantly improving the system's defense capabilities.

[0064] 2. This invention uses random initialization and regularization techniques when initializing weights to make the initial weight distribution more reasonable. Initial weights are randomly generated from a uniform or normal distribution and regularized to ensure that the sum of the weights is 1. This method avoids the local optimum problem caused by improper weight initialization, optimizes the effect of weight adjustment during the iterative process, and improves the stability and convergence speed of the algorithm.

[0065] 3. This method continuously iterates and optimizes the weight configuration to find the optimal weight configuration that maximizes the KL divergence. Using gradient ascent or other optimization algorithms, the weights are dynamically adjusted to maximize the difference between the original and synthesized voiceprints. This approach ensures optimality during the iterative process, resulting in the ultimate adversarial audio with maximum differentiation, thereby enhancing defense effectiveness.

[0066] 4. This invention further ensures comprehensive privacy protection and defense by adjusting the voiceprint features to avoid matching the voice characteristics of any other real speakers. This method maximizes the KL divergence while maintaining the uniqueness and unidentifiable nature of the voiceprint features. This approach not only improves system security but also effectively protects user privacy, preventing the misuse or leakage of personal voiceprint data. By ensuring that the adjusted voiceprint features do not overlap with the voice characteristics of other speakers, forgery attacks and identity theft can be prevented. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The technical solution of the present invention is further illustrated below using embodiments and specific implementation methods. In addition, some drawings are also used in the process of illustrating the technical solution. For those skilled in the art, other drawings and the intention of the present invention can be obtained based on these drawings without making any creative efforts.

[0068] Figure 1 This is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0070] Please see Figure 1 This embodiment provides a multi-defense method for voiceprint identity based on anonymization, including the following steps:

[0071] Step 1: First, extract the voiceprint features of each speaker's original voiceprint data, and use the Softmax function to convert the extracted voiceprint feature vectors of each speaker into a probability distribution to form an initial voiceprint feature library;

[0072] In one embodiment, the specific implementation of step 1 includes the following sub-steps:

[0073] S101: Process the entire audio data set and divide the continuous voiceprint data of N speakers into a series of short time frames. Perform necessary preprocessing on each frame, including denoising and normalization. The original signal is s(t), and the preprocessed signal is represented as Where t represents the time index.

[0074] Denoising: Use filters to remove noise from the signal, for example:

[0075]

[0076] in is the estimated noise signal.

[0077] Normalization: Normalize each frame so that the signal amplitude is within a certain range, such as:

[0078]

[0079] where μ and σ are the mean and standard deviation of the signal, respectively.

[0080] S102: Extract voiceprint feature vectors from each pre-processed frame and generate a representative feature vector for each speaker. This embodiment uses Mel Frequency Cepstral Coefficients (MFCC):

[0081] MFCC = DCT(log(Mel(s)));

[0082] Where DCT stands for discrete cosine transform, and Mel means converting the spectrum into Mel frequencies.

[0083] The specific steps are:

[0084] Framing: The pre-processed signal Divide into overlapping short time frames, each frame length is N, the frame shift is M, that is:

[0085]

[0086] Windowing: Each frame of signal is multiplied by a window function w(n). In this embodiment, a Hamming window is selected to reduce spectrum leakage:

[0087]

[0088] Fourier transform: Perform a fast Fourier transform (FFT) on each windowed frame to obtain the spectrum:

[0089]

[0090] Mel filter bank: Pass the spectrum through the Mel filter bank to obtain the Mel spectrum:

[0091]

[0092] Among them, H k (f) is the frequency response of the kth Mel filter.

[0093] Logarithmic operation: Take the logarithm of the Mel spectrum:

[0094] log(Mel n (f))=log(Mel n (f));

[0095] Discrete cosine transform: Perform discrete cosine transform on the logarithmic Mel spectrum to obtain the MFCC feature vector:

[0096] MFCC n =DCT(log(Mel n (f)))

[0097] S103: Use the Softmax function to transform the feature vectors V of N speakers k(where k represents the speaker index) is converted into a probability distribution. This facilitates comprehensive analysis and comparison using, for example, KL divergence in subsequent steps:

[0098]

[0099] where x ki is the feature vector V of speaker k k The i-th element in n k is the dimension of the vector, and T is the temperature parameter, which is set to 0.1 in this embodiment to adjust the sensitivity of the Softmax function to make it more suitable for capturing subtle differences in features.

[0100] Step 2: Use the Gaussian mixture model to perform cluster analysis on the voiceprint feature library and select the cluster center with the largest difference from the original voiceprint data;

[0101] In one embodiment, the specific sub-steps of step 2 are as follows:

[0102] S201: Define the GMM model. This embodiment uses the voiceprint feature library obtained in S103, which contains N feature vectors after Softmax conversion:

[0103] P={P(x1),P(x2),...,P(x N )};

[0104] Define the probability density function of the GMM model:

[0105]

[0106] Among them, π k is the mixing coefficient of the k-th Gaussian distribution, Indicates that the mean is μ k , covariance is Σ k Gaussian distribution.

[0107] S203: Train the GMM model and randomly initialize the mixing coefficient π k , mean vector μ k and the covariance matrix Σ k .

[0108]

[0109] Where α is the parameter of the Dirichlet distribution, μ0 and Σ0 are the initial values ​​of the mean and covariance matrix, respectively.

[0110] The EM algorithm is used to iteratively update the GMM parameters until the parameters converge to the optimal solution.

[0111] Step E (expectation):

[0112]

[0113] Among them, γ ik Represents the sample P(x i ) belongs to the expectation of the k-th Gaussian distribution.

[0114] M step (maximization step):

[0115]

[0116] Repeat the E and M steps until the parameter changes meet the convergence conditions:

[0117] |L (t+1) -L (t) |<∈;

[0118] Among them, L (t) is the log-likelihood of the t-th iteration, ∈ is the convergence threshold, which is set to 10 -4 .

[0119] Step 3: For each cluster center, calculate the KL divergence between its probability distribution and the probability distribution of the original voiceprint data; this calculation helps quantify the information differences between different voiceprints and provides a scientific basis for feature selection.

[0120] In one embodiment, the probability distribution P(x0) of the original voiceprint data x0 and the probability distribution Q(μ k ) between the KL divergence D KL (P(x0)||Q(μ k )) to measure the difference between the original voiceprint and each cluster center;

[0121]

[0122] Among them, μ k is the mean vector of each cluster center; x 0i is the feature vector V of the protected speaker k The i-th element in μ ki is the feature vector V of the selected speaker k The i-th element in n k is the eigenvector V k The dimension of .

[0123] Step 4: For each cluster center, a weight is calculated based on its KL divergence with the original voiceprint data, and a new voiceprint feature vector is synthesized using a weighted average method. This step is the core of the entire defense strategy, aiming to provide a sufficiently anonymized feature learning direction for subsequent adversarial audio generation. The weights are continuously adjusted through an iterative update strategy to find the optimal weight configuration that maximizes the difference between the original voiceprint features and the original voiceprint features.

[0124] In one embodiment, a new voiceprint feature vector is synthesized by a weighted average method, and the weights are continuously adjusted through an iterative update strategy to find the optimal weight configuration with the largest difference from the original voiceprint feature;

[0125] The specific implementation includes the following steps:

[0126] Step 4.1: Assign an initial weight to each cluster center using random initialization And regularize it so that its sum is 1;

[0127]

[0128] Among them, v k is a randomly generated positive value from a uniform or normal distribution, and K is the number of cluster centers;

[0129] Step 4.2: Calculate the weighted average voiceprint feature vector based on the weight and the KL divergence obtained in step 3

[0130]

[0131] in, is the new voiceprint feature vector in the tth iteration, μ k is the mean vector of the k-th cluster center, is the weight in the tth iteration;

[0132] Step 4.3: Calculate the new voiceprint feature vector KL divergence D with the original voiceprint data x0 KL , and accordingly update the weights of each cluster center to maximize the difference between the synthesized features and the original voiceprint until the preset convergence condition is reached;

[0133]

[0134] Among them, x 0i is the feature vector V of the protected speaker k The i-th element in is the new voiceprint feature vector in the tth iteration, n k is the eigenvector V k , P(x0) is the probability distribution of the original voiceprint data x0, is the new voiceprint feature vector The probability distribution of

[0135] According to the calculation results of KL divergence, adjust the weight w k , to maximize the KL divergence between the original voiceprint data and the cluster center;

[0136]

[0137] in: is the weight of the kth cluster center in the tth iteration; η is the learning rate, which is set to 0.01; represents the KL divergence with respect to weight w k The partial derivative of Q(μ ki ) is the probability distribution of the k-th cluster center;

[0138] Step 4.4: Check the change in KL divergence Is it less than the set threshold ∈, to determine whether the algorithm has converged: if it is not less than the set threshold, return to step 4.2; if it is less than the set threshold, return to the new voiceprint feature vector calculated at this time.

[0139] This embodiment also provides a multi-defense system for voiceprint identity based on anonymization, including the following modules:

[0140] The feature extraction and probability distribution conversion module is used to first extract the voiceprint features of each speaker's original voiceprint data, and use the Softmax function to convert the extracted voiceprint feature vectors of each speaker into probability distribution to form an initial voiceprint feature library;

[0141] The feature selection and clustering module is used to perform cluster analysis on the voiceprint feature library using a Gaussian mixture model and select the cluster center with the greatest difference from the original voiceprint data;

[0142] The KL divergence calculation module is used to calculate the KL divergence between the probability distribution of each cluster center and the probability distribution of the original voiceprint data;

[0143] The voiceprint feature vector synthesis module is used to calculate the weight of each cluster center based on its KL divergence with the original voiceprint data, and synthesize a new voiceprint feature vector through the weighted average method.

[0144] The present invention is further described below through experiments.

[0145] In order to prove the anti-cloning capability of the present invention (the difference between the cloned voice after defense and the original voice is, in principle, the larger the better), the Defend index is used to quantify the anti-cloning capability.

[0146] (1) Defense success rate Defend;

[0147] In order to minimize the difference between the original speech and the synthesized speech obtained by inputting the defense speech into the voice cloning model, so that the voice of the correct target speaker cannot be synthesized, the defense success rate Defend is defined to measure the difference between the two.

[0148] When calculating Defend, we first calculate the cosine similarity of the feature vectors between the original speech and the synthesized speech generated by the speech cloning model, and then take the complement of the similarity (i.e. 1 minus the similarity value). The calculation formula is as follows:

[0149]

[0150] in, and The feature vectors representing the original speech and the defense speech synthesized using the cloning model are shown in Figure 2. This formula quantifies the similarity between the two vectors by calculating the cosine angle between them. The lower this similarity, the higher the defense value.

[0151] See Table 1. After research, we selected the VCTK, LibriSpeech, and TIMIT datasets for testing. This experiment randomly sampled speech from the data sample library. The custom dataset includes 90 speakers, 30 speakers from each source corpus, and five speech samples from each speaker, ranging from short (3-4 seconds) to long (6-7 seconds).

[0152] VCTK: The CSTR VCTK dataset contains more than 44 hours of speech data generated by 109 English speakers with various accents. The recorded phrases come from different text materials, including newspapers, language texts, and speech-rich sentences.

[0153] LibriSpeech: LibriSpeech is a large corpus containing approximately 1,000 hours of English speech. This experiment uses the test-clean subset, which contains spoken phrases from 40 English speakers.

[0154] TIMIT: TIMIT consists of recordings of 630 speakers from eight major dialects of American English, each reading 10 phonetically rich sentences.

[0155] Table 1: VCTK, LibriSpeech, TIMIT datasets

[0156]

[0157] As shown in Table 1 above, the selected speech samples cover a comprehensive set of speech contents, sample lengths, and speakers with a wide range of genders, age groups, regions, and accents.

[0158] In order to verify the anti-cloning ability of the present invention, this experiment used YourTTS and StyleTTS as test models.

[0159] YourTTS (ICML 2022): YourTTS is a zero-shot multi-speaker TTS system with multilingual capabilities, launched in 2022. It consists of three main components: the H / ASP model as the speaker encoder, the custom VITS model as the decoder, and the HiFi-GAN as the vocoder. It is the first to apply a multilingual approach to zero-shot multi-speaker voice cloning technology.

[0160] StyleTTS2 (NIPS 2023): StyleTTS2 is a text-to-speech (TTS) model that leverages diffusion and adversarial training of speech attributes (intonation, timbre, emotion) in conjunction with a large speech language model. Unlike its predecessor, StyleTTS2 models speech attributes as latent random variables through a diffusion model, thereby generating a style that best fits the text. StyleTTS2 outperforms human recordings on the single-speaker LJSpeech dataset and rivals them on the multi-speaker VCTK dataset, achieving recognition from native English speakers. Furthermore, when trained on the LibriTTS dataset, the model outperforms previously publicly available zero-shot speaker adaptation models.

[0161] In this experiment, the new voiceprint feature vector obtained using the above method is used as the target speech. Embedding loss is introduced during model training. The main purpose of this loss function is to minimize the difference between the voiceprint embedding of the generated speech and the target speech embedding. In adversarial voice generation, this loss ensures that the generated speech effectively mimics the target speech in terms of voiceprint characteristics, intentionally generating a speech that is different from the original speech but similar to the target speech.

[0162] The voiceprint embedding vectors of the generated speech and the target speech are set as e adv and e tgt Using these two vectors, this experiment defines the embedding loss L emb As the sum of squared Euclidean distances between vectors:

[0163]

[0164] Where n represents the dimension of the embedding vector, and e adv [i] and e tgt [i] is the element of the two vectors in the i-th dimension.

[0165] This ensures the anti-cloning ability of the present invention.

[0166] This experiment uses emb attack and AntiFake active defense methods as baselines.

[0167] To eliminate the impact of defects in the synthesis model itself, this experiment compares the original speech generated by the YourTTS and StyleTTS models to clone the speech (which should be similar to the original speech) and the speech generated by the clone model after adding defense in this experiment. The corresponding Defend indicator values ​​are obtained, see Table 2.

[0168] Table 2: Defend indicators

[0169] Method YourTTS StyleTTS none 6.78％ 5.42％ emb attack 65.18％ 56.28％ AntiFake 79.24％ 73.72％ ours 87.75％ 85.83％

[0170] This invention effectively enhances the security and defense capabilities of voice systems by altering the probability distribution of voiceprint features and using KL divergence to measure and increase the difference between the original and forged voiceprints. Furthermore, this invention also takes into account the fact that the adjusted voiceprint does not resemble the voice characteristics of any other real speakers, further ensuring comprehensive privacy protection and defense.

[0171] It should be understood that the embodiments described above are only some of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention may be arbitrarily combined with each other to form a feasible technical solution. Such combination is not restricted by the order of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually inconsistent or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0172] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A multi-defense method for voiceprint identity based on anonymization, characterized in that: The following steps are involved: Step 1: First, extract the voiceprint features of each speaker's original voiceprint data, and use the Softmax function to convert the extracted voiceprint feature vectors of each speaker into a probability distribution to form an initial voiceprint feature library; Step 2: Use the Gaussian mixture model to perform cluster analysis on the voiceprint feature library and select the cluster center with the largest difference from the original voiceprint data; Define the GMM model, use the obtained voiceprint feature library, including N feature vectors P = {P(x1), P(x2), ..., P(x N )}; Define the probability density function of the GMM model Among them, π k is the mixing coefficient of the k-th Gaussian distribution, Indicates that the mean is μ k , covariance is Σ k Gaussian distribution; train the GMM model and randomly initialize the mixing coefficient π k , mean vector μ k and the covariance matrix Σ k , Where α is the parameter of the Dirichlet distribution, μ0 and Σ0 are the initial values ​​of the mean and covariance matrix respectively; the GMM parameters are iteratively updated using the EM algorithm until the parameters converge to the optimal solution; Step 3: For each cluster center, calculate the KL divergence between its probability distribution and the probability distribution of the original voiceprint data; Step 4: For each cluster center, calculate the weight based on its KL divergence with the original voiceprint data, and synthesize a new voiceprint feature vector through the weighted average method.

2. The multi-defense method for voiceprint identity based on anonymization according to claim 1 is characterized by: In step 1, the Softmax function is used to transform the feature vectors V of N speakers into k Convert to probability distribution P k (x ki ); Among them, x ki is the feature vector V of the kth speaker k The i-th element in n k is the eigenvector V k The dimension of , T is the temperature parameter.

3. The multi-defense method for voiceprint identity based on anonymization according to claim 1, characterized in that: In step 3, the probability distribution P(x0) of the original voiceprint data x0 and the probability distribution Q(μ k ) between the KL divergence D KL (P(x0)∥Q(μ k )) to measure the difference between the original voiceprint and each cluster center; Among them, μ k is the mean vector of each cluster center; x 0i is the feature vector V of the protected speaker k The i-th element in μ ki is the feature vector V of the selected speaker k The i-th element in n k is the eigenvector V k The dimension of .

4. The multi-defense method for voiceprint identity based on anonymization according to any one of claims 1 to 3, characterized in that: In step 4, a new voiceprint feature vector is synthesized through a weighted average method, and the weight is continuously adjusted through an iterative update strategy to find the optimal weight configuration with the largest difference from the original voiceprint feature; The specific implementation includes the following steps: Step 4.1: Assign an initial weight to each cluster center using random initialization And regularize it so that its sum is 1; Among them, v k is a randomly generated positive value from a uniform or normal distribution, and K is the number of cluster centers; Step 4.2: Calculate the weighted average voiceprint feature vector in, is the new voiceprint feature vector in the t-th iteration, is the weight in the tth iteration; Step 4.3: Calculate the new voiceprint feature vector KL divergence D with the original voiceprint data x0 KL , and accordingly update the weights of each cluster center to maximize the difference between the synthesized features and the original voiceprint until the preset convergence condition is reached; Among them, x 0i is the feature vector V of the protected speaker k The i-th element in is the new voiceprint feature vector in the tth iteration, n k is the eigenvector V k , P(x0) is the probability distribution of the original voiceprint data x0, is the new voiceprint feature vector The probability distribution of According to the calculation results of KL divergence, adjust the weight w k , to maximize the KL divergence between the original voiceprint data and the cluster center; in: is the weight of the kth cluster center in the tth iteration; η is the learning rate, represents the KL divergence with respect to weight w k The partial derivative of Q(μ ki ) is the probability distribution of the k-th cluster center; Step 4.4: Check the change in KL divergence Is it less than the set threshold ∈, to determine whether the algorithm has converged: if it is not less than the set threshold, return to step 4.2; if it is less than the set threshold, return to the new voiceprint feature vector calculated at this time.

5. A multi-defense system for voiceprint identity based on anonymization, characterized by: Includes the following modules: The feature extraction and probability distribution conversion module is used to first extract the voiceprint features of each speaker's original voiceprint data, and use the Softmax function to convert the extracted voiceprint feature vectors of each speaker into probability distribution to form an initial voiceprint feature library; The feature selection and clustering module is used to perform cluster analysis on the voiceprint feature library using a Gaussian mixture model and select the cluster center with the greatest difference from the original voiceprint data; Define the GMM model, use the obtained voiceprint feature library, including N feature vectors P = {P(x1), P(x2), ..., P(x N )}; Define the probability density function of the GMM model Among them, π k is the mixing coefficient of the k-th Gaussian distribution, Indicates that the mean is μ k , covariance is Σ k Gaussian distribution; train the GMM model and randomly initialize the mixing coefficient π k , mean vector μ k and the covariance matrix Σ k , Where α is the parameter of the Dirichlet distribution, μ0 and Σ0 are the initial values ​​of the mean and covariance matrix respectively; the GMM parameters are iteratively updated using the EM algorithm until the parameters converge to the optimal solution; The KL divergence calculation module is used to calculate the KL divergence between the probability distribution of each cluster center and the probability distribution of the original voiceprint data; The voiceprint feature vector synthesis module is used to calculate the weight of each cluster center based on its KL divergence with the original voiceprint data, and synthesize a new voiceprint feature vector through the weighted average method.

6. The anonymized voiceprint identity multi-defense system according to claim 5, characterized in that: The feature extraction and probability distribution conversion module uses the Softmax function to convert the feature vectors V of N speakers into k Convert to probability distribution P k (x ki ); Among them, x ki is the feature vector V of the kth speaker k The i-th element in n k is the eigenvector V k The dimension of , T is the temperature parameter.

7. The anonymized voiceprint identity multi-defense system according to claim 5, characterized in that: The KL divergence calculation module calculates the probability distribution P(x0) of the original voiceprint data x0 and the probability distribution Q(μ k ) between the KL divergence D KL (P(x0)∥Q(μ k )) to measure the difference between the original voiceprint and each cluster center; Among them, μ k is the mean vector of each cluster center; x 0i is the feature vector V of the protected speaker k The i-th element in μ ki is the feature vector V of the selected speaker k The i-th element in n k is the eigenvector V k The dimension of .

8. The anonymized voiceprint identity multi-defense system according to any one of claims 5 to 7, characterized in that: The voiceprint feature vector synthesis module synthesizes a new voiceprint feature vector through a weighted average method, and continuously adjusts the weight through an iterative update strategy to find the optimal weight configuration with the largest difference from the original voiceprint feature; specific Includes the following submodules: Submodule 1 is used to assign an initialization weight to each cluster center using random initialization And regularize it so that its sum is 1; Among them, v k is a randomly generated positive value from a uniform or normal distribution, and K is the number of cluster centers; Submodule 2, used to calculate the weighted average voiceprint feature vector in, is the new voiceprint feature vector in the t-th iteration, is the weight in the tth iteration; Submodule 3, used to calculate the new voiceprint feature vector KL divergence D with the original voiceprint data x0 KL , and accordingly update the weights of each cluster center to maximize the difference between the synthesized features and the original voiceprint until the preset convergence condition is reached; Among them, x 0i is the feature vector V of the protected speaker k The i-th element in is the new voiceprint feature vector in the tth iteration, n k is the eigenvector V k , P(x0) is the probability distribution of the original voiceprint data x0, is the new voiceprint feature vector The probability distribution of According to the calculation results of KL divergence, adjust the weight w k , to maximize the KL divergence between the original voiceprint data and the cluster center; in: is the weight of the kth cluster center in the tth iteration; η is the learning rate, represents the KL divergence with respect to weight w k The partial derivative of Q(μ ki ) is the probability distribution of the k-th cluster center; Submodule 4, used to check the change of KL divergence Whether it is less than the set threshold ∈ is used to determine whether the algorithm has converged: if it is not less than the set threshold, it returns to the execution submodule 2; if it is less than the set threshold, it returns the new voiceprint feature vector calculated at this time.

Images