A speaker anonymization system and method based on differential privacy

Abstract

A speaker anonymization system and method based on differential privacy, relating to the field of speech processing, is disclosed. The method includes: extracting speaker vectors, fundamental frequency F0, and content features for each original speech; performing batch anonymization processing on the extracted speaker vectors using a generalized differential privacy algorithm to obtain anonymous speaker vectors; and synthesizing the anonymous speaker vectors, fundamental frequency F0, and content features to generate desensitized anonymized speech waveform data. This invention combines generalized differential privacy with a language-transferable speaker anonymization framework based on a self-supervised model. This allows for the protection of speaker identity privacy data even when attackers possess some prior knowledge. Furthermore, it ensures both the privacy and usability of the anonymized speech, better addressing the balance between WER (Written Emissions) and EER (External Emissions).

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and specifically to a speaker anonymization system and method based on differential privacy. Background Technology

[0002] With the advancement of voice-based human-computer interaction technology and the development of smart devices, voice data has become a new dimension of big data. Collecting and sharing real-world voice data can not only improve innovative services and products, such as Apple's Siri and Google Assistant, but also promote research into intelligent algorithms. However, privacy and security concerns may hinder the collection and sharing of real-world voice data. First, voice data contains speaker identification information, represented as a voiceprint (similar to a fingerprint), such as the speaker's age, gender, geographical background, health, or emotional state, which is considered a biometric identifier. Second, exposing personal voiceprints may pose security risks. In practical applications of voice, there are many uncertainties regarding voice protection. Because voiceprints are used in many authentication systems, attackers may launch spoong attacks against voice authentication systems. Simultaneously, the speaker's identity from unprotected voice recordings can be resynthesized, cloned, and converted using advanced speech synthesis (SS) technology, potentially leading to privacy risks in voiceprint recognition systems.

[0003] Therefore, to protect speaker privacy, speaker anonymization methods, also known as speaker de-identification methods, have been proposed. Speaker anonymization refers to a method that suppresses personal identity information in speech signals while retaining other attributes unchanged. Its aim is to suppress the speaker's identity, i.e., hide the speaker's information, while maintaining readability and naturalness as much as possible. Anonymization requires not only changing the speaker's voice but also altering other features and states, words in the spoken content, and background sounds. This is because combining these factors with external data could potentially reveal the speaker's identity.

[0004] Currently, speaker anonymization methods are still under exploration. Researchers have proposed several approaches, including obfuscation, encryption, distributed learning, and anonymization. However, these methods all have some technical problems. For example, obfuscation methods make the anonymized speech unusable for downstream tasks, encryption methods have excessively high computational complexity, and decentralized or federated learning methods still have the potential to leak the original data. Existing language transferable speaker anonymization methods based on self-supervised models achieve the highest security when the attacker does not have prior knowledge by selecting candidate vectors far from the original speaker. However, once the attacker gains some prior knowledge, the security of this method fluctuates significantly, and its anonymization process has some defects. This shows that simple distance-based mapping cannot achieve more robust anonymization. Although existing speaker anonymization methods have made great progress thanks to the two Voice Privacy competitions, there are still significant problems regarding the balance between the system's word error rate (WER) and equal error rate (EER) and how to achieve more robust anonymization. Summary of the Invention

[0005] To address the problems of existing speaker anonymization methods, this invention aims to enhance the reliability and security of the published speech database. It proposes a speaker anonymization system and method based on differential privacy. This invention utilizes a generalized differential privacy method based on privacy metrics to ensure higher security for the speech database. Simultaneously, it guarantees that the anonymized speech retains high usability and can be used for various downstream speech tasks.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] This invention discloses a speaker anonymization system based on differential privacy, comprising: an F0 extractor, a HuBERT-based soft content encoder, an ECAPA-TDNN speaker encoder, a HiFiGAN neural vocoder, and a privacy measurement module; the F0 extractor is used to extract the fundamental frequency F0 of the input raw speech; the HuBERT-based soft content encoder is used to extract the content features of the input raw speech; the ECAPA-TDNN speaker encoder is used to extract the speaker vector of the input raw speech; the privacy measurement module is used to perform batch anonymization processing on the extracted speaker vectors using a generalized differential privacy algorithm to obtain anonymous speaker vectors; the HiFiGAN neural vocoder is used to synthesize the fundamental frequency F0, content features, and anonymous speaker vectors into the final anonymized speech waveform data.

[0008] Furthermore, the HuberT-based soft content encoder and the ECAPA-TDNN speaker encoder are used after pre-training.

[0009] This invention provides a speaker anonymization method based on differential privacy, implemented using the aforementioned speaker anonymization system based on differential privacy. The method includes the following steps:

[0010] Step 1: For each original speech, extract the speaker vector, fundamental frequency F0, and content features;

[0011] Step 2: Use the generalized differential privacy algorithm to perform batch anonymization on the extracted speaker vectors to obtain anonymous speaker vectors;

[0012] Step 3: Combine the anonymous speaker vector, fundamental frequency F0, and content features to generate anonymized speech waveform data.

[0013] Furthermore, the specific operational procedures for step one are as follows:

[0014] The input is raw speech. For the input raw audio waveform, the speaker vector is extracted using the ECAPA-TDNN speaker encoder, the fundamental frequency F0 is extracted using the F0 extractor, and the content features are extracted using the HuBERT-based soft content encoder.

[0015] Furthermore, the input raw speech is derived from a speech database, using D1 = u1, u2, ..., u n Represents a speech database, where u i This represents the i-th statement in the speech database; using D2 = x1, x2, ..., x n Let x represent the voiceprint database extracted from the speech database D1, where x ∈ X, and x represents the voiceprint and X represents the voiceprint domain.

[0016] Furthermore, the specific operational procedures for step two are as follows:

[0017] The extracted speaker vectors are batch anonymized using a privacy metric method from the generalized differential privacy algorithm to obtain anonymous speaker vectors; the privacy metric is defined as follows:

[0018] For satisfying d X The mechanism of privacy, K: X→P(Z), holds only if for any x, x'∈X:

[0019]

[0020] Where X represents the voiceprint domain, x and x′ both represent the voiceprint in the voiceprint domain X, and d XLet X represent the distance metric of the voiceprint domain, Z represent a set of query results, K(x) and K(x′) represent the probability distributions of voiceprint x and voiceprint x′ on a set of query results Z, respectively, and F Z Let Z represent the σ-algebra over a set of query results Z, and let P(Z) represent the set of probability measures over a set of query results Z.

[0021] According to formula (1), the privacy metric includes: a representation of the secret and the distance between two secrets; the secret refers to the speaker’s voiceprint in the voiceprint database; according to the requirements of the privacy metric, the distance between speaker vectors needs to be metric; angular distance is used for distance metric to maintain the similarity between voiceprints.

[0022] The beneficial effects of this invention are:

[0023] This invention addresses privacy protection by protecting the privacy of published speaker voice identity data. In other words, publicly available voice databases should ensure that the embedded speaker voice identity attributes are indistinguishable. Therefore, this invention combines generalized differential privacy with a language-transferable speaker anonymization framework based on a self-supervised model to achieve a differential privacy-based speaker anonymization method. This invention can guarantee the security of speaker identity privacy data even when attackers possess some prior knowledge. Furthermore, this invention departs from the traditional, simple linear distance correlation method for candidate vector selection, ensuring both the privacy and usability of the anonymized speech, thus better addressing the balance between the system's word error rate (WER) and equal error rate (EER).

[0024] Specifically, this invention uses a generalized differential privacy scheme based on metric privacy to anonymize a batch of speech, and uses angular distance as the metric, which satisfies the inequality property and can maintain the similarity between voiceprints. Attached Figure Description

[0025] Figure 1 This is a structural framework diagram of a speaker anonymization system based on differential privacy according to the present invention.

[0026] Figure 2 This is a flowchart of a speaker anonymization method based on differential privacy according to the present invention.

[0027] Figure 3 This is an application example of a speaker anonymization method based on differential privacy according to the present invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings.

[0029] See Figure 1 The present invention provides a speaker anonymization system based on differential privacy, which mainly comprises five parts: an F0 extractor, a HuberT-based soft content encoder, an ECAPA-TDNN speaker encoder, a HiFiGAN neural vocoder, and a privacy measurement module. Both the HuberT-based soft content encoder and the ECAPA-TDNN speaker encoder require prior pre-training before use.

[0030] The F0 extractor is used to extract the fundamental frequency F0 of the input raw speech.

[0031] A HuberT-based soft content encoder is used to extract content features from the raw input speech.

[0032] ECAPA-TDNN speaker encoder is used to extract speaker vectors from the raw input speech;

[0033] The privacy measurement module is used to perform batch anonymization processing on the extracted speaker vectors using a generalized differential privacy algorithm to obtain anonymous speaker vectors.

[0034] The HiFiGAN neural vocoder is used to synthesize the fundamental frequency F0, content features, and anonymous speaker vectors into the final anonymized speech waveform data.

[0035] See Figure 2 To illustrate, the present invention provides a speaker anonymization method based on differential privacy, which is implemented using the aforementioned speaker anonymization system based on differential privacy. For each original speech, its speaker vector, fundamental frequency F0, and content features are first extracted. Then, the speaker vector is anonymized using a generalized differential privacy algorithm. Finally, the anonymized speaker vector, fundamental frequency F0, and content features are synthesized to generate anonymized speech waveform data.

[0036] The present invention provides a speaker anonymization method based on differential privacy, the specific operation steps of which are as follows:

[0037] Step 1: Feature extraction;

[0038] The input is raw speech. For the input raw audio waveform, the speaker vector is extracted using the ECAPA-TDNN speaker encoder, the fundamental frequency F0 is extracted using the F0 extractor, and the content features are extracted using the HuBERT-based soft content encoder.

[0039] The input raw speech comes from a speech database. In this invention, D1 = u1, u2, ..., u n Represents a speech database, where u i This represents the i-th statement in the speech database; using D2 = x1, x2, ..., x n This represents the voiceprint database extracted from the speech database D1, where x∈X, and x represents the voiceprint and X represents the voiceprint domain. Note that the speech database D1 and the voiceprint database D2 contain the same amount of data n.

[0040] Step 2: Speaker vector anonymization;

[0041] The extracted speaker vectors are batch anonymized using a generalized differential privacy algorithm to obtain anonymous speaker vectors.

[0042] Specifically, the extracted speaker vectors are batch anonymized using the privacy metric method within the generalized differential privacy algorithm to obtain anonymous speaker vectors. Differential privacy, first proposed by Dwork in 2008, rigorously proves through mathematical proof that the Randomized Response method ensures that the impact of a single record on the output information of the dataset remains below a certain threshold. This prevents third parties from judging whether a single record has been modified or added / deleted based on changes in the output, and is considered the most secure method among perturbation-based privacy protection methods currently available. Furthermore, the privacy metric (MetricPrivacy) is a type of generalized differential privacy that defines a distance metric between secrets and guarantees that the indistinguishability of any two secrets is proportional to the distance between each secret.

[0043] The definition of privacy metrics is as follows:

[0044] For satisfying d X The mechanism of privacy, K: X→P(Z), holds only if for any x, x'∈X:

[0045]

[0046] Where X represents the voiceprint domain, x and x′ both represent the voiceprint in the voiceprint domain X, and d X Let X represent the distance metric of the voiceprint domain, Z represent a set of query results, K(x) and K(x′) represent the probability distributions of voiceprint x and voiceprint x′ on a set of query results Z, respectively, and F Z Let P(Z) represent the σ-algebra over a set of query results Z, and let P(Z) represent the set of probability measures over a set of query results Z; it can be applied to voiceprints.

[0047] The definition of σ-algebra is as follows: Let X be a non-empty set and A be a set class on X. If A is closed under the union and complement operations of finite sets, it is called an algebra on X. An algebra that is closed under the union of countable sets is called a σ-algebra.

[0048] According to formula (1), the standard (i.e., the privacy metric) includes: the representation of the secret and the distance between the two secrets. Here, the secret refers to the speaker's voiceprint in the voiceprint database. According to the privacy metric requirements, the distance between speaker vectors needs to be quantified. Therefore, this invention uses angular distance as the distance metric because it satisfies inequality properties and can preserve the similarity between voiceprints.

[0049] Step 3: Synthesis of anonymized speech waveform data;

[0050] The HiFiGAN neural vocoder is used to synthesize the fundamental frequency F0, content features, and anonymous speaker vectors into the final anonymized speech waveform data.

[0051] This invention employs a privacy metric method from the generalized differential privacy algorithm to anonymize the speaker vector of each input raw speech. It uses an exponential mechanism to add randomized noise to the speaker vector, as this mechanism is well-suited for non-numerical outputs such as images and speech. Furthermore, it uses a HiFiGAN neural vocoder to generate anonymized speech waveform data. This invention enhances the security of publicly available speech databases while achieving speech transferability.

[0052] like Figure 3 As shown, the original speech database is transformed using a speaker anonymization method based on differential privacy according to the present invention to obtain a corresponding protected speech database. Specifically, the speech data corresponding to speaker A in the original speech database is record 1, the speech data corresponding to speaker B is record 2, the speech data corresponding to speaker C is record 3, and so on. After transformation using the speaker anonymization method based on differential privacy according to the present invention, the speech data of speaker A, i.e., record 1, becomes the voiceprint of speaker C; the speech data of speaker B, i.e., record 2, becomes the voiceprint of speaker A; and the speech data of speaker C, i.e., record 3, becomes the voiceprint of speaker B. Through the speaker anonymization method based on differential privacy according to the present invention, it is difficult for attackers to distinguish whether the original voiceprint comes from A, B, or C, thus better protecting speaker privacy.

[0053] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A speaker anonymization method based on differential privacy, characterized in that, A speaker anonymization system based on differential privacy is employed. This system includes: an F0 extractor, a HuberT-based soft content encoder, an ECAPA-TDNN speaker encoder, a HiFi GAN neural vocoder, and a privacy measurement module. The F0 extractor extracts the fundamental frequency F0 of the input raw speech. The HuberT-based soft content encoder extracts the content features of the input raw speech. The ECAPA-TDNN speaker encoder extracts the speaker vector of the input raw speech. The privacy measurement module performs batch anonymization processing on the extracted speaker vectors using a generalized differential privacy algorithm to obtain anonymous speaker vectors. The HiFi GAN neural vocoder synthesizes the fundamental frequency F0, content features, and anonymous speaker vectors into the final anonymized speech waveform data. The method includes the following steps: Step 1: For each original speech, extract the speaker vector, fundamental frequency F0, and content features; Step 2: Use the generalized differential privacy algorithm to perform batch anonymization on the extracted speaker vectors to obtain anonymous speaker vectors; The extracted speaker vectors are batch anonymized using a privacy metric method from the generalized differential privacy algorithm to obtain anonymous speaker vectors; the privacy metric is defined as follows: Targeting the satisfaction -privacy mechanism K: Only when any At that time, there were: Where X represents the voiceprint domain, and x and x′ both represent the voiceprints in the voiceprint domain X. Let X represent the distance metric of the voiceprint domain X, Z represent a set of query results, and K(x) and K(x′) represent the probability distributions of voiceprint x and voiceprint x′ on a set of query results Z, respectively. Denotes the σ-algebra over a set of query results Z. Represents a set of query results A set of probabilities; According to formula (1), the privacy metric includes: a representation of the secret and the distance between two secrets; the secret refers to the speaker's voiceprint in the voiceprint database; according to the requirements of the privacy metric, the distance between speaker vectors needs to be metric; angular distance is used for distance metric to maintain the similarity between voiceprints; Step 3: Combine the anonymous speaker vector, fundamental frequency F0, and content features to generate anonymized speech waveform data.

2. The speaker anonymization method based on differential privacy according to claim 1, characterized in that, The HuberT-based soft content encoder and ECAPA-TDNN speaker encoder are used after pre-training.

3. The speaker anonymization method based on differential privacy according to claim 1, characterized in that, The specific operation process for step one is as follows: The input is raw speech. For the input raw audio waveform, the speaker vector is extracted using the ECAPA-TDNN speaker encoder, the fundamental frequency F0 is extracted using the F0 extractor, and the content features are extracted using the HuBERT-based soft content encoder.

4. The speaker anonymization method based on differential privacy according to claim 3, characterized in that, The input raw speech comes from a speech database, using D1 = {u1, u2, ..., u...} n } represents a speech database, where u i This represents the i-th statement in the speech database; using D2 = {x1, x2, ..., x...} n } represents the voiceprint database extracted from the speech database D1. Where x represents the voiceprint and X represents the voiceprint domain.

Images