Anonymous speaker attack method based on graph convolutional network
The high-order structural information of the F0 feature is extracted through a graph convolutional network and combined with the acoustic model to generate anonymized speech, which solves the problem of ignoring the time frame correlation in existing technologies and improves the attacker's system attack capability.
Patent Information
- Application Number
- CN202510192231.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-21
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2045-02-21
AI Technical Summary
In existing speech anonymization technologies, attackers use a variety of technical means to try to restore the speaker's identity or extract personal information, but ignore the feature correlation between different time frames in the speech signal, resulting in the attack model being unable to effectively capture the speaker's identity information.
A graph convolutional network is used to extract the high-order structural information of the F0 feature and concatenate it with the original F0 feature. Combined with the ASR-BN feature of the acoustic model, anonymized speech is generated. The attacker's attack capability is evaluated by training the attacker system.
By considering the temporal correlation between different frames of the F0 feature and utilizing the graph convolutional network and F0 feature to collaborate with anonymous speaker identity information, the performance of the attacker's system is improved, and the accuracy and effectiveness of the attack capability are enhanced.
Smart Images
Figure CN120048241B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of language conversion technology, and in particular to an anonymous speaker attack method based on a graph convolutional network. Background Art
[0002] Speech anonymization technology processes speech signals to render the speaker's identity unrecognizable. In speech anonymization, the audio content is typically modified or perturbed, while preserving the semantics of the speech as much as possible. This ensures that the speech can still be understood or used for specific tasks, but can no longer be directly linked to the original speaker. However, attackers use various techniques to attempt to defeat these anonymization measures, recover the speaker's identity, or extract valuable personal information from the anonymized speech. Anonymized speaker attack systems aim to recover or identify the original speaker's identity by analyzing anonymized speech. Even after the speech has been anonymized, attack systems attempt to defeat the anonymization algorithm and recover individual speaker characteristics, such as voiceprint information, gender, and age, or associate the anonymized speech with a specific speaker. These attacks are typically implemented using a variety of techniques (such as deep learning, feature extraction, and machine learning) to bypass anonymization measures and recover the protected original speech features.
[0003] While these techniques have achieved some success in experiments, some challenges remain. Previous attack methods rarely considered the correlation between speaker features across different time frames, and attack models often overlooked the complexity of temporal dynamics in speech signals. In reality, speech signals are highly temporally sequential, and features across different time frames are not only correlated with the current speech content but may also carry speaker identity information. For example, pitch, speaking rate, and intonation in speech have strong temporal correlations. Ignoring the temporal nature of these features may prevent attack models from effectively capturing speaker identity information.
[0004] In speech signals, F0 (fundamental frequency) is a key feature that describes the periodicity of sound. F0 reflects the pitch variations in speech signals and is crucial for tasks such as speaker recognition, emotion recognition, and speech synthesis. Extracting F0 typically relies on signal periodicity analysis methods, such as autocorrelation and time-frequency analysis.
[0005] In practical applications, F0 feature extraction often relies on classic audio analysis tools, such as Yaapt (Yet Another Algorithm for Pitch Tracking), a pitch tracking method based on autocorrelation analysis that effectively extracts accurate F0 feature trajectories from speech signals. F0 not only reflects the speaker's pitch characteristics but is also closely related to the speaker's emotion, intonation, and other information. Therefore, F0 is often used as a key feature for modeling in sentiment analysis and speech synthesis tasks.
[0006] Graph Convolutional Network (GCN) is a deep learning method specifically designed to process graph-structured data. Unlike traditional convolutional neural networks (CNNs) that process regular grid data (such as images), GCN can effectively process data with irregular structures, such as social networks, molecular structures, knowledge graphs, etc. The core idea of GCN is to transfer information in the adjacency relationship of the graph through convolution operations, so that each node can combine information from adjacent nodes in the multi-layer convolution process to learn the deep features of the node. The basic principle of GCN comes from graph signal processing theory. It defines the relationship between nodes in the graph through the adjacency matrix and obtains information from adjacent nodes through convolution operations. By stacking multiple layers of GCN layers, GCN can aggregate the information of adjacent nodes layer by layer, thereby obtaining high-order structural information of the data points.
[0007] Therefore, it is necessary to propose an anonymous speaker attack method based on graph convolutional networks, which considers the temporal correlation between different frames of F0 features and utilizes graph convolutional networks and F0 features to coordinate anonymous speaker identity information to improve the attacker's system performance. Summary of the Invention
[0008] The purpose of this invention is to provide an anonymous speaker attack method based on graph convolutional networks, aiming to consider the temporal correlation between different frames of F0 features, and utilize graph convolutional networks and F0 features to coordinate anonymous speaker identity information to improve the attacker's system performance.
[0009] To achieve the above objectives, the present invention adopts an anonymous speaker attack method based on a graph convolutional network, comprising the following steps:
[0010] The graph convolutional network is used to extract the high-order structural information of the F0 feature obtained by the feature extractor, and then it is combined with the original F0 feature to form a new F0 feature.
[0011] Use the acoustic model to extract the ASR-BN features of the audio and perform vector quantization on them;
[0012] The processed F0 features and ASR-BN features are concatenated and anonymized speech is generated through a synthesizer;
[0013] Use the generated anonymized data to train the attacker's system;
[0014] For a given trial utterance and enrollment utterance, compute speaker embeddings from both the trial utterance and the enrollment utterance and compare the two;
[0015] Output the similarity score between the anonymized test utterance embedding and the anonymized enrollment utterance embedding, and determine whether they belong to the same speaker based on the score;
[0016] Through multiple trials and registered utterance pairs, the error rate is calculated as a performance indicator to evaluate the attacker's system's ability to attack the anonymization system.
[0017] Among them, in the step of using the graph convolutional network to extract the high-order structural information of the F0 feature obtained by the feature extractor, and splicing and fusing it with the original F0 feature as a new F0 feature:
[0018] The feature extractor extracts features from the input speech waveform through a pitch tracking algorithm and normalizes the features after extraction. The frame length during extraction is 35.0ms, the frame shift is 20ms, and the threshold is 0.25.
[0019] Among them, in the step of using the graph convolutional network to extract the high-order structural information of the F0 feature obtained by the feature extractor, and splicing and fusing it with the original F0 feature as a new F0 feature:
[0020] The embedded features output by the extractor are input into the graph convolutional network with the dimension [B, T, D];
[0021] Where B is the batch size, T is the number of time frames, and D is the sum of feature dimensions.
[0022] Among them, in the step of extracting the ASR-BN features of the audio using the acoustic model and performing vector quantization on it:
[0023] The acoustic model consists of a pre-trained model and three TDNN-F layers; the output dimension of the pre-trained model is 1024, and the dimension of the bottleneck layer is 256.
[0024] Among them, in the step of splicing the processed F0 features and ASR-BN features and generating anonymized speech through the synthesizer:
[0025] The new F0 feature containing high-order structural information after the F0 feature is processed by the graph convolutional network is concatenated with the original F0 feature, and its dimension is adjusted to be consistent with the original output dimension through a linear layer, and then concatenated with the ASR-BN feature.
[0026] Among them, in the step of using the graph convolutional network to extract the high-order structural information of the F0 feature obtained by the feature extractor and concatenating it with the original F0 feature as the new F0 feature, the graph convolutional network processes the F0 feature as follows:
[0027] Get the original F0 feature; the dimension of the original F0 feature is (B, C, T), where B is the batch size, C is the feature dimension, and T is the number of time steps;
[0028] In order to align with the time step of ASR-BN feature, the F0 feature is interpolated; let the time step of BN be T bn , F0 feature after interpolation The number of time steps will be adjusted to T bn :
[0029]
[0030] The feature dimension after interpolation is (B, C, T bn );
[0031] Input the F0 feature into the graph convolutional network to obtain the graph feature F', that is:
[0032]
[0033] Among them, W (0) and W (1) is the weight matrix in the graph convolutional network, A k is the adjacency matrix:
[0034]
[0035] The output feature F' is different from the original feature The new F0 feature is obtained by splicing, namely:
[0036]
[0037] The present invention discloses a graph convolutional network-based anonymous speaker attack method, which employs a process of speech anonymization, attacker system training, and performance evaluation. The speech anonymization includes: extracting high-order structural information of F0 features obtained by a feature extractor using a graph convolutional network, and concatenating and fusing them with the original F0 features to form new F0 features; extracting ASR-BN features from the audio using an acoustic model and performing vector quantization on them; concatenating the processed F0 features and ASR-BN features, and generating anonymized speech through a synthesizer. The attacker system training and performance evaluation includes: training the attacker system using the generated anonymized data; calculating speaker embeddings from the test and enrollment utterances for a given test utterance and enrollment utterance, and comparing the two; outputting a similarity score between the anonymized test utterance embedding and the anonymized enrollment utterance embedding, and determining whether they belong to the same speaker based on the score; and calculating the equal error rate (ERR) as a performance metric across multiple test and enrollment utterance pairs to evaluate the attacker system's attack capability against the anonymized system. By considering the temporal correlation between F0 features across different frames, the graph convolutional network and F0 features are used in conjunction with anonymous speaker identity information to improve the attacker system's performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0039] Figure 1 This is a flowchart of the steps of the anonymous speaker attack method based on graph convolutional network of the present invention.
[0040] Figure 2 It is a flow chart of speech anonymization of the present invention.
[0041] Figure 3 It is a flowchart of the attacker system training and performance evaluation of the present invention. DETAILED DESCRIPTION
[0042] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.
[0043] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
[0044] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0045] See also Figures 1 to 3 The present invention provides an anonymous speaker attack method based on a graph convolutional network, comprising the following steps:
[0046] S100: Use the graph convolutional network to extract the high-order structural information of the F0 feature obtained by the feature extractor, and concatenate it with the original F0 feature as a new F0 feature;
[0047] S200: Extracting ASR-BN features of the audio using the acoustic model and performing vector quantization on them;
[0048] S300: concatenate the processed F0 features and ASR-BN features, and generate anonymized speech through a synthesizer;
[0049] S400: Use the generated anonymized data to train the attacker's system;
[0050] S500: For a given test utterance and enrollment utterance, calculate speaker embeddings from the test utterance and the enrollment utterance, and compare the two;
[0051] S600: Output similarity scores between the anonymized test utterance embedding and the anonymized enrollment utterance embedding, and determine whether they belong to the same speaker based on the scores;
[0052] S700: Through multiple trials and registered utterance pairs, the error rate is calculated as a performance indicator to evaluate the attacker's system's ability to attack the anonymization system.
[0053] In this embodiment, a process of speech anonymization, attacker system training, and performance evaluation is employed. The speech anonymization process includes: extracting high-order structural information from the F0 features obtained by a feature extractor using a graph convolutional network, and concatenating and fusing them with the original F0 features to create new F0 features; extracting ASR-BN features from the audio using an acoustic model and performing vector quantization on them; concatenating the processed F0 features and ASR-BN features, and generating anonymized speech through a synthesizer. The attacker system training and performance evaluation process includes: training the attacker system using the generated anonymized data; calculating speaker embeddings from the test and enrollment utterances for a given test utterance and enrollment utterance, and comparing them; outputting a similarity score between the anonymized test utterance embedding and the anonymized enrollment utterance embedding, and determining whether they belong to the same speaker based on the similarity score; and calculating the equal error rate (ERR) as a performance metric across multiple test and enrollment utterance pairs to evaluate the attacker system's attack capability against the anonymized system. By considering the temporal correlation between F0 features across different frames, the graph convolutional network and F0 features are used in conjunction with anonymous speaker identity information to improve the attacker system's performance.
[0054] In voice anonymization:
[0055] The feature extractor extracts features from the input speech waveform through a pitch tracking algorithm and normalizes the features after extraction. The frame length during extraction is 35.0ms, the frame shift is 20ms, and the threshold is 0.25.
[0056] The embedded features output by the extractor are input into the graph convolutional network with the dimension [B, T, D];
[0057] Where B is the batch size, T is the number of time frames, and D is the sum of feature dimensions.
[0058] The acoustic model consists of a pre-trained model and three TDNN-F layers; the output dimension of the pre-trained model is 1024, and the dimension of the bottleneck layer is 256.
[0059] The new F0 feature containing high-order structural information after the F0 feature is processed by the graph convolutional network is concatenated with the original F0 feature, and its dimension is adjusted to be consistent with the original output dimension through a linear layer, and then concatenated with the ASR-BN feature.
[0060] The graph convolutional network processes the F0 feature as follows:
[0061] Get the original F0 feature; the dimension of the original F0 feature is (B, C, T), where B is the batch size, C is the feature dimension, and T is the number of time steps;
[0062] In order to align with the time step of ASR-BN feature, the F0 feature is interpolated; let the time step of BN be T bn , F0 feature after interpolation The number of time steps will be adjusted to T bn :
[0063]
[0064] The feature dimension after interpolation is (B, C, T bn );
[0065] Input the F0 feature into the graph convolutional network to obtain the graph feature F', that is:
[0066]
[0067] Among them, W (0) and W (1) is the weight matrix in the graph convolutional network, A k is the adjacency matrix:
[0068]
[0069] The output feature F' is different from the original feature The new F0 feature is obtained by splicing, namely:
[0070]
[0071] In this embodiment, speech to be anonymized is prepared, including a training set which is a train-clean-360 subset of the LibriSpeech corpus. In addition to the provided anonymous training data, the original train-clean-360 data is also used. LibriSpeech is an English spoken speech corpus derived from audiobooks, designed for ASR research. LibriSpeech contains 960 hours of speech sampled at 16kHz, which will be used for ASV and ASR evaluation. IEMOCAP is an emotional audio-visual dataset that will be used for SER evaluation. IEMOCAP contains 12 hours of speech sampled at 16kHz, corresponding to improvised and scripted two-person conversations between 5 female and 5 male British actors.
[0072] Use graph convolutional networks to extract the high-order structural information of the F0 feature obtained by the feature extractor, and concatenate it with the original F0 feature as a new F0 feature;
[0073] The processing of F0 features by the graph convolutional network is the key to the entire process. Its construction process includes the following steps:
[0074] The dimension of the original F0 feature is (B, C, T), where B is the batch size, C is the feature dimension, and T is the number of time steps.
[0075] In order to align with the time step of ASR-BN features, F0 features need to be interpolated. Let the time step of BN be T bn , F0 feature after interpolation The number of time steps will be adjusted to T bn :
[0076]
[0077] The feature dimension after interpolation becomes (B, C, T bn ).
[0078] The F0 feature is input into the graph convolutional network, namely:
[0079]
[0080] Among them, W (0) and W (1) is the weight matrix in the graph convolutional network, A k is the adjacency matrix. According to the speech graph shift operator, we consider the potential relationship between each frame and the surrounding k frames, that is:
[0081] Thus, the graph feature F' is obtained.
[0082] The features F' output by the graph convolutional network are then combined with the original features The new F0 feature is obtained by splicing, namely:
[0083]
[0084] Use the wav2vec2.0_tdnnf acoustic model to extract the ASR-BN features of the audio and perform vector quantization on them;
[0085] The processed F0 features and ASR-BN features are concatenated and anonymized speech is generated through the Hifi-GAN synthesizer.
[0086] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.
[0087] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.
Claims
1. A method for attacking anonymous speakers based on graph convolutional networks, characterized in that: The steps include: The graph convolutional network is used to extract the high-order structural information of the F0 feature obtained by the feature extractor, and then it is combined with the original F0 feature to form a new F0 feature. Use the acoustic model to extract the ASR-BN features of the audio and perform vector quantization on them; The processed F0 features and ASR-BN features are concatenated and anonymized speech is generated through a synthesizer; Use the generated anonymized data to train the attacker's system; For a given trial utterance and enrollment utterance, compute speaker embeddings from both the trial utterance and the enrollment utterance and compare the two; Output the similarity score between the anonymized test utterance embedding and the anonymized enrollment utterance embedding, and determine whether they belong to the same speaker based on the score; Through multiple trials and registered utterance pairs, we calculate the error rate as a performance indicator to evaluate the attacker's ability to attack the anonymization system; In the step of using the graph convolutional network to extract the high-order structural information of the F0 feature obtained by the feature extractor and concatenating it with the original F0 feature as the new F0 feature, the graph convolutional network processes the F0 feature as follows: Get the original F0 feature; the dimension of the original F0 feature is (B, C, T), where B is the batch size, C is the feature dimension, and T is the number of time steps; In order to align with the time step of ASR-BN feature, the F0 feature is interpolated; let the time step of BN be T bn , F0 feature after interpolation The number of time steps will be adjusted to T bn : The feature dimension after interpolation is (B, C, T bn ); Input the F0 feature into the graph convolutional network to obtain the graph feature F', that is: Among them, W (0) and W (1) is the weight matrix in the graph convolutional network, A k is the adjacency matrix: The output feature F' is different from the original feature The new F0 feature is obtained by splicing, namely:
2. The anonymous speaker attack method based on graph convolutional network according to claim 1, characterized in that: In the step of using the graph convolutional network to extract the high-order structural information of the F0 feature obtained by the feature extractor and concatenating it with the original F0 feature as a new F0 feature: The feature extractor extracts features from the input speech waveform through a pitch tracking algorithm and normalizes the features after extraction. The frame length during extraction is 35.0ms, the frame shift is 20ms, and the threshold is 0.
25.
3. The anonymous speaker attack method based on graph convolutional network according to claim 1, characterized in that: In the step of using the graph convolutional network to extract the high-order structural information of the F0 feature obtained by the feature extractor and concatenating it with the original F0 feature as a new F0 feature: The embedded features output by the extractor are input into the graph convolutional network with the dimension [B, T, D]; Where B is the batch size, T is the number of time frames, and D is the sum of feature dimensions.
4. The anonymous speaker attack method based on graph convolutional network according to claim 1, characterized in that: In the steps of extracting ASR-BN features of audio using the acoustic model and performing vector quantization on them: The acoustic model consists of a pre-trained model and three TDNN-F layers; the output dimension of the pre-trained model is 1024, and the dimension of the bottleneck layer is 256.
5. The anonymous speaker attack method based on graph convolutional network according to claim 1, characterized in that: In the step of concatenating the processed F0 features and ASR-BN features and generating anonymized speech through the synthesizer: The new F0 feature containing high-order structural information after the F0 feature is processed by the graph convolutional network is concatenated with the original F0 feature, and its dimension is adjusted to be consistent with the original output dimension through a linear layer, and then concatenated with the ASR-BN feature.