A privacy-enhanced voice spoofing detection method based on WeSpeaker architecture
By proposing a privacy-enhanced speech spoofing detection method based on the WeSpeaker architecture, and through acoustic-semantic decoupling and lightweight improved feature extraction, this method solves the problems of privacy leakage, high complexity and limited generalization ability in existing technologies, and achieves efficient and privacy-preserving spoofing detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING LONGYUAN INFORMATION TECH CO LTD
- Filing Date
- 2025-07-23
- Publication Date
- 2026-05-19
AI Technical Summary
Existing voice spoofing detection technologies suffer from privacy risks, high model complexity, mismatch with target tasks, and limited generalization ability.
A privacy-enhanced speech spoofing detection method based on the WeSpeaker architecture is adopted. Privacy-preserving preprocessing is performed through acoustic-semantic decoupling technology, audio feature extraction is performed using a lightweight improved WeSpeaker architecture, and spoofing discrimination is performed through a lightweight fully connected binary classification layer, which weakens semantic dependence and enhances generalization ability.
It improves detection accuracy and generalization ability without revealing the semantic content of speech, is suitable for resource-constrained devices, and meets high-level privacy and security requirements.
Smart Images
Figure CN120748412B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio processing technology, and in particular to a privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture. Background Technology
[0002] With the rapid development of AI-generated content (AIGC) technology, text-to-speech (TTS) systems can now generate highly natural and almost indistinguishable human speech. In particular, the continuous advancements in deep neural networks (DNNs), diffusion models, multi-speaker modeling, and timbre transfer techniques in recent years have enabled attackers to use publicly available TTS tools to synthesize fake voices of target speakers in a short time, thereby carrying out security attacks such as identity fraud, voice manipulation, and social media scams. Therefore, anti-spoofing, as an important research direction in the field of voice security, is dedicated to automatically identifying whether speech was generated by a synthetic system or a replay attack, and is a core security defense in the deployment of modern speech recognition, voiceprint recognition, and voice assistant systems. At the same time, with increasing emphasis on data privacy, more and more research is focusing on how to perform speech recognition or voice security detection without exposing semantic information.
[0003] Existing technologies include voiceprint-based detection, deep learning-based end-to-end detection methods, and forgery detection mechanisms based on content stripping and privacy decoupling. However, existing methods have the following problems:
[0004] 1. Strong reliance on semantic content makes privacy leakage risks unavoidable: Detection methods based on voiceprint features or deep learning typically process raw speech or its spectral representation directly. These representations contain a large amount of information related to semantic content and speaker identity. In practical deployments, this approach often inevitably collects, processes, and even stores complete user voice data, posing a serious risk of privacy leakage, especially in regulatory or privacy-critical scenarios (such as intelligent assistants and financial voice interaction systems), making it difficult to promote.
[0005] 2. High model complexity and mismatch with target tasks, making effective transfer and utilization difficult: For example, while some existing high-performance voiceprint recognition architectures (such as WeSpeaker) possess powerful speaker modeling capabilities, their final classification layer is designed for speaker identity discrimination tasks and cannot be directly applied to "real / forged" binary classification scenarios. Using the original architecture directly results in redundant speaker feature learning and hinders adaptation to forgery detection tasks. Furthermore, the complex structure of such models presents challenges for deployment on resource-constrained devices.
[0006] 3. Overfitting to specific speech features limits generalization ability: Current deep learning-based forgery detection models tend to become dependent on the synthesis methods, semantic content, or speaker styles in the training data during training, leading to a significant drop in performance when faced with unfamiliar forgery methods. Especially in recent years, with the continuous updates and iterations of TTS synthesis models, the adaptability of detection models to unknown attacks has become a key bottleneck to their practicality.
[0007] In summary, existing voice spoofing detection technologies have several drawbacks in practical applications, including privacy risks, high model complexity, mismatch with target tasks, and limited generalization ability. Summary of the Invention
[0008] The purpose of this invention is to provide a privacy-enhanced speech spoofing detection method based on the WeSpeaker architecture, which aims to solve the technical problems of existing speech spoofing detection technologies in practical use, such as privacy leakage risks, high model complexity, mismatch with target tasks, and limited generalization ability.
[0009] To achieve the above objectives, this invention employs a privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture, comprising the following stages:
[0010] Phase 1: Receive the audio input to be detected and perform privacy-preserving preprocessing based on acoustic-semantic decoupling technology;
[0011] Phase 2: Deep extraction of audio features using a lightweight, improved WeSpeaker architecture;
[0012] The third stage involves using a lightweight fully connected binary classification layer to determine whether the extracted features are "real" or "fake," and then providing the detection result of whether the audio is fake.
[0013] In the first phase, the specific methods are as follows:
[0014] First, the audio input to be detected is received, and basic features are extracted.
[0015] Then, an improved neural audio codec architecture is adopted to achieve acoustic-semantic decoupling processing, decomposing the speech signal into acoustic representation and semantic representation.
[0016] Through a designed filter network, the system extracts semantically relevant information from the features, retaining only the acoustic portion devoid of semantic information for subsequent processing. This process effectively prevents the acquisition, reconstruction, or leakage of speech semantic content, making it unsuitable for speech-to-text tasks. The semantic recovery success rate is less than 5%, and even against specially designed semantic reconstruction attacks, the system maintains a high level of privacy protection, meeting high-level privacy and security requirements.
[0017] In the second stage, the specific approach is as follows: WeSpeaker, which has been structurally simplified and optimized, is used as the backbone network for feature extraction.
[0018] The simplification and optimization of the WeSpeaker structure includes:
[0019] Remove the speaker mapping layer to weaken speaker dependency;
[0020] Preserve and optimize the ECAPA-TDNN residual structure;
[0021] Improve the statistical pooling mechanism to capture time series anomalies;
[0022] Introducing a channel attention mechanism to enhance spectrum perception capabilities.
[0023] In this study, the WeSpeaker, which has been structurally simplified and optimized, is used as the backbone network for feature extraction. The speaker mapping layer is removed, and the speaker-dependent speech is weakened to enhance the system's generalization ability to different fake audio generators and avoid misjudging "speaker changes" as "fake".
[0024] In this study, when using the simplified and optimized WeSpeaker as the backbone network for feature extraction, the residual structure of ECAPA-TDNN is retained and optimized. This inherits the residual network structure of ECAPA-TDNN with multi-scale temporal context modeling and channel attention mechanism, which enhances the temporal-frequency domain coupling expression capability of features without increasing additional parameter overhead. By deeply extracting local and global acoustic structure information from speech signals, the system can more easily identify unnatural distortions in synthesized speech at the microscopic level.
[0025] In the feature extraction backbone network using the simplified and optimized WeSpeaker, the configuration of the statistical pooling layer is optimized to make it more sensitive to capturing nonlinear temporal changes in speech and enhance the ability to detect rhythm / prosody anomalies in synthesized speech.
[0026] In the feature extraction backbone network using the simplified and optimized WeSpeaker, when introducing a channel attention mechanism to enhance spectrum perception, a lightweight SE channel attention module is added to the original TDNN framework. This module adaptively weights the importance of each frequency channel, enabling the model to automatically highlight the abnormal energy distribution of forged speech in specific frequency bands, thereby further improving the accuracy of forgery recognition.
[0027] In the third stage, a newly added lightweight fully connected binary classification layer is used to directly distinguish between "real" and "fake" features.
[0028] The discriminative model is trained with multi-source fake data, including fake data generated by various representative TTS techniques. The diversity of training data is enhanced by reverberation, noise, and speed variation techniques, and a domain adversarial learning strategy is introduced to reduce overfitting of specific features of the synthesis techniques.
[0029] Among them, several representative TTS technologies used to generate fake data include autoregressive synthesis, non-autoregressive streaming synthesis, diffusion model synthesis, vocoder fusion, and neural voice conversion.
[0030] This invention discloses a privacy-enhanced speech spoofing detection method based on the WeSpeaker architecture. In practical application, this method comprises three stages: the first stage is an audio input and privacy-preserving preprocessing stage, which achieves privacy protection of the speech content through acoustic-semantic decoupling technology. The second stage is a feature extraction stage based on an improved WeSpeaker architecture, utilizing a lightweight, modified WeSpeaker architecture for deep extraction of audio features. The third stage is a spoofing discrimination and decision stage, using a lightweight fully connected binary classification layer to determine whether the extracted features are "real" or "spoofed." Finally, it provides a detection result indicating whether the audio is spoofed. This approach addresses the technical problems of existing speech spoofing detection technologies, such as privacy leakage risks, high model complexity, mismatch with the target task, and limited generalization ability, in practical applications. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a flowchart of the privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture of the present invention. Detailed Implementation
[0033] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.
[0034] Please see Figure 1 , Figure 1 This is a flowchart of the privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture of the present invention.
[0035] This invention provides a privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture, comprising the following stages:
[0036] Phase 1: Receive the audio input to be detected and perform privacy-preserving preprocessing based on acoustic-semantic decoupling technology;
[0037] In this specific implementation, the audio input to be detected is first received, and basic features are extracted.
[0038] Then, an improved neural audio codec architecture is adopted to achieve acoustic-semantic decoupling processing, decomposing the speech signal into acoustic representation and semantic representation.
[0039] Through a designed filter network, the system extracts semantically relevant information from the features, retaining only the acoustic portion devoid of semantic information for subsequent processing. This process effectively prevents the acquisition, reconstruction, or leakage of speech semantic content, making it unsuitable for speech-to-text tasks. The semantic recovery success rate is less than 5%, and even against specially designed semantic reconstruction attacks, the system maintains a high level of privacy protection, meeting high-level privacy and security requirements.
[0040] Phase 2: Deep extraction of audio features using a lightweight, improved WeSpeaker architecture;
[0041] In this specific implementation, the simplified and optimized WeSpeaker is used as the feature extraction backbone network.
[0042] The simplification and optimization of the WeSpeaker structure includes:
[0043] By removing the speaker mapping layer, the speaker dependency is weakened. In this invention, the original WeSpeaker design included a mapping layer for speaker classification tasks, which is completely removed. This significantly reduces the risk of overfitting the model to speaker features, thereby enhancing the system's generalization ability to different fake audio generators and avoiding misclassifying "speaker changes" as "fakes."
[0044] The ECAPA-TDNN residual structure is retained and optimized. For this purpose, the residual network structure of multi-scale temporal context modeling and channel attention mechanism in ECAPA-TDNN is inherited, which enhances the temporal-frequency domain coupling expression capability of features without increasing additional parameter overhead. By deeply extracting local and global acoustic structure information in speech signals, the system can more easily identify unnatural distortions in synthesized speech at the micro level.
[0045] The statistical pooling mechanism is improved to capture temporal anomalies. In this regard, in response to the common characteristics of fake audio such as "lack of frequency periodicity" and "overly smooth temporal dynamics", the configuration of the statistical pooling layer is optimized to make it more sensitive to capturing nonlinear temporal changes in speech and enhance the ability to detect rhythm / prosody anomalies in synthesized speech.
[0046] A channel attention mechanism is introduced to enhance spectral perception capabilities. To address common characteristics of forged audio, such as "lack of frequency periodicity" and "overly smooth temporal dynamics," this invention optimizes the configuration of the statistical pooling layer, making it more sensitive to capturing nonlinear temporal changes in speech and strengthening the detection capability of rhythmic / prosodic anomalies in synthesized speech.
[0047] The third stage involves using a lightweight fully connected binary classification layer to determine whether the extracted features are "real" or "fake," and then providing the detection result of whether the audio is fake.
[0048] In this specific implementation, at this stage, the system uses a newly added lightweight fully connected binary classification layer to directly distinguish between "real" and "fake" features. This discriminator is trained with multi-source fake data, including fake data generated by five representative TTS techniques: autoregressive synthesis, non-autoregressive streaming synthesis, diffusion model synthesis, vocoder fusion, and neural voice conversion. The diversity of training data is enhanced through techniques such as reverberation, noise, and speed variation, and a domain adversarial learning strategy is introduced to reduce overfitting of specific features of the synthesis techniques.
[0049] By removing the speaker mapping layer (the fully connected layer used for speaker classification) from the original WeSpeaker, its residual structure and feature extraction capabilities are retained. A lightweight fully connected binary classification layer is added at its end, directly serving the task of distinguishing between "real" and "fake" audio. This improvement inherits WeSpeaker's strong modeling capabilities while reducing the risk of speaker overfitting, making it more suitable for forgery detection tasks.
[0050] By integrating the core mechanism of neural audio encoding and decoding structures, acoustic-semantic decoupling processing is performed on the original speech, and only the acoustic part without semantic information is retained to participate in subsequent feature extraction and discrimination, thereby preventing the collection, reconstruction or leakage of speech semantic content and achieving a higher level of privacy protection.
[0051] This invention employs various representative TTS techniques (five mainstream TTS methods) to construct fake data during the training phase, and uses a novel, unseen TTS technique for evaluation during the validation phase, ensuring that the proposed model possesses good generalization ability across synthetic systems. Through a privacy enhancement mechanism, the model's dependence on semantic content is further reduced, allowing it to focus more on unnatural acoustic structural differences in the fake sounds.
[0052] The beneficial effects of this invention are as follows:
[0053] 1. Acoustic-semantic decoupling ensures voice privacy: This invention introduces a neural audio codec architecture in the audio input stage to deeply decouple the acoustics and semantics of the voice signal, retaining only non-semantic acoustic features for forgery detection, effectively preventing the voice content from being reverse-engineered or misused, significantly improving the system's privacy protection capabilities, and is suitable for practical scenarios with higher data compliance requirements.
[0054] 2. Optimized WeSpeaker Feature Extractor: While retaining the advantages of the ECAPA-TDNN residual network architecture, the system removes the speaker mapping layer, optimizes the statistical pooling strategy, and adds a channel attention mechanism. This enhances the model's ability to perceive features such as spectral anomalies and temporal inconsistencies in speech forgery, and improves the generalization detection effect on unknown forgery types.
[0055] 3. Multi-source forged sample training strategy: During the discrimination stage, the system introduces multiple types of TTS forged speech data (including diffusion models, neural vocoders, etc.) for training. At the same time, it enhances the diversity of samples through reverberation, noise and other methods. Combined with domain adversarial training, it further alleviates the overfitting of the model to specific technologies, so that the system can maintain high accuracy and robustness in complex environments and unknown forged technologies.
[0056] 4. Lightweight design adapts to edge deployment requirements: The system implementing this method adopts a modular and lightweight design, which balances detection performance and privacy protection without significantly increasing the computational burden. It has strong practicality and is suitable for low-power and real-time application scenarios such as smart terminals, voice customer service, and in-vehicle voice systems.
[0057] This invention presents a privacy-enhanced speech spoofing detection method based on the WeSpeaker architecture. In practical application, this method comprises three stages: The first stage is the audio input and privacy-preserving preprocessing stage, which achieves speech content privacy protection through acoustic-semantic decoupling technology. The second stage is the feature extraction stage based on an improved WeSpeaker architecture, utilizing a lightweight, modified WeSpeaker architecture for deep extraction of audio features. The third stage is the spoofing discrimination and decision stage, using a lightweight fully connected binary classification layer to determine whether the extracted features are "real" or "spoofed." Finally, it provides a detection result indicating whether the audio is spoofed. This approach addresses the technical problems of existing speech spoofing detection technologies in practical use, including privacy leakage risks, high model complexity, mismatch with the target task, and limited generalization ability.
[0058] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.
Claims
1. A privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture, characterized in that, Includes the following stages: The first stage involves receiving the audio input to be detected and performing privacy-preserving preprocessing based on acoustic-semantic decoupling. The acoustic-semantic decoupling includes receiving the audio input to be detected and extracting basic features. The speech signal is decomposed into acoustic and semantic representations through decoupling processing, and semantic information is stripped away using a filter network, retaining only the acoustic features. The second stage involves deep extraction of audio features using a lightweight WeSpeaker architecture, which includes removing the speaker mapping layer and introducing a channel attention mechanism to form a lightweight feature extraction network adapted to the forgery detection task. The third stage involves using a lightweight fully connected binary classification layer to determine whether the extracted features are "real" or "fake," and then providing the detection result of whether the audio is fake.
2. The privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture as described in claim 1, characterized in that, The lightweight WeSpeaker architecture also includes an optimized ECAPA-TDNN residual structure, which includes a residual network structure that inherits the multi-scale temporal context modeling and channel attention mechanisms in ECAPA-TDNN.
3. The privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture as described in claim 2, characterized in that, The lightweight WeSpeaker architecture also includes a statistical pooling mechanism, which optimizes the configuration of the statistical pooling layer when capturing timing anomalies.
4. The privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture as described in claim 3, characterized in that, When introducing the channel attention mechanism to enhance spectrum sensing capability, a lightweight SE channel attention module is added to the original TDNN framework to adaptively weight the importance of each frequency channel.
5. The privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture as described in claim 1, characterized in that, In the third stage, a newly added lightweight fully connected binary classification layer is used to directly distinguish between "real" and "fake" features. The discriminative model is trained with multi-source fake data, including fake data generated by various representative TTS techniques. The diversity of training data is enhanced by reverberation, noise, and speed variation techniques, and a domain adversarial learning strategy is introduced.
6. The privacy-enhanced voice spoofing detection method based on the WeSpeaker architecture as described in claim 5, characterized in that, Representative TTS techniques used to generate fake data include autoregressive synthesis, non-autoregressive streaming synthesis, diffusion model synthesis, vocoder fusion, and neurospeech conversion.