Audio deepfake identification method and system based on inconsistency detection characterization

Abstract

This invention provides a method and system for identifying deepfake audio based on representation inconsistency detection, belonging to the field of audio recognition technology. The method processes the audio waveform to be detected using a low-to-high consistency detection model, outputting the forgery probability of the detected audio waveform. The low-to-high consistency detection model includes a staged feature extraction module comprising a 24-layer XLS-R model. The first 21 layers of the XLS-R model output low-level acoustic representations, and the last 3 layers output high-level semantic representations. This invention proposes that the core detectable clue for deepfake audio lies in the mismatch between low-level acoustic representations and high-level semantic representations. This innovative discovery fundamentally reconstructs the focusing logic of the detection task, upgrading the traditional passive identification targeting specific flaws to active verification based on internal consistency. It possesses solid theoretical support and can overcome scenario limitations, providing stronger generalization ability and long-term application value for cross-domain and multi-type deepfake audio detection.

Description

Technical Field

[0001] This invention relates to the field of audio recognition technology, and specifically to an audio deepfake recognition method and system based on representation inconsistency detection. Background Technology

[0002] The field of deep learning audio spoofing detection has been developing for many years, and the existing technology is based on deep learning-based audio deception detection methods.

[0003] Existing deep learning-based audio spoofing detection methods mainly include using basic convolutional neural networks (CNNs) to automatically learn spoofing patterns, introducing deep residual networks into audio spoofing detection, or combining the self-supervised learning model Wav2Vec 2.0 with data augmentation detection methods.

[0004] However, existing deep learning-based audio spoofing detection methods often learn to identify repetitive, superficial acoustic flaws generated by specific spoofing algorithms in the training data during training. This "shortcut" fails when faced with spoofed audio using novel generation algorithms or in different acoustic environments, leading to a sharp decline in model performance. In other words, existing deep learning-based audio spoofing detection models lack generalization ability when facing unseen spoofing attacks. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an audio deepfake detection method and system based on representation inconsistency detection, which solves the technical problem that existing deep learning-based audio spoofing detection models have insufficient generalization ability when facing unseen spoofing attacks.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides an audio deepfake identification method based on representation inconsistency detection. The audio deepfake identification method processes the audio waveform to be detected through a pre-constructed low-high consistency detection model and outputs the forgery probability of the audio waveform to be detected. The low-high consistency detection model includes a phased feature extraction module, a fusion module, a dependency capture module, and a classification module. The phased feature extraction module includes a pre-trained 24-layer XLS-R model. The XLS-R model is used to extract features from the audio waveform. The first 21 layers of the XLS-R model output low-level acoustic representations, and the last 3 layers output high-level semantic representations. The fusion module is used to fuse low-level acoustic representations and high-level semantic representations to obtain fused features; The dependency capture module is used to learn a general forgery pattern from the fused features and output global features; The classification module is used to process global features and output the forgery probability of the audio waveform to be detected.

[0007] Preferably, the fusion module is an asymmetric fusion module, comprising two independent 1×1 convolutional layers and a max-pooling layer. Two Res2Net backbone networks of different depths, a first Res2Net backbone network and a second Res2Net backbone network, and an attention network; The two independent 1×1 convolutional layers and max-pooling layers respectively characterize the lower-level acoustics. and high-level semantic representation Projecting onto a unified intermediate dimension yields... and ; The first Res2Net backbone network pair Processing is performed to obtain features. The second Res2Net backbone network for Processing is performed to obtain features. , The attention network is based on features and The concatenated features are processed to generate two channel weights. and , for features and Perform weighted fusion and output fused features.

[0008] Preferably, the first Res2Net backbone network includes three dual-bottleneck structures; the second Res2Net backbone network includes two dual-bottleneck structures.

[0009] Preferably, the dependency capture module includes a Transformer encoder block; Specifically, a squeeze-excitation submodule is embedded before the Transformer encoder block, which includes three stacked multi-head attention mechanism layers.

[0010] Preferably, the classification module includes global average pooling, a fully connected layer, and an activation function layer; the classification module performs global feature processing. The expression for the processing procedure is as follows: in, The probability of forgery of the output audio waveform to be detected. This represents the activation function. Indicates global average pooling. This indicates fully connected layer processing.

[0011] Preferably, the audio waveform to be detected is an audio waveform after enhancement processing of the original audio waveform.

[0012] Preferably, the loss function during the training process of the pre-built low-high consistency detection model includes: in, , All represent weight parameters. The probability of forgery output by the model. This represents the label value of the sample.

[0013] Secondly, the present invention provides an audio deepfake identification system based on representation inconsistency detection, the audio deepfake identification system being used to perform the audio deepfake identification method based on representation inconsistency detection as described above.

[0014] Thirdly, the present invention provides a computer-readable storage medium storing a computer program for audio deepfake identification based on representation inconsistency detection, wherein the computer program causes a computer to perform the audio deepfake identification method based on representation inconsistency detection as described above.

[0015] Fourthly, the present invention provides an electronic device, comprising: One or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the audio depth forgery recognition method based on representation inconsistency detection as described above.

[0016] (III) Beneficial Effects This invention provides a method and system for identifying deepfake audio based on representation inconsistency detection. Compared with existing technologies, it has the following advantages: This invention proposes that the core detectable clue for deepfake audio lies in the mismatch between low-level acoustic representation and high-level semantic representation. This innovative discovery fundamentally restructures the focusing logic of detection tasks, upgrading the traditional passive identification of specific flaws to active verification based on internal consistency. It has both solid theoretical support and can overcome scenario limitations, providing stronger generalization ability and long-term application value for cross-domain and multi-type deepfake audio detection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural diagram of the low-high consistency detection model. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This application provides an audio deepfake detection method and system based on representation inconsistency detection, which solves the technical problem that existing deep learning-based audio spoofing detection models have insufficient generalization ability when facing unseen spoofing attacks, and fundamentally improves the model's generalization ability.

[0021] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: This invention proposes a method and system for deep audio forgery detection based on representation inconsistency detection, aiming to address the problem of insufficient generalization ability of existing deep learning-based audio spoofing detection models when facing unseen forgery attacks. The core idea of ​​this invention is that the most common forgery traces are not acoustic defects, but representation inconsistencies between the low-level acoustic features and high-level semantic attributes of the signal.

[0022] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0023] This invention provides a method for identifying deepfake audio based on representation inconsistency detection. The audio deep forgery detection method processes the audio waveform to be detected through a pre-constructed low-high consistency detection model and outputs the forgery probability of the audio waveform to be detected. The low-high consistency detection model includes a phased feature extraction module, a fusion module, a dependency capture module, and a classification module. The phased feature extraction module includes a pre-trained 24-layer XLS-R model, which extracts features from the audio waveform. The first 21 layers of the XLS-R model output low-level acoustic representations, and the last 3 layers output high-level semantic representations. The fusion module is used to fuse low-level acoustic representations and high-level semantic representations to obtain fused features; The dependency capture module is used to learn general spoofing patterns from the fused features and output global features; The classification module is used to process global features and output the forgery probability of the audio waveform to be detected.

[0024] This invention proposes that the core detectable clue for deep audio forgery lies in the mismatch between low-level acoustic representation and high-level semantic representation. This innovative discovery fundamentally restructures the focusing logic of detection tasks, upgrading the traditional passive identification of specific flaws to active verification based on internal consistency. It has both solid theoretical support and can overcome scenario limitations, providing stronger generalization ability and long-term application value for cross-domain and multi-type deep audio forgery detection.

[0025] It should be noted that, in this embodiment of the invention, the original audio waveform needs to be data augmented before being input into the low-high consistency detection model. In this embodiment of the invention, the original audio waveform is augmented using data augmentation methods such as RawBoost or Wav2Vec 2.0.

[0026] The following is combined with, for example Figure 1 The structure diagram shown provides a detailed description of each module in the low-high consistency detection model: Phased feature extraction module: In this embodiment of the invention, a 24-layer XLS-R model pre-trained on a large amount of unlabeled audio data is selected as a phased feature extraction module to encode the enhanced audio waveform input into the model.

[0027] In this embodiment of the invention, layer-by-layer center kernel alignment (CKA) similarity analysis of the XLS-R model reveals a clear hierarchical functional partitioning. Simultaneously, analysis of the feature difference norm between the real audio and its forged version reveals significant and systematic representational differences at higher levels (layers 22-24). Based on this finding, the output of the first 21 layers of the XLS-R model is defined as the low-level acoustic representation. Together, they describe the generation process and acoustic details of the audio signal. The outputs of the last three layers (layers 22-24) are defined as high-level semantic representations. They encapsulate conclusive information such as the speaker's identity and the content of their speech.

[0028] in, It is the first Hidden state output of the layer This indicates splicing along the channel dimension.

[0029] In this embodiment of the invention, the process of hierarchically partitioning the XLS-R model is as follows: L2 norm analysis (energy distribution): First, the activation energy of the model on four datasets was analyzed. It was found that the energy of L1-21 layers was low and stable, while the energy of L22-24 layers was significantly higher. This initially proved that there are two different processing stages in the model.

[0030] CKA analysis (structural similarity): Secondly, the functional similarity between layers was analyzed using CKA. The heat map clearly shows that layers L1-21 are a compact "acoustic block", while layers L22-24 are another independent "abstract block", which provides strong structural evidence for the stratification.

[0031] Feature difference norm analysis (functional relevance): Finally, the feature differences between genuine and fake audio were compared, and it was found that the differences were small in layers L1-21, but suddenly increased sharply at the boundary of layer L22. This decisively proved that the high-level blocks are the key to detecting forgery.

[0032] Fusion Module: This fusion module is an asymmetric fusion module responsible for receiving and fusing data. and This is to detect inconsistencies between the two.

[0033] Input uniformity: due to and The channel dimensions are different, so two independent ones are used first. Convolutional layers and max-pooling layers project them onto a uniform intermediate dimension to obtain features. and characteristics .

[0034] Asymmetric processing: Two feature streams enter parallel Res2Net backbone networks of different depths. The flow passes through a relatively deep network (three Bottle2necks), and The processed features are passed through a shallower network (two Bottle2necks) to match their respective feature complexities. and .

[0035] Attention Fusion Network: and The concatenated data is fed into a small attention network to generate two channel weights. and Through weighted fusion, the model can dynamically adjust its focus on low-level and high-level information, ultimately outputting fused features. .

[0036] Dependency capture module: The core structure of this module includes an improved Transformer encoder block, which comprises three stacked multi-head attention mechanism layers.

[0037] Passband recalibration: A Squeeze-and-Excitation (SE) submodule is added before the improved Transformer encoder block to adaptively adjust the weights of each feature channel, prompting the model to focus on the channel with the most information.

[0038] Key Design Elements: Weight Sharing and Residual Connections: This module employs stacked multi-head attention layers and introduces a weight sharing mechanism. Similar to the standard Transformer, the module retains residual connections to ensure training stability and smooth information flow. The core innovation of this design lies in weight sharing, which forces attention modules at different levels to learn a general, abstract spoofing pattern, rather than redundant features specific to a particular level, thereby significantly improving the model's generalization ability. The final output is global features. .

[0039] Category module: Global features output by the dependency capture module After compressing the time dimension using global average pooling (GAP), the result is fed into a fully connected layer (FC) and a sigmoid activation function to obtain the final spoofing probability. In practice, the model will output two scores; if the false score is higher, it is considered false; otherwise, it is considered true.

[0040] It is worth noting that, in practice, the classification module may also take other forms. It should be noted that the low-high consistency detection model uses a weighted binary cross-entropy loss function for optimization during training to address the imbalance between the number of real and fake samples. The expression of the loss function is as follows: in, , Both represent weight parameters, which are determined based on the number of true and false data points in the training set. The probability of forgery output by the model. This represents the label value of the sample.

[0041] The effectiveness of the embodiments of the present invention will be verified through the following verification experiments: Training data: The ASVspoof 2019 LA dataset was mainly used for model training.

[0042] Test data: To comprehensively evaluate the model's generalization ability, it was tested on several publicly available cross-domain datasets, including In-the-Wild (ITW, a dataset in a natural environment), Fake-or-Real (FoR, a fake or real dataset), and FakeAVCeleb (FAC, a fake audio-video celebrity dataset).

[0043] Evaluation metrics: Equal Error Rate (EER) and Average EER (AvgEER) are used as the main performance metrics. The lower the EER value, the better the performance.

[0044] Experimental environment: The model was trained on an NVIDIA A100 GPU using the Adam optimizer with a learning rate of [missing information]. .

[0045] Existing models: RawGAT-ST is a forgery detection method proposed in 2021 that combines graph attention networks (GAT) and spatiotemporal modeling (ST).

[0046] AASIST, proposed in 2022, is a model based on graph attention networks to refine the modeling of spectrum-time relationships. This model constructs a heterogeneous graph from the spectral and temporal information of audio and uses a graph attention mechanism to simultaneously model the complex relationships between them. It is a representative work that uses advanced network structures to improve detection performance.

[0047] SSLAS, proposed in 2022, is a detection method that combines the self-supervised learning model Wav2Vec 2.0 with data augmentation. This method utilizes a model pre-trained on large-scale unlabeled data to extract high-quality, general audio representations, marking a new stage in detection technology by leveraging large-scale pre-trained models.

[0048] PSDL is a forgery detection method proposed in 2022 based on self-supervised feature distillation learning for pseudo-labels.

[0049] AMSDF is a forgery detection method based on attention-based multi-scale differential feature learning, proposed in 2024.

[0050] SLS is a forgery detection method proposed in 2024 that combines self-supervised feature fusion and contrastive learning.

[0051] aletheia is an anti-spoofing and forgery identification method proposed in 2024, featuring robust feature purification.

[0052] Nest2Net is an end-to-end model with a nested dual-network structure proposed in 2025 for forgery detection. The embodiments of the present invention have achieved state-of-the-art performance on multiple datasets, especially in the generalization ability test.

[0053] Table 1. Comparison of the overall performance of the embodiments of the present invention (Ours) with various existing models. This invention also provides an audio deepfake identification system based on representation inconsistency detection, which is used to perform the audio deepfake identification method based on representation inconsistency detection as described above.

[0054] This invention also provides a computer-readable storage medium storing a computer program for audio deepfake identification based on representation inconsistency detection, wherein the computer program causes a computer to execute the audio deepfake identification method based on representation inconsistency detection as described above.

[0055] This invention also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the audio depth forgery recognition method based on representation inconsistency detection as described above.

[0056] In summary, compared with existing technologies, it has the following beneficial effects: 1. This invention, for the first time, proposes and verifies that the core detectable clue for deepfake audio lies in the mismatch between low-level acoustic representation and high-level semantic representation. This innovative discovery fundamentally restructures the focusing logic of the detection task, upgrading the traditional passive identification targeting specific flaws to active verification based on internal consistency. It has both solid theoretical support and can overcome scenario limitations, providing stronger generalization ability and long-term application value for cross-domain and multi-type deepfake audio detection.

[0057] 2. The asymmetric fusion module designed in this embodiment of the invention forces the model to integrate and coordinate information from different semantic levels through asymmetric processing and gated fusion. Verified by singular value decomposition analysis, this module can effectively generate feature representations with higher effective rank and denser information, structurally resisting low-rank degradation of the feature space.

[0058] 3. The core innovation of the dependency capture module in this embodiment of the invention lies in the introduction of a weight sharing mechanism. By allowing stacked attention modules to share parameters, this design forces the model to learn a more abstract and general representation of the fake subspace, rather than learning a separate set of parameters for each layer. This design acts as an efficient regularization method, preventing the model from overfitting to features at specific layers and significantly improving its generalization ability against unseen attacks. Simultaneously, the module retains standard residual connections to ensure the stability and efficiency of deep network training.

[0059] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0060] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying deepfake audio based on representation inconsistency detection, characterized in that, The audio deep forgery detection method processes the audio waveform to be detected through a pre-constructed low-high consistency detection model and outputs the forgery probability of the audio waveform to be detected. The low-high consistency detection model includes a phased feature extraction module, a fusion module, a dependency capture module, and a classification module. The phased feature extraction module includes a pre-trained 24-layer XLS-R model. The XLS-R model is used to extract features from the audio waveform. The first 21 layers of the XLS-R model output low-level acoustic representations, and the last 3 layers output high-level semantic representations. The fusion module is used to fuse low-level acoustic representations and high-level semantic representations to obtain fused features; The dependency capture module is used to learn a general forgery pattern from the fused features and output global features; The classification module is used to process global features and output the forgery probability of the audio waveform to be detected.

2. The audio deepfake identification method based on representation inconsistency detection as described in claim 1, characterized in that, The fusion module is an asymmetric fusion module, comprising two independent 1×1 convolutional layers and max pooling layers, two first Res2Net backbone networks and a second Res2Net backbone network of different depths, and an attention network; The two independent 1×1 convolutional layers and max-pooling layers respectively characterize the lower-level acoustics. and high-level semantic representation Projecting onto a unified intermediate dimension yields... and ; The first Res2Net backbone network pair Processing is performed to obtain features. The second Res2Net backbone network for Processing is performed to obtain features. , The attention network is based on features and The concatenated features are processed to generate two channel weights. and , for features and Perform weighted fusion and output fused features.

3. The audio deepfake identification method based on representation inconsistency detection as described in claim 2, characterized in that, The first Res2Net backbone network includes three dual-bottleneck structures; the second Res2Net backbone network includes two dual-bottleneck structures.

4. The audio deepfake identification method based on representation inconsistency detection as described in claim 1, characterized in that, The dependency capture module includes a Transformer encoder block; Specifically, a squeeze-excitation submodule is embedded before the Transformer encoder block, which includes three stacked multi-head attention mechanism layers.

5. The audio deepfake identification method based on representation inconsistency detection as described in claim 1, characterized in that, The classification module includes global average pooling, a fully connected layer, and an activation function layer; the classification module processes global features. The expression for the processing procedure is as follows: in, The probability of forgery of the output audio waveform to be detected. This represents the activation function. Indicates global average pooling. This indicates fully connected layer processing.

6. The audio deepfake identification method based on representation inconsistency detection as described in any one of claims 1 to 5, characterized in that, The audio waveform to be detected is the audio waveform after enhancement processing of the original audio waveform.

7. The audio deepfake identification method based on representation inconsistency detection as described in any one of claims 1 to 5, characterized in that, The loss function during the training process of the pre-built low-high consistency detection model includes: in, , All represent weight parameters. The probability of forgery output by the model. This represents the label value of the sample.

8. An audio deepfake detection system based on representation inconsistency detection, characterized in that, The audio deepfake detection system is used to perform the audio deepfake detection method based on representation inconsistency detection as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program for audio deepfake identification based on representation inconsistency detection, wherein the computer program causes a computer to perform the audio deepfake identification method based on representation inconsistency detection as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising methods for performing the audio depth forgery identification method based on representation inconsistency detection as described in any one of claims 1 to 7.

Images