Audio forgery detection method and system based on lightweight federated adversarial training

Abstract

The invention relates to the technical field of audio forgery detection, in particular to an audio forgery detection method and system based on lightweight federated adversarial training, and the method comprises the steps: a client extracts audio features through a lightweight network, and updates a model through local adversarial training; the server aggregates the client parameters to generate a global model and issues the global model; and the client performs forgery detection by using the global model. According to the method, a federal learning framework is adopted, original audio is not out of the local, only compressed model parameters are uploaded, high-performance collaborative learning under privacy protection is realized, a data island is broken, and the generalization ability is improved; time domain, frequency domain and frequency spectrum residual multi-dimensional features are fused with cross-modal attention, and adversarial disturbance training is introduced locally, so that feature resolution and anti-interference robustness are enhanced; through combination of pruning, quantification and distillation lightweight processing, the communication and calculation overhead is remarkably reduced, and mobile and IoT edge equipment can be efficiently deployed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio forgery detection, and particularly relates to an audio forgery detection method and system based on light federated adversarial training. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, deep learning has made breakthrough progress in speech synthesis. Speech synthesis technologies based on deep generative models such as generative adversarial networks (GAN), variational autoencoders (VAE), and diffusion models are becoming increasingly mature, and can generate extremely realistic artificial synthesized speech. These technologies play an important role in voice assistants, audiobook production, multilingual dubbing, and other fields, but at the same time, they also pose serious security risks. Malicious users can use these technologies to create false audio content, engage in identity impersonation, fraud, and false information dissemination, and other illegal activities, posing a serious threat to personal privacy, social security, and public trust.

[0003] Traditional audio forgery detection methods often rely on centralized data training, which has problems such as data privacy leakage and large computational resource consumption. At the same time, existing audio forgery detection methods lack adaptability to new forgery methods, and their detection accuracy and robustness need to be improved. SUMMARY

[0004] The present application aims to provide an audio forgery detection method and system based on light federated adversarial training, which aims to solve the problem of large computational and communication overhead of existing centralized audio forgery detection schemes.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides an audio forgery detection method based on light federated adversarial training, comprising the following steps:

[0006] The client updates the model after extracting audio features using a light network through local adversarial training;

[0007] The server aggregates the parameters of each client to generate a global model and distributes it;

[0008] The client uses the global model for forgery detection.

[0009] In the step of "the client updates the model after extracting audio features using a light network through local adversarial training", the client extracting audio features using a light network specifically includes:

[0010] Collecting raw audio;

[0011] Extracting audio deep features using a light backbone network after preprocessing the audio data;

[0012] The local generator receives the adversarial samples generated by the features and updates the local model parameters.

[0013] The step of "server aggregating parameters from various clients to generate a global model and distributing it" includes the following steps:

[0014] Receive encrypted model parameters from each client;

[0015] Execute the FedProx aggregation algorithm;

[0016] Generate a global model and distribute it to each client.

[0017] The section on "Client-side use of global model for forgery detection" includes the following steps:

[0018] Use the received global model to detect forgery in local audio;

[0019] Collect the detected audio data, perform detection on the audio data, and cache and encrypt the detection results locally before sending them back to the server.

[0020] In a second aspect, an audio forgery detection system based on lightweight federated adversarial training is used in the audio forgery detection method based on lightweight federated adversarial training described in the first aspect, comprising a data collection and preprocessing module, a lightweight audio forgery detection model construction module, a federated adversarial training module, and an audio forgery detection module.

[0021] The data collection and preprocessing module is used to collect audio data and preprocess it to extract audio features;

[0022] The lightweight audio forgery detection model building module is used to build a lightweight audio forgery detection model, including a feature extraction module, an adversarial training module, and a classification module.

[0023] The federated adversarial training module is used to conduct federated adversarial training on multiple devices. The devices share model parameter updates through a secure communication protocol to protect user data privacy.

[0024] The audio forgery detection module is used to detect forgery of unknown audio using a trained model and output the detection results.

[0025] This invention discloses an audio spoofing detection method based on lightweight federated adversarial training, comprising the following steps: A client extracts audio features using a lightweight network and updates the model through local adversarial training; a server aggregates parameters from each client to generate a global model and distributes it; the client uses the global model for spoofing detection. This invention employs a federated learning framework, ensuring the original audio remains locally, only uploading compressed model parameters, achieving high-performance collaborative learning while protecting privacy, breaking down data silos, and improving generalization ability. It enhances feature discrimination and robustness against interference by fusing multi-dimensional features from the time domain, frequency domain, and spectral residuals with cross-modal attention, and introducing adversarial perturbation training locally. Combined with lightweight processing such as pruning, quantization, and distillation, it significantly reduces communication and computational overhead, enabling efficient deployment on mobile and IoT edge devices. This solves the problem of high computational and communication overhead in existing centralized audio spoofing detection schemes. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an audio forgery detection method based on lightweight federated adversarial training provided by the present invention.

[0028] Figure 2 This is a flowchart showing how the client uses a lightweight network to extract audio features and then updates the model through local adversarial training.

[0029] Figure 3 It is a flowchart of the server aggregating parameters from various clients to generate a global model and then distributing it.

[0030] Figure 4 This is a flowchart of the client using a global model to perform forgery detection.

[0031] Figure 5 This is a schematic diagram of an audio forgery detection system based on lightweight federated adversarial training provided by the present invention.

[0032] In the diagram: 1-Data collection and preprocessing module, 2-Lightweight audio forgery detection model building module, 3-Federated adversarial training module, 4-Audio forgery detection module. Detailed Implementation

[0033] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0034] Please see Figures 1 to 4 In a first aspect, the present invention provides an audio forgery detection method based on lightweight federated adversarial training, comprising the following steps:

[0035] The S1 client uses a lightweight network to extract audio features and then updates the model through local adversarial training.

[0036] S11 captures raw audio;

[0037] Specifically, the client's local audio dataset Where x i For audio clips, y i ∈{0,1} is its label (0 indicates fake, 1 indicates real).

[0038] S12 preprocesses the audio data and then uses a lightweight backbone network to extract audio depth features.

[0039] The local generator receives features, generates adversarial examples, and updates the local model parameters.

[0040] Specifically, for the input audio x i The process includes standardization, silence removal, and framing. This invention does not directly input the original waveform into the network, but rather extracts three types of features in parallel to comprehensively capture forgery traces:

[0041] Temporal Features: Extracting features such as short-time energy and zero-crossing rate to form a temporal feature vector V. temp

[0042] Spectral Features: Calculate the Mel-Spectrogram or Mel-Frequency Cepstral Coefficients (MFCCs) to obtain the frequency domain feature matrix M. freq

[0043] Spectral Residual Features: This involves performing saliency detection on the spectrogram and calculating its spectral residuals. The aim is to highlight anomalous frequency domain patterns introduced during the forgery process that are imperceptible to the human ear, resulting in the residual feature matrix M. res .

[0044] The S13 local generator receives features, generates adversarial examples, and updates the local model parameters.

[0045] Specifically, the server initializes a global audio forgery detection model. The server distributes the current version of the global model to all clients participating in this round of training. Each client independently trains its model on its local dataset. This process not only includes the usual forward and backward propagation, but also integrates three key steps: multi-feature extraction and fusion, adversarial example generation, and lightweight post-training processing.

[0046] During local training, adversarial perturbation δ is generated based on the current model's loss function using methods such as Fast Signed Gradient Method (FGSM) or Projected Gradient Descent (PGD).

[0047]

[0048] Where ε is the perturbation magnitude. The perturbation is applied to the original input x, resulting in the adversarial sample x′ = x + δ. The original sample x and the adversarial sample x′ are used together in training. The loss function is the sum of the original loss and the adversarial loss:

[0049] L total =L(θ,x,y)+λL(θ,x′,y)

[0050] Here, λ is a tradeoff hyperparameter. In this way, while learning the features of normal samples, the model is also forced to remain stable against adversarial perturbations, thereby significantly improving its robustness.

[0051] Lightweight operation: After local training is completed, the model θ is adjusted. k Perform one or more of the following operations to generate a lightweight model.

[0052] Pruning: Remove connections in the model whose weights are below a threshold to create a sparse model.

[0053] Quantization: Converting model weights from 32-bit floating-point numbers to low-precision data (such as 8-bit integers).

[0054] Knowledge distillation: Fitting the behavior of the original model with a small-scale "student model".

[0055] Upload: The client only updates the lightweight model. Uploading to the server greatly reduces the amount of communication data.

[0056] The S2 server aggregates parameters from each client to generate a global model and then distributes it.

[0057] S21 receives encrypted model parameters from each client;

[0058] Specifically, lightweight model updates collected from K clients

[0059] S22 executes the FedProx aggregation algorithm;

[0060] Specifically, the server employs an adaptive weighted aggregation strategy, such as the classic FedAvg algorithm:

[0061]

[0062] Where n k Let k be the amount of data from client k, and N be the total amount of data from all participating clients. Servers can also employ more advanced algorithms (such as Moon) to handle the Non-Identical Distribution (Non-IID) problem.

[0063] S23 generates a global model and distributes it to each client.

[0064] Specifically, the updated global model This will be used for the next round of distribution and training.

[0065] The S3 client uses a global model for forgery detection.

[0066] S31 uses the received global model to detect forgery in the local audio;

[0067] Specifically, the global model is loaded, forward inference is performed on real-time audio, and the forgery probability and tampering time point are output.

[0068] S32 collects the detected audio data, performs detection on the audio data, and locally caches and encrypts the detection results before sending them back to the server.

[0069] Specifically, the client loads the global model, performs forward inference on the real-time audio stream, and outputs the forgery probability and tampering time point; the detection results are cached locally and encrypted before being sent back to the server for continuous closed-loop optimization.

[0070] Please see Figure 5 Secondly, an audio forgery detection system based on lightweight federated adversarial training is used in the audio forgery detection method based on lightweight federated adversarial training described in the first aspect, including a data collection and preprocessing module, a lightweight audio forgery detection model construction module, a federated adversarial training module, and an audio forgery detection module.

[0071] The data collection and preprocessing module is used to collect audio data and preprocess it to extract audio features;

[0072] The lightweight audio forgery detection model building module is used to build a lightweight audio forgery detection model, including a feature extraction module, an adversarial training module, and a classification module.

[0073] The federated adversarial training module is used to conduct federated adversarial training on multiple devices. The devices share model parameter updates through a secure communication protocol to protect user data privacy.

[0074] The audio spoofing detection module utilizes a pre-trained model. In this embodiment, the data collection and preprocessing module collects and preprocesses audio data to extract audio features; the lightweight audio spoofing detection model construction module constructs a lightweight audio spoofing detection model, including a feature extraction module, an adversarial training module, and a classification module; the federated adversarial training module performs federated adversarial training on multiple devices, with devices sharing model parameter updates via a secure communication protocol to protect user data privacy; and the audio spoofing detection module utilizes the pre-trained model.

[0075] I. This invention employs a Federated Learning (FL) framework as its basic architecture. Under this framework, the technical characteristics are as follows: the raw audio data of each client is always kept locally, and only model updates (gradients or weights) are uploaded to the server for aggregation. This technology fundamentally eliminates the risk of user data privacy leakage during training, perfectly complying with increasingly stringent data security regulations (such as GDPR and the Personal Information Protection Act). Simultaneously, it successfully breaks down "data silos," enabling the effective utilization of diverse local data (covering different dialects, recording devices, environmental noise, and forgery techniques) scattered across various edge terminals. The direct result is that the final trained global model learns broader and more general feature representations, thus exhibiting superior generalization ability and higher detection accuracy when faced with speakers, recording environments, or novel forgery tools not encountered during training.

[0076] II. This invention innovatively integrates a multi-feature attention fusion module and an adversarial training module locally on the client side. 1) The technical feature of dynamically fusing multiple features (time domain, frequency domain, spectral residual) with a cross-modal attention mechanism allows the model to no longer treat all features equally, but rather adaptively focus on and amplify the most discriminative forgery traces in the input audio. 2) The technical feature of actively generating adversarial examples and participating in training is equivalent to providing the model with "stress testing" and "immunization" during the training phase, forcing the model to learn to maintain stable decision boundaries under perturbation. The former greatly enhances the model's feature representation ability, enabling it to more sensitively capture the subtle and varied artifacts left by different forgery techniques. The latter significantly improves the model's robustness and security, enabling it to maintain stable performance when facing adversarial attacks aimed at deceiving the model, greatly reducing the risk of being maliciously bypassed in actual deployment. The combination of the two ensures the high reliability and high availability of the detection system in complex real-world environments.

[0077] Third, it significantly reduces communication and computing costs, enabling efficient deployment of advanced models on resource-constrained terminals. After local training on the client side, this invention introduces a key technical feature: lightweight model processing (including pruning and quantization operations). This processing greatly compresses the size of the model updates that need to be uploaded, thereby significantly reducing the communication bandwidth and transmission latency required for each round of federated learning, effectively solving the communication bottleneck problem in federated learning. Simultaneously, the lightweight model itself also has less computation and lower energy consumption. The direct benefit of this is that the resource requirements of this invention are significantly reduced on the client side, enabling edge terminals such as mobile phones and IoT devices with limited computing power, storage space, and battery power to participate in federated training and deploy the final advanced detection model feasiblely and efficiently, greatly expanding the application scenarios and scope of the technology.

[0078] Beneficial effects:

[0079] 1. Achieving High-Performance Collaborative Learning with Privacy Protection: By introducing a federated learning framework, the raw audio data from each client does not need to leave its local machine; only the processed model update parameters are uploaded to the server for aggregation. This fundamentally eliminates the risk of data privacy leakage during training, legally and compliantly utilizes data resources scattered across various locations, successfully breaks down "data silos," and enables the final global model to learn a wider and more diverse feature distribution, thereby significantly improving the model's generalization detection capability on unknown data and unknown forgery methods.

[0080] 2. Significantly Enhanced Feature Discriminative Power and Anti-interference Robustness: First, by extracting multi-dimensional features such as time domain, frequency domain, and spectral residuals, and utilizing a cross-modal attention mechanism for dynamic weighted fusion, the model can adaptively focus on and strengthen the feature regions most sensitive to forgery, improving the quality of feature representation and discriminative power. Second, by actively introducing adversarial perturbation samples during local training, the model learns how to resist such attacks during the training phase, essentially "immunizing" the model. This greatly enhances the model's stability and robustness against malicious adversarial samples, improving the overall security of the system.

[0081] 3. Significantly Reduced Communication and Computational Overhead, Enabling Edge Deployment: By performing lightweight processing such as pruning, quantization, and distillation on the locally trained model, and then updating and uploading the compressed model, the amount of data transmitted in each federated iteration is greatly reduced, alleviating communication bottlenecks and saving bandwidth and energy consumption. This enables the solution to run efficiently on edge terminals such as mobile phones and IoT devices with limited computing and communication resources, achieving the widespread deployment of advanced detection capabilities, broadening the application boundaries of the technology, and truly making it suitable for distributed audio security detection application scenarios.

[0082] The above-disclosed embodiments are merely preferred embodiments of the audio forgery detection method and system based on lightweight federated adversarial training of the present invention. Of course, they should not be construed as limiting the scope of the present invention. Those skilled in the art can understand that implementing all or part of the above embodiments and making equivalent changes in accordance with the claims of the present invention still fall within the scope of the invention.

Claims

1. A method for audio forgery detection based on lightweight federated adversarial training, characterized in that, The method comprises the following steps: The client updates the model through local adversarial training after extracting audio features using a lightweight network; The server aggregates the parameters of each client to generate a global model and distributes it; The client uses the global model for forgery detection.

2. The lightweight federated adversarial training based audio deepfake detection method of claim 1, wherein, In the step of "the client updates the model through local adversarial training after extracting audio features using a lightweight network", the client extracting audio features using a lightweight network comprises the following steps: Collecting original audio; Extracting audio deep features using a lightweight backbone network after preprocessing the audio data; The local generator receives the features to generate adversarial samples and update the local model parameters.

3. The lightweight federated adversarial training based audio deepfake detection method of claim 2, wherein, In the step of "the server aggregates the parameters of each client to generate a global model and distributes it", the method comprises the following steps: Receiving encrypted model parameters from each client; Executing the FedProx aggregation algorithm; Generating a global model and distributing it to each client.

4. The lightweight federated adversarial training based audio deepfake detection method of claim 3, wherein, In the step of "the client uses the global model for forgery detection", the method comprises the following steps: Using the received global model to detect local audio forgery; Collecting the detected audio data for detection, and encrypting the detection results and returning them to the server.

5. An audio deepfake detection system based on lightweight federated adversarial training, used for the audio deepfake detection method based on lightweight federated adversarial training in any one of claims 1-4, characterized in that, The method comprises a data collection and preprocessing module, a lightweight audio forgery detection model construction module, a federated adversarial training module, and an audio forgery detection module; The data collection and preprocessing module is used to collect audio data and perform preprocessing to extract audio features; The lightweight audio forgery detection model construction module is used to construct a lightweight audio forgery detection model, including a feature extraction module, an adversarial training module, and a classification module; The federated adversarial training module is used to perform federated adversarial training on multiple devices, and the devices share model parameter updates through a secure communication protocol to protect user data privacy; The audio forgery detection module is used to use the trained model to detect unknown audio forgery and output the detection results.

Images