Audio depth forgery detection method based on adaptive feature fusion and adversarial disturbance

Abstract

The invention discloses an audio deep forgery detection method based on adaptive feature fusion and adversarial disturbance. The audio deep forgery detection method comprises the following steps: S1, constructing an audio deep forgery detection framework; s2, a mel-band antagonism disturbance module is adopted to generate an antagonism sample; s3, processing the audio features by adopting a self-adaptive audio feature fusion module; and S4, inputting the output features into a decoder of a Whisper model, optimizing model parameters through end-to-end training, and after the training is completed, receiving an input audio and outputting a deep forgery detection result by the model. The audio deep forgery detection method based on adaptive feature fusion and adversarial disturbance overcomes the defects that an existing audio deep forgery detection method is limited in generalization ability and insufficient in robustness, and the robustness of the audio deep forgery detection method is improved by dynamically adjusting feature compression dimensions and injecting targeted adversarial disturbance. And the adaptation capability of the model to heterogeneous audio data and the resistance to unknown attacks are improved.

Description

Technical Field

[0001] This invention belongs to the field of audio processing and artificial intelligence security technology, specifically relating to an audio deepfake detection method based on adaptive feature fusion and adversarial perturbation. Background Technology

[0002] In recent years, speech generation technology has developed rapidly, with text-to-speech and speech conversion systems continuously improving the naturalness and quality of the generated speech, leading to its widespread application in virtual assistants, content creation, and language learning. However, the misuse of this technology has given rise to deepfake audio content. Malicious attackers spread forged audio through social media platforms and conferencing tools, causing security and privacy issues such as the spread of false information, damage to personal reputation, and manipulation of public opinion, posing a serious threat to the trust system of the online ecosystem.

[0003] However, existing audio deep forgery detection (ADD) methods have two major problems: First, their generalization ability is limited. The fixed bottleneck structure cannot adapt to the heterogeneous information density caused by different sources, recording environments and forgery techniques. The performance of the model drops significantly when facing unknown datasets or forgery methods. Second, their robustness is insufficient. Existing methods are mostly based on learning from known synthesis domains and do not fully consider perturbations in the real world (such as background noise and speech rate changes) and unknown forgery attacks, making it difficult to reliably distinguish between real and forged samples.

[0004] Therefore, there is an urgent need for an audio deepfake detection method that can balance compression and fidelity and improve robustness against unknown attacks. Summary of the Invention

[0005] The main objective of this invention is to provide an audio deepfake detection method based on adaptive feature fusion and adversarial perturbation, which overcomes the shortcomings of existing audio deepfake detection methods, such as limited generalization ability and insufficient robustness. By dynamically adjusting the feature compression dimension and injecting targeted adversarial perturbations, the method improves the model's adaptability to heterogeneous audio data and its resistance to unknown attacks.

[0006] To achieve the above objectives, this invention provides an audio deepfake detection method based on adaptive feature fusion and adversarial perturbation, comprising the following steps: Step S1: Construct an audio deep fake detection framework. The framework uses the Whisper model as the backbone network and integrates a Mel band adversarial perturbation module and an adaptive audio feature fusion module. Step S2: Generate adversarial samples using the Mel band adversarial perturbation module; Step S3: The audio features are processed using an adaptive audio feature fusion module, and the optimal compression fusion method is dynamically selected to obtain the output features; Step S4: Input the output features into the decoder of the Whisper model, optimize the model parameters through end-to-end training, and after training is completed, the model receives the input audio and outputs the deepfake detection results.

[0007] As a further preferred technical solution to the above technical solution, step S2 is specifically implemented as follows: Step S2.1: Convert the audio input into a Mel-spectral domain representation to determine the target Mel frequency band; Step S2.2: Generate adversarial perturbations within the target Mel frequency band based on the projection gradient descent algorithm; Step S2.3: Add the generated adversarial perturbation to the original Mel spectrum to obtain the Mel spectrum with adversarial perturbation, and input the Mel spectrum into the encoder of the Whisper model to obtain the encoded audio features.

[0008] As a further preferred technical solution to the above technical solution, step S3 is specifically implemented as follows: Step S3.1: Construct a compression pool, which contains C compression layers with different pooling ratios. Each compression layer compresses the encoded audio features into the latent space of the corresponding dimension. Step S3.2: Design an adaptive router, generate selection weights based on encoding features, and increase the diversity of weight selection by injecting Gaussian noise; Step S3.3: Select the top k compression results with the highest weights, perform weighted fusion on them, and obtain the final output features.

[0009] As a further preferred technical solution of the above technical solution, in step S4, the input of the audio depth fake detection framework is the original waveform with a sampling rate of 16kHz. An early stopping mechanism is adopted during the model training process. When the error rate of the validation set does not improve for 5 consecutive cycles, the training is stopped.

[0010] As a further preferred technical solution to the above technical solution, the framework also includes a cross-language adaptation module, which adjusts the model based on multilingual datasets to improve cross-language detection capabilities.

[0011] The beneficial effects of this invention are as follows: 1. Improved generalization ability: Through the dynamic compression pool and adaptive routing of the AAFI module, the model can flexibly select the optimal compression method according to the input audio features, effectively adapt to heterogeneous audio data with different information densities, and significantly improve the generalization ability across datasets and spoofing methods.

[0012] 2. Enhanced robustness: The Mel-BAP mechanism injects adversarial perturbations into the Mel spectral domain, prompting the model to learn robust features aligned with human auditory perception, thereby improving its resistance to background noise, speech rate variations, and unknown forgery attacks. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0014] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0015] In the preferred embodiments of the present invention, those skilled in the art should note that the Whisper model and the like involved in the present invention can be considered as prior art.

[0016] Preferred embodiment.

[0017] like Figure 1 As shown, this invention discloses an audio deepfake detection method based on adaptive feature fusion and adversarial perturbation, comprising the following steps: Step S1: Construct an audio deep fake detection framework. The framework uses the Whisper model as the backbone network and integrates the Mel-BAP (Mel Band Adversarial Perturbation) module and the Adaptive Audio Feature Fusion (AAFI) module. Step S2: Generate adversarial examples using the Mel-BAP module; Step S3: The audio features are processed using the Adaptive Audio Feature Fusion (AAFI) module, and the optimal compression fusion method is dynamically selected to obtain the output features; Step S4: Input the output features into the Whisper model's decoder, optimize the model parameters through end-to-end training. After training, the model receives the input audio and outputs the deepfake detection results (using the AdamW optimizer with a learning rate of 1*10). -4 Weight decay 1*10 -5 End-to-end training is performed using weighted cross-entropy loss (with a weight of 1.0 for real samples and 0.8 for fake samples).

[0018] Specifically, step S2 is implemented as follows: Step S2.1: Convert the audio input to Mel-spectral domain representation and determine the target Mel frequency band (using Whisper's default preprocessing workflow, convert the original audio with a 16kHz sampling rate to Mel-spectral domain representation). , The transformed Mel spectrum is represented by T, where T is the number of time frames and F is the number of Mel frequency bins. Based on synthetic artifact analysis, the frequency band with the most significant synthetic artifacts (such as the Mel frequency band corresponding to 4-8kHz [59,79]) is selected as the target perturbation frequency band to ensure the perturbation is targeted. and These represent the lower and upper bound indices of the Mel frequency, respectively. (Constitutes the target Mel band) Step S2.2: Generate adversarial perturbations within the target Mel frequency band based on the Projected Gradient Descent (PGD) algorithm. The iterative formula for generating adversarial perturbations based on the Projected Gradient Descent (PGD) algorithm is as follows: ; This represents the adversarial perturbation in the (k+1)th iteration; For the projection function, Projected onto a radius of of On a spherical surface, ensure that the disturbance amplitude is within a reasonable range; The step size controls the magnitude of the perturbation update in each iteration; Used to extract gradient direction; The loss function L is expressed with respect to The gradient at the k-th iteration; The parameter is Detection model; (Indicates the true label of the audio sample (real or fake)); Step S2.3: Add the generated adversarial perturbation to the original Mel spectrum to obtain the Mel spectrum with adversarial perturbation (by restricting the perturbation to only act on the target frequency band using a binary mask D, the perturbed Mel spectrum is obtained). The Mel spectrum is then input into the encoder of the Whisper model to obtain the encoded audio features.

[0019] More specifically, step S3 is implemented as follows: Step S3.1: Construct a compression pool, which contains C compression layers with different compression pooling ratios (CPR). Each compression layer compresses the encoded audio features into a latent space of the corresponding dimension (each compression layer compresses the audio features output by the encoder). ( express It is A 1-dimensional real vector (The dimension of the encoder output features) is compressed to the corresponding dimension. Potential space, This enables feature representation coverage ranging from high compression to high fidelity, meeting the processing needs of different heterogeneous information densities. Step S3.2: Design an adaptive router, generate selection weights based on encoding features, and increase the diversity of weight selection by injecting Gaussian noise (by injecting Gaussian noise). To avoid concentrating weight selection on a few compressed layers, the exploratory nature of the process is increased. Weight calculation employs... The function ensures non-negativity, and the formula is as follows: , This represents the generated selection weight vector. It is a learnable weight matrix used to encode features. Mapped to the weight space, B represents the number of compression layers in the compression pool. It is a learnable weight matrix used to weight Gaussian noise. Step S3.3: Select the top k compressed results with the highest weights, perform weighted fusion on them, and obtain the final output features (through an upsampling network). The compressed features are reconstructed to the original dimensions, and then weighted and fused according to the weight ratios to obtain the final feature output. Balancing feature information content with computational efficiency This represents the final output feature after weighted fusion. This is the index set of the first k compressed results. The weights corresponding to the i-th compression result are: Compression features used to output the i-th compression layer Reconstructing to the original dimension ).

[0020] Furthermore, in step S4, the input to the audio depth fake detection framework is the original waveform with a sampling rate of 16kHz. An early stopping mechanism is used during model training. Training is stopped when the equal error rate (EER) of the validation set does not improve for 5 consecutive cycles.

[0021] Furthermore, the framework also includes a cross-language adaptation module, which adjusts the model based on multilingual datasets to improve cross-language detection capabilities.

[0022] This invention effectively solves the generalization and robustness problems of existing audio deepfake detection methods through the collaborative design of adaptive feature fusion and adversarial perturbation, providing a scalable and efficient solution for real-world audio security.

[0023] It is worth mentioning that the technical features such as the Whisper model involved in this patent application should be regarded as prior art. The specific structure, working principle, and possible control methods and spatial arrangement methods of these technical features can be adopted using conventional choices in the field, and should not be regarded as the inventive point of this patent. This patent will not be further elaborated in detail.

[0024] For those skilled in the art, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A method for detecting deep audio forgery based on adaptive feature fusion and adversarial perturbation, characterized in that, Includes the following steps: Step S1: Construct an audio deep fake detection framework. The framework uses the Whisper model as the backbone network and integrates a Mel band adversarial perturbation module and an adaptive audio feature fusion module. Step S2: Generate adversarial samples using the Mel band adversarial perturbation module; Step S3: The audio features are processed using an adaptive audio feature fusion module, and the optimal compression fusion method is dynamically selected to obtain the output features; Step S4: Input the output features into the decoder of the Whisper model, optimize the model parameters through end-to-end training, and after training is completed, the model receives the input audio and outputs the deepfake detection results.

2. The audio deepfake detection method based on adaptive feature fusion and adversarial perturbation according to claim 1, characterized in that, Step S2 is specifically implemented as follows: Step S2.1: Convert the audio input into a Mel-spectral domain representation to determine the target Mel frequency band; Step S2.2: Generate adversarial perturbations within the target Mel frequency band based on the projection gradient descent algorithm; Step S2.3: Add the generated adversarial perturbation to the original Mel spectrum to obtain the Mel spectrum with adversarial perturbation, and input the Mel spectrum into the encoder of the Whisper model to obtain the encoded audio features.

3. The audio deepfake detection method based on adaptive feature fusion and adversarial perturbation according to claim 2, characterized in that, Step S3 is specifically implemented as follows: Step S3.1: Construct a compression pool, which contains C compression layers with different pooling ratios. Each compression layer compresses the encoded audio features into the latent space of the corresponding dimension. Step S3.2: Design an adaptive router, generate selection weights based on encoding features, and increase the diversity of weight selection by injecting Gaussian noise; Step S3.3: Select the top k compression results with the highest weights, perform weighted fusion on them, and obtain the final output features.

4. The audio deepfake detection method based on adaptive feature fusion and adversarial perturbation according to claim 3, characterized in that, In step S4, the input to the audio depth fake detection framework is the original waveform with a sampling rate of 16kHz. An early stopping mechanism is used during model training. Training is stopped when the error rate of the validation set does not improve for 5 consecutive cycles.

5. The audio deepfake detection method based on adaptive feature fusion and adversarial perturbation according to claim 4, characterized in that, The framework also includes a cross-language adaptation module, which adjusts the model based on multilingual datasets to improve cross-language detection capabilities.

Images