A method, system, device and medium for enhancing detection features for counterfeit audio

Abstract

The application discloses a kind of detection feature enhancement methods, systems equipment and medium for counterfeit audio, specifically related to belong to information security field.First, the original audio is encoded into high-dimensional latent representation by variational autoencoder, and the reference embedding parameters generated by audio-text contrast learning model are input into diffusion model;Use diffusion model to denoise latent representation in multiple steps, generate pure speech features and decode into enhanced audio.Afterwards, the audio is processed by the feature quality optimization module, and the detection score matrix is obtained by inputting the deep fake detection model, and whether to perform model iteration optimization is judged according to the preset threshold value: if the weighted score exceeds the threshold value, the detection model parameters are updated by loss function, and finally the optimized detection model and result are output.The application can process different types of noisy speech input, has important potential to adapt to unknown types of noise, and is used for high-precision detection of counterfeit audio features.

Description

Technical Field

[0001] This invention belongs to the field of information security, and specifically relates to a method, system, device and medium for enhancing the detection features of fake audio. Background Technology

[0002] In recent years, the rapid development of deep synthesis technologies such as Generative Adversarial Networks (GANs) and autoencoders has significantly lowered the technical barrier to speech forgery. Attackers can use open-source tools to generate high-fidelity forged audio in batches using a small number of samples. These forged audios are then disseminated on a large scale through social media, severely undermining the credibility of public information. Existing detection techniques typically rely on differential analysis of acoustic features, such as traditional features like Mel-frequency cepstral coefficients (MFCCs). However, during feature extraction, the quality of features extracted by the model is affected by noise or other sounds in the samples, challenging the model's accuracy in detecting samples. Therefore, developing forged audio detection techniques that can efficiently mask noise features has become an urgent problem to be solved in the field of information security.

[0003] Currently, the main feature extraction methods used for detecting fake audio are:

[0004] (1) Pham L, Lam P, Nguyen T, et al. Deepfake audio detection using spectrogram-based feature and ensemble of deep learning models[C] / / 2024IEEE 5th International Symposium on the Internet of Sounds(IS2).IEEE,2024:1-5. This method uses a combination of various transforms and filters to generate spectrograms to measure the characteristics of audio signals. By analyzing the spectral information of audio in multiple dimensions, it captures subtle differences in fake audio, such as artifacts at different frequencies and changes in specific frequency bands.

[0005] This method achieves certain detection performance by leveraging the learning capabilities of deep learning models, but it does not mention its detection capabilities across different datasets and lacks generalization ability. Furthermore, it extracts a large number of features from each audio file while neglecting important features, which increases the burden on the classification model and reduces its efficiency.

[0006] (2) Iqbal F, Abbasi A, Javed AR, et al. Deepfake Audio Detection via Feature Engineering And Machine Learning [C] / / CIKMWorkshops.2022:1-12. This work proposes a method for detecting fake audio based on multiple feature extraction and selection techniques. This method utilizes multiple feature extraction techniques to obtain audio signal features. By analyzing audio from different dimensions, it captures key features that can distinguish real and fake audio, such as the spectral characteristics and zero-crossing rate of the audio.

[0007] The method described does not utilize speech noise to simulate real-world noisy environments. In the real world, speech signals are often affected by noise and dynamic environments, which can severely degrade the detection performance of this method, thus limiting its practicality. Summary of the Invention

[0008] The purpose of this invention is to provide a method, system, device, and medium for enhancing the detection features of fake audio.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] This invention provides a feature enhancement method for detecting fake audio, the specific steps of which are as follows:

[0011] A feature enhancement method for detecting fake audio, the specific steps of which are as follows:

[0012] Step 1: The encoder of the variational encoder encodes the one-dimensional audio waveform to be detected into a high-dimensional latent representation x. m ;

[0013] Step 2: Utilize the audio-text contrastive learning pre-trained model to represent the feature description of the text as the reference embedding of the audio, and combine it with the time step t of the diffusion model as the input of the diffusion model to generate conditional parameters to control the scaling and offset of the diffusion model.

[0014] Step 3: Combining latent representation and reference embedding, multi-step denoising sampling is performed using a diffusion model to generate a clean speech latent representation;

[0015] Step 4: Decode the generated clean speech latent representation into a speech waveform using a variational autoencoder, and output the enhanced one-dimensional audio x0∈R. N×C ;

[0016] Step 5: Input the enhanced audio from Step 4 into the deep fake detection model Model to obtain the detection result Res∈R 1×N The audio scoring matrix is ​​Score∈RN×5 And calculate the weight matrix Weights;

[0017] Step 6: Set the hyperparameter ScoreD, score the enhanced audio to obtain the score matrix ScoreS, and perform row-by-row dot product through the weight matrix Weights to generate the weighted score ScoreO; if ScoreO>ScoreO, proceed to step 7, otherwise proceed to steps 1 to 4, and regenerate the weighted score ScoreO according to step 6.

[0018] Step 7: Optimize the loss function of the model Model based on the weighted scoring matrix ScoreO;

[0019] Step 8: Output the optimized detection model parameters and detection results for high-precision detection of forged audio features. Further, the latent representation extracted in Step 1 is x. m :

[0020] x m ∈R N×C

[0021] Where N is the number of feature frames, determined by the audio length and frame length, and C is the dimension of the latent features.

[0022] Furthermore, given the noise of the diffusion model at a time step t in step 3, the probability distribution of the data at time step t-1 is defined by the following formula:

[0023]

[0024] in, The variance scheduling parameter β, artificially set during positive noise addition. t The variance of the denoising step in the reverse process is obtained. p represents an estimate of the mean of the data from the previous time step. θ Represents conditional probability. Let I represent a multivariate normal distribution, and let I represent the identity matrix.

[0025] Furthermore, the weight matrix Weights in step 5 is:

[0026] Weights=softmax(Res·Score).

[0027] Furthermore, the weighted scoring matrix ScoreS in step 6 is:

[0028] ScoreO = ScoreS·Weights.

[0029] Furthermore, in step 7, ScoreO is added to the loss function of the Model for weighting. Let the previous loss of the Model be LossO, and the new loss be:

[0030] Loss = αScoreO·LossO.

[0031] A computer device / apparatus / system includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement steps of a method for enhancing the detection features of spoofed audio.

[0032] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements steps of a method for enhancing the detection features of spoofed audio.

[0033] An electronic device, characterized in that it comprises:

[0034] Memory, used to store computer programs;

[0035] A processor for executing the computer program to implement the instruction tracing method described in the method for enhancing the detection features of spoofed audio.

[0036] The beneficial effects of this invention are as follows:

[0037] This invention proposes an innovative feature enhancement method for detecting fake audio. It enhances speech features from the perspective of improving the quality of the audio to be detected, aiming to improve audio samples with high noise levels. Unlike existing feature extraction methods, this invention focuses on filtering unnecessary components in the audio to be detected, thereby enhancing the features of the speech portion of the audio. Whether it is traditional manual feature extraction or end-to-end feature learning, this method can enhance the effectiveness of features, further enhance speech features, and weaken the impact of features such as noise on the stability of fake audio detection. By enhancing the speech features of fake audio detection from the perspective of data preprocessing and optimizing the model's extraction of effective features, the proposed method has the potential to improve the accuracy of detection results.

[0038] Compared to existing works that only process single types of noise (lacking generalization ability), this invention, by leveraging the powerful generation capabilities and large-scale data volume of the audio-text contrastive learning pre-trained model (CLAP) and diffusion model, can process different types of noisy speech input and has significant potential to adapt to unknown types of noise. Attached Figure Description

[0039] Figure 1 A flowchart of the method of the present invention. Detailed Implementation

[0040] The present invention will now be further described with reference to the accompanying drawings.

[0041] This invention discloses a method, system, device, and medium for enhancing the detection features of forged audio. The method is based on... Figure 1 The specific steps are as follows:

[0042] Step 1: Extract the latent representation of the audio to be detected. Given a one-dimensional audio waveform to be detected, a variational autoencoder (VAE) can encode it into a high-dimensional latent representation. The core architecture of VAE consists of two main parts: an encoder and a decoder. In the encoder stage, multi-layer convolutional neural networks (CNNs) and fully connected neural networks (FCNs) are used to perform detailed feature extraction on the input audio waveform along the time dimension. The extracted latent representation is x. m :

[0043] x m ∈R N×C ,

[0044] Where N is the number of feature frames, determined by the audio length and frame length, and C is the dimension of the latent features.

[0045] Step 2: Represent the feature description as a reference embedding. Given a feature description, the Audio-Text Contrastive Learning Pre-trained Model (CLAP) can represent the text feature description as a reference embedding of the audio. This embedding will be used as conditional information along with the time step t of the diffusion model, with the aim of regressing the scaling and offset parameters in each dimension. These parameters will be integrated into each module of the diffusion model.

[0046] Step 3: Denoise and sample the audio to be detected to generate a latent speech representation. Using the reference embedding obtained in Step 2 and time step t, perform multi-step denoising sampling on the noisy latent representation to obtain a clean speech latent representation. Given a noisy frequency (noise from the diffusion model) at time step t, the probability distribution of the data at time step t-1 is defined by the following equation:

[0047]

[0048] in, The variance scheduling parameter β, artificially set during positive noise addition. t The variance of the denoising step in the reverse process is obtained. p represents an estimate of the mean of the data from the previous time step. θ Represents conditional probability. Let I represent a multivariate normal distribution, and let I represent the identity matrix.

[0049] Step 4: Obtain the clean speech to be detected through variational autoencoder (VAE decoder). The decoding process decodes the speech latent representation generated by the diffusion model into a speech waveform, obtaining audio with enhanced speech features. The enhanced one-dimensional audio is then output as follows:

[0050] x0∈R N×C .

[0051] Step 5: Input the enhanced audio from Step 4 into the existing model for inference to obtain the indicator weights. Given a deepfake detection model Model, input the audio into Model to obtain the detection result Res, where...

[0052] Res∈R 1×N

[0053] The audio scoring matrix is ​​Score, where

[0054] Score∈R N×5

[0055] Calculate the weight of each score using the following formula:

[0056] Weights=softmax(Rres·Score)

[0057] Step 6: Optimize model training using Weights, set the hyperparameter Score, score the enhanced audio to obtain the score matrix ScoreS, perform row-wise dot product through the weight matrix Weights to generate the weighted score ScoreO; if ScoreO>ScoreD, proceed to step 7, otherwise proceed to steps 1 to 4, and regenerate the weighted score ScoreO according to step 6.

[0058] Step 7: Perform feature enhancement on the audio to obtain a new ScoreO. The Score can be added to the model's loss function for weighting. Let the previous Loss of the model be LossO, and the new Loss be:

[0059] Loss=αScoreOLossO

[0060] This allows the model to focus more on high-quality audio information, thereby optimizing the training process.

[0061] Step 8: Output the optimized detection model parameters and detection results for high-precision detection of fake audio features.

[0062] Example 1:

[0063] The effectiveness of this invention is evaluated by performing feature enhancement on the audio waveform. After speech enhancement, the audio in the dataset is assessed using National Information Quality Assurance (NISQA). This invention evaluates two types of datasets: a clean speech dataset (CFAD) for fake speech detection, and a noisy dataset (VoiceBank+Demand) with noise (including current noise, ambient noise, etc.) for a noisy, robust text-to-speech synthesis system using a deep recurrent neural network. The overall Mean Opinion Score (MOS), noise level, timbre, loudness, and discontinuities of the audio are compared before and after enhancement.

[0064] Table 1. Changes in scores after enhancement of Clean Speech Aid (CFAD)

[0065]

[0066] As shown in Table 1, the noise scores of the 320 clean speech samples were already high before the enhancement process of this invention. Therefore, there was no significant improvement in the NISQA score after speech enhancement, and the overall score decreased slightly. The results are consistent with the purpose of this invention.

[0067] Table 2. Changes in scores after enhancement of noisy speech (VoiceBank + Demand)

[0068]

[0069] As shown in Table 2, the NISQA scores of 320 noisy speech samples showed a significant improvement after the enhancement process of this invention. Because the speech samples themselves had low noise scores, the enhancement process of this invention was necessary to enhance the speech features.

[0070] Analysis of the two tables above shows that this invention aims to improve samples with high noise intensity. Noise scoring can be performed first. For speech audio with low noise scores, speech feature enhancement using this invention is required. If the sample noise intensity score is high, it can directly proceed to the downstream task.

[0071] All relevant content of each step involved in the aforementioned embodiments of the feature enhancement method for detecting fake audio can be referenced in the functional description of the corresponding functional module of the deep fake detection interpretable system based on causal analysis in the embodiments of the present invention, and will not be repeated here.

[0072] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0073] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a detection feature enhancement method for forged audio.

[0074] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the detection feature enhancement method for forged audio in the above embodiments.

[0075] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for enhancing the detection features of forged audio, characterized in that: The specific steps are as follows: Step 1: Encode the one-dimensional audio waveform to be detected into a high-dimensional latent representation using a variational autoencoder. ; Step 2: Utilize the audio-text contrastive learning pre-trained model to represent the feature description of the text as the reference embedding of the audio, and combine it with the time step t of the diffusion model as the input of the diffusion model to generate conditional parameters to control the scaling and offset of the diffusion model. Step 3: Combining latent representation and reference embedding, multi-step denoising sampling is performed using a diffusion model to generate a clean speech latent representation; Step 4: The generated clean speech latent representation is decoded into a speech waveform using a variational autoencoder, and the enhanced one-dimensional audio is output. ; Step 5: Input the enhanced audio from Step 4 into the deep fake detection model to obtain the detection results. The audio rating matrix is And calculate the weight matrix. ; Step 6: Set hyperparameters The enhanced audio is scored to obtain a scoring matrix. Through the weight matrix Perform row-by-row dot product to generate a weighted score. ;like > Proceed to step 7; otherwise, proceed to steps 1 through 4, and regenerate the weighted score according to step 6. ; Step 7: Based on the weighted scoring matrix Optimize the loss function of the model; Step 8: Output the optimized detection model parameters and detection results for high-precision detection of fake audio features.

2. The method for enhancing the detection features of forged audio according to claim 1, characterized in that: The latent representation extracted in step 1 is as follows : ； Where N is the number of feature frames, determined by the audio length and frame length, and C is the dimension of the latent features.

3. The method for enhancing the detection features of forged audio according to claim 1, characterized in that: Given the noise of the diffusion model at a time step t in step 3, the probability distribution of the data at time step t-1 is defined by the following formula: ； in, The variance scheduling parameter artificially set during positive noise addition The variance of the denoising step in the reverse process is obtained. This represents an estimate of the mean of the data from the previous time step. Represents conditional probability. Represents a multivariate normal distribution. Represents the identity matrix.

4. The method for enhancing the detection features of fake audio according to claim 1, characterized in that: The weight matrix in step 5 for: 。 5. The method for enhancing the detection features of fake audio according to claim 1, characterized in that: The weighted scoring matrix in step 6 for: 。 6. The method for enhancing the detection features of fake audio according to claim 1, characterized in that: Step 7 will O is added to the loss function of the Model for weighting, assuming the Model's previous loss function... yes new for: 。 7. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as claimed in any one of claims 1 to 6.

Images