Audio depth forgery detection method and device based on deep neural network, and product
Through deep neural networks and contrast learning optimization feature extraction, combined with self-attention mechanism classification network, the existing audio depth forgery detection is solved, and a more accurate and robust audio detection effect is achieved.
Patent Information
- Application Number
- CN202411309433.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2025-01-07
AI Technical Summary
Existing audio depth forgery detection methods are susceptible to manipulation attacks. Manipulation processing such as adjusting volume, speed, sampling rate, etc. hardly affects human perception, but will falsely alarm the detection results, resulting in inaccurate judgment results of the detection system.
Using a deep neural network-based detection method, the feature vector is extracted through the encoder and optimized model parameters using contrast learning and length loss, combined with a classification network of self-attention mechanism, capture complex relationships between features and enhance the model's resistance to manipulation attacks.
It improves the robustness of the model for manipulation attacks, ensures the accuracy and stability of detection results, and can more effectively distinguish between real audio and audio synthesized by deep forgery technology.
Smart Images
Figure CN119274580A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of cyberspace security technology, and relates to an audio deep fake detection method, device and product, and specifically to an audio deep fake detection method, device and product based on a deep neural network. Background Art
[0002] In recent years, deep learning has made significant progress in speech synthesis and speech conversion, allowing people to synthesize speech that sounds very natural and real. These technologies can help blind people read, help people with aphasia to speak again, and greatly reduce the threshold for dubbing. But on the other hand, technological progress also brings hidden dangers. Malicious attackers may use these technologies to synthesize the voice of a specific person, that is, audio deep fake, to spread false statements, telecommunications fraud, bypass voiceprint authentication systems, etc. These threats have made the detection of audio deep fakes a hot research issue that has attracted much attention.
[0003] For the detection of audio deepfakes, researchers focus on improving the model's detection effect on audio samples generated by unknown synthesis algorithms. The main work is to design the model's feature extraction module to extract more general and stable features. The current detection mechanism is mainly divided into two categories based on feature extraction methods: based on traditional manual design features and based on deep learning feature extraction. The traditional manual feature method uses carefully designed time-frequency spectrum features such as linear frequency cepstral coefficients and linear prediction cepstral coefficients, while the deep learning feature extraction method extracts features by reasonably designing the model and using a large amount of data to learn the model's parameters.
[0004] The above methods have achieved excellent results on classic large-scale datasets, but they still have limitations because they ignore an attack that is easy to launch: manipulation attacks. Attackers may manipulate the audio, such as adjusting the volume, speed, sampling rate, adding noise, etc. These operations have little effect on human perception of the audio, but they will greatly affect the judgment results of existing detection methods. For example, an audio deep fake may be detected as an audio deep fake with a very high confidence by the detection system when it is not manipulated. However, if the attacker simply adjusts the waveform and lowers the volume of the audio, it will be mistakenly reported as real audio by the detector. Manipulation attacks do not require professional knowledge and prior knowledge of detection methods, and do not require complex calculations and processing, which poses a great threat to existing detection methods. Summary of the invention
[0005] In order to solve the above problems, the present invention proposes an audio deep fake detection method, device and product based on deep neural network.
[0006] The technical solution adopted by the method of the present invention is: an audio deep fake detection method based on a deep neural network, wherein the deep neural network includes an encoder and a classifier; for the input audio to be detected, the encoder is used to extract a feature vector, and the classifier is used to classify the feature vector of the audio, so as to perform binary classification on the feature vector extracted by the encoder to determine whether the audio is real audio or audio deep fake;
[0007] The classifier includes a reshaping layer, a self-attention layer, a flattening layer, a multi-layer perceptron layer, and a classification layer that are sequentially connected; the multi-layer perceptron layer is composed of a first dense layer, a first ReLU activation function layer, a second dense layer, and a second ReLU activation function layer that are sequentially connected.
[0008] Preferably, the encoder is composed of a sequentially connected Conv1d+MaxPooling+BN+LReLU layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer It consists of a one-dimensional convolutional layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer and a GRU layer; Conv1d represents a one-dimensional convolutional layer, LReLU is a leaky linear rectification activation function, BN represents a batch normalization layer, MaxPooling represents a maximum pooling layer, FMS represents filter feature map scaling, and GRU represents a gated recurrent unit.
[0009] Preferably, the encoder is a pre-trained encoder; during the training process, the audio in the training set is subjected to two random manipulation processes to obtain two different manipulated audios, which are respectively sent to two encoders to obtain feature vectors q and feature vectors k; during the training, a past feature vector queue is maintained, and the feature vector q, feature vector k and the past feature vector queue are used to calculate the comparative learning loss, and before entering the next training iteration, the feature vector used this time is added to the past feature vector queue; the feature vector q and the audio label are used to calculate the length loss; after the two losses are calculated, the back propagation algorithm is used to optimize and update the model parameters of the encoder.
[0010] Preferably, the random manipulation process is randomly selected from seven manipulation processes, including adding Gaussian white noise, adding ambient sound, volume adjustment, fading in and out, time stretching, resampling, time domain shifting and adding echo.
[0011] Preferably, the contrastive learning loss for:
[0012]
[0013] Where N is the batch size, K is the size of the past feature queue, τ is the temperature hyperparameter, and q i and is the feature vector q and feature vector k of the i-th sample in the batch data; k j is the jth sample in the past feature vector queue.
[0014] As a preference, the length loss for:
[0015]
[0016] Where N is the batch size, q i is the feature vector q,y of the i-th sample in the batch data i is the label of the i-th sample, when y i = 1, the sample is real speech, when y i =0, the sample is a voice deepfake; w is the category weight, which is used to balance the importance of different categories to deal with unbalanced data sets, and margin is a parameter used to control the degree of distinction between the feature vectors of real audio and audio deepfakes;
[0017] The comprehensive loss function used in encoder pre-training is for:
[0018]
[0019] Where λ is the weight parameter used to control the two loss components.
[0020] Preferably, the classifier is a trained classifier; during training, a binary cross entropy loss function is used.
[0021]
[0022] Where N is the batch size, y i is the label of the i-th sample, when y i = 1, the sample is real speech, when y i = 0, the sample is a deep fake voice, p iis the probability that the classifier model predicts that the i-th sample is real speech.
[0023] The technical solution adopted by the device of the present invention is: an audio deep fake detection device based on a deep neural network, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the audio deep fake detection method based on the deep neural network is implemented.
[0024] The technical solution adopted by the product of the present invention is: an audio deep fake detection product based on a deep neural network, including a computer program, which implements the audio deep fake detection method based on a deep neural network when executed by a processor.
[0025] The present invention adopts encoder pre-training, performs random manipulation processing on the input audio, uses the feature vector obtained by the encoder output, calculates the contrastive learning loss and the length loss, and optimizes the parameters of the encoder to train the encoder to extract stable and discriminative features; compared with the prior art, the present invention has the following beneficial effects:
[0026] (1) The present invention uses contrast loss to help the encoder learn stable and discriminative features by reducing the distance between feature vectors corresponding to audio processed with different random manipulations and increasing the distance between feature vectors corresponding to different audio.
[0027] (2) The present invention uses length loss to shorten the length of the feature vector corresponding to the real audio and lengthen the length of the feature vector corresponding to the audio deep fake, so as to help the encoder aggregate the feature vector of the real speech and learn features that are easier to classify.
[0028] (3) The present invention uses a classification network enhanced by a self-attention mechanism to replace the traditional linear layer. It can capture the complex relationship between feature vectors, thereby more accurately distinguishing between real human voices and audio synthesized by deep fake technology.
[0029] (4) The random manipulation processing used in the present invention includes commonly used audio processing methods, such as adding Gaussian white noise, adding ambient sound, volume adjustment, fading in and out, time stretching, resampling, time domain shifting, adding echo, etc.; these processing methods cover the most commonly used manipulation attack methods. By using these processing methods, the model's resistance to potential manipulation attacks can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.
[0031] Figure 1 A diagram showing the structure of an encoder implemented in the present invention;
[0032] Figure 2 A structural diagram of a classifier implemented in the present invention;
[0033] Figure 3 Schematic diagram of encoder training principle implemented in the present invention. DETAILED DESCRIPTION
[0034] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.
[0035] The present embodiment provides an audio deep fake detection method based on a deep neural network, which is characterized by: the deep neural network includes an encoder and a classifier; for the input audio to be detected, the encoder is used to extract a feature vector, and the classifier is used to classify the feature vector of the audio to determine whether the audio is real audio or audio deep fake.
[0036] The encoder processes the speech input into the detection model and extracts each speech into a feature vector that is discriminative and less affected by the manipulation process, using a deep neural network;
[0037] The classifier performs binary classification on the feature vector extracted by the encoder to determine whether the input audio is real audio or audio deep fake. A simple classifier such as a single-layer neural network can be used.
[0038] Please see Figure 1In one embodiment, the encoder is composed of a sequentially connected Conv1d+MaxPooling+BN+LReLU layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, It consists of xPooling+FMS layer, Conv1d+BN+LReLU+Conv1d layer, MaxPooling+FMS layer, Conv1d+BN+LReLU+Conv1d layer, MaxPooling+FMS layer and GRU layer; Conv1d represents a one-dimensional convolutional layer, LReLU is a leaky linear rectification activation function, BN represents a batch normalization layer, MaxPooling represents a maximum pooling layer, FMS represents filter-side feature map scaling, and GRU represents a gated recurrent unit.
[0039] The encoder used in this embodiment first fixes the input audio signal tensor length to 64000 by repeated padding or cropping; the model structure of the encoder is shown in Table 1 below:
[0040] Table 1 Encoder model structure
[0041]
[0042]
[0043] In some implementations, other audio feature extraction deep neural networks may also be used to extract a 1024-dimensional feature vector from the input audio.
[0044] Please see Figure 2 In one embodiment, the classifier includes a sequentially connected reshaping layer, a self-attention layer, a flattening layer, a multi-layer perceptron layer, and a classification layer; the multi-layer perceptron layer is composed of a sequentially connected first dense layer, a first ReLU activation function layer, a second dense layer, and a second ReLU activation function layer.
[0045] Please see Figure 3 In one embodiment, the encoder is a pre-trained encoder; in the pre-training, the encoder parameters are optimized using contrastive learning loss and length loss to train the encoder to extract stable and discriminative features;
[0046] In this embodiment, the audio in the training set is randomly manipulated twice to obtain two different manipulated audios, which are respectively sent to two encoders to obtain feature vector q and feature vector k. At the same time, the model maintains a past feature vector queue, and uses feature vector q, feature vector k and past feature vector queue to calculate the contrast loss. After calculation, before entering the next training iteration, the feature vector used this time is added to the past feature vector queue. In addition, feature vector q and audio labels are used to calculate length loss. After the two losses are calculated, the back propagation algorithm is used to optimize and update the model parameters of the encoder.
[0047] The random manipulation process used in this embodiment is randomly selected from 7 manipulation processes. The 7 manipulation processes are: adding Gaussian white noise, adding ambient sound, volume adjustment, fading in and out, time stretching, resampling, time domain shifting and adding echo. The addition of Gaussian white noise is controlled by the signal-to-noise ratio, and the random range of the signal-to-noise ratio is 15 to 25 decibels. The ambient noise has a fixed signal-to-noise ratio of 20 decibels, and noise from the ESC-50 data set is randomly added. The ambient noise includes seven categories: wind, footsteps, breathing, coughing, rain, clocks, and sneezing. The volume adjustment randomly adjusts the amplitude of the audio waveform as a whole to 0.1 to 0.5 times the original. For fading in and out, the random range of the gradient ratio is 0.1 to 0.5, and the gradient shape randomly selects one of linear, exponential, quarter-cycle sine waveform, half-cycle sine waveform, and logarithmic. For time stretching, the FFT length of 128 is fixed, and the time stretch factor with a random range of 0.9 to 1.1 is used. For resampling, the target resampling rates were randomly selected from 15000, 15500, 16500, and 17000. For time domain translation, the translation length was randomly selected from 1600 to 32000. For adding echo, the number of delayed sampling points of the echo compared to the original signal was randomly selected from 1000 to 3000, and the attenuation rate of the echo was randomly selected from 0.1 to 0.5.
[0048] The contrastive learning loss used in this embodiment uses cosine similarity to evaluate the distance between feature vector q and feature vector k, as well as the distance between feature vector q and feature vectors of other audio in the past feature vector queue. In order to enable the model to learn robust features under manipulation attacks, feature vector q should be as far away from feature vector k as possible, and as far away from feature vectors in the past feature vector queue as possible. The contrastive loss can be calculated by the following formula:
[0049]
[0050] Where N is the batch size, K is the size of the past feature queue, τ is the temperature hyperparameter, and q i and is the feature vector q and feature vector k of the i-th sample in a batch of data. k j is the jth sample in the past feature vector queue.
[0051] The length loss used in this embodiment is calculated by the feature vector q and the audio label. If the audio corresponding label is real audio, the feature vector length should be as small as possible. If the audio corresponding label is audio deep fake, the feature vector length should be as large as possible. The length loss can be calculated by the following formula:
[0052]
[0053] Where N is the batch size, q i is the feature vector q, y of the i-th sample in a batch of data i is the label of the i-th sample, when y i = 1, the sample is real speech, when y i =0, the sample is a voice deep fake, w is the category weight, which is used to balance the importance of different categories to deal with unbalanced data sets, and margin is a parameter used to control the degree of distinction between the feature vectors of real audio and audio deep fakes.
[0054] The comprehensive loss function used in encoder pre-training in this example can be calculated as follows:
[0055]
[0056] Where λ is the weight parameter used to control the two loss components.
[0057] This example uses the Adam optimizer to update the model parameters. The optimizer uses an initial learning rate of 0.0005 and uses a cosine learning rate decay strategy for decay. The weight decay is set to 0.0001. The model is trained for 150 iterations and a batch size of 24. The size of the feature queue is 6144. The temperature hyperparameter τ is set to 0.07, margin is set to 4, w is set to 9.0, and the weight λ of the length loss is set to 2.0.
[0058] In one embodiment, the classifier is a trained classifier; during training, a binary cross entropy loss function is used, and the loss function can be calculated by the following formula:
[0059]
[0060] Where N is the batch size, y i is the label of the i-th sample, when y i = 1, the sample is real speech, when y i = 0, the sample is a deep fake voice, pi is the probability that the model predicts that the i-th sample is real speech.
[0061] This example uses the Adam optimizer to update the model parameters, uses a learning rate of 0.001, sets the weight decay to 0.0001, trains the model for 10 iterations, and uses a batch size of 16.
[0062] This embodiment also provides an audio deep fake detection device based on a deep neural network, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the audio deep fake detection method based on a deep neural network when executing the program.
[0063] This embodiment also provides an audio deep fake detection product based on a deep neural network, including a computer program, which, when executed by a processor, implements the audio deep fake detection method based on a deep neural network.
[0064] The present invention is further described below through specific experiments.
[0065] The experiment uses the popular audio deep fake detection dataset ASVspoof 2019LA as the experimental data, and uses the default training and evaluation dataset divisions of the dataset. The experiment uses the model classification threshold when the equal error rate is obtained as the classification threshold used for evaluation, and evaluates the false acceptance rate (False Acceptance Rate, FAR) of the audio deep fake detection scheme under manipulation attacks. A higher false acceptance rate indicates that the detection method is easily deceived by manipulation attacks and has poor robustness. With the same encoder architecture, a deep neural network model using a single linear layer as a downstream classifier is used as a baseline scheme for comparison with this embodiment.
[0066] The volume manipulation attack is evaluated, and the attacker adjusts the volume to 50% and 10% of the original volume. The experimental results of the baseline scheme and this embodiment are shown in Table 2.
[0067] Table 2 FAR of different schemes under volume adjustment manipulation attack
[0068] Volume adjustment parameters 50％ 10％ Baseline scenario 0.0624 0.3442 This embodiment 0.0536 0.1097
[0069] The manipulation attack of adding Gaussian white noise is evaluated, and the attacker controls the intensity of the added white noise at a signal-to-noise ratio of 15 dB, 20 dB, and 25 dB. The experimental results of the baseline scheme and this embodiment are shown in Table 3.
[0070] Table 3 FAR of different schemes under manipulation attack with Gaussian white noise added
[0071] Signal-to-Noise Ratio 15 dB 20 dB 25 dB Baseline scenario 0.1766 0.1095 0.0812 This embodiment 0.0760 0.0662 0.0571
[0072] To evaluate the fade-in and fade-out manipulation attack, the attacker selects a fade ratio of 0.5 and uses linear, exponential, and half-cycle sine waveform fade shapes to apply fade-in and fade-out transformations to the audio. The experimental results of the baseline scheme and this embodiment are shown in Table 4.
[0073] Table 4 FAR of different schemes under gradual in and out manipulation attack
[0074] Gradient shape Linear Exponential One-half cycle sine wave Baseline scenario 0.1542 0.2877 0.3228 This embodiment 0.0734 0.0947 0.1130
[0075] It can be seen from the experimental results that this embodiment achieves a lower FAR under all manipulation attack settings, which means that the model is more robust to manipulation attacks.
[0076] In view of the problem that the features extracted by the existing audio deep fake detection model are easily affected by manipulation attacks, the present invention proposes an audio deep fake monitoring method, device and product that uses contrastive learning to improve the robustness of the model's extracted features, so as to enhance the security of the detection system. The method uses contrastive learning to extract intrinsic audio features that remain robust under manipulation attacks, and at the same time uses length loss to aggregate the feature vectors of real audio to obtain an encoder that can extract discriminative features, thereby enhancing the effect of the detection method, and has the advantages of good accuracy and robustness under manipulation attacks.
[0077] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.
[0078] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.
Claims
1. An audio deep fake detection method based on deep neural network, characterized by: The deep neural network includes an encoder and a classifier; for the input audio to be detected, the encoder is used to extract a feature vector, and the classifier is used to classify the feature vector of the audio, so as to perform binary classification on the feature vector extracted by the encoder to determine whether the audio is real audio or audio deep fake; The classifier includes a reshaping layer, a self-attention layer, a flattening layer, a multi-layer perceptron layer, and a classification layer that are sequentially connected; the multi-layer perceptron layer is composed of a first dense layer, a first ReLU activation function layer, a second dense layer, and a second ReLU activation function layer that are sequentially connected.
2. The method for detecting deep audio forgery based on a deep neural network according to claim 1, characterized in that: The encoder is composed of a sequentially connected Conv1d+MaxPooling+BN+LReLU layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, a Conv1d+BN+LReLU+Conv1d layer, a MaxPooling+FMS layer, and a GRU layer; Among them, Conv1d represents a one-dimensional convolutional layer, LReLU is a leaky linear rectifier activation function, BN represents a batch normalization layer, MaxPooling represents a maximum pooling layer, FMS represents filter-side feature map scaling, and GRU represents a gated recurrent unit.
3. The method for detecting deep audio forgery based on a deep neural network according to claim 1 or 2, characterized in that: The encoder is a trained encoder; During the pre-training process, the audio in the training set is randomly manipulated twice to obtain two different manipulated audios, which are sent to two encoders respectively to obtain feature vectors q and k. During the training, a past feature vector queue is maintained, and the feature vector q, feature vector k and the past feature vector queue are used to calculate the comparative learning loss. Before entering the next training iteration, the feature vector used this time is added to the past feature vector queue. The feature vector q and the audio label are used to calculate the length loss. After the two losses are calculated, the back propagation algorithm is used to optimize and update the model parameters of the encoder.
4. The method for detecting deep audio forgery based on a deep neural network according to claim 3, characterized in that: The random manipulation process is randomly selected from seven manipulation processes, including adding Gaussian white noise, adding ambient sound, volume adjustment, fading in and out, time stretching, resampling, time domain shifting and adding echo.
5. The method for detecting deep audio forgery based on a deep neural network according to claim 3, characterized in that: The contrastive learning loss for: Where N is the batch size, K is the size of the past feature queue, τ is the temperature hyperparameter, and q i and is the feature vector q and feature vector k of the i-th sample in the batch data; k j is the jth sample in the past feature vector queue.
6. The method for detecting deep audio forgery based on a deep neural network according to claim 3, characterized in that: The length loss for: Where N is the batch size, q i is the feature vector q,y of the i-th sample in the batch data i is the label of the i-th sample, when y i = 1, the sample is real speech, when y i =0, the sample is a voice deep fake; w is the category weight, which is used to balance the importance of different categories to deal with unbalanced data sets, and margin is a parameter used to control the degree of distinction between the feature vectors of real audio and audio deep fakes; The comprehensive loss function used in encoder pre-training is for: Where λ is the weight parameter used to control the two loss components.
7. The method for detecting deep audio forgery based on a deep neural network according to claim 1 or 2, characterized in that: The classifier is a trained classifier; During training, the binary cross entropy loss function is used Where N is the batch size, y i is the label of the i-th sample, when y i = 1, the sample is real speech, when y i = 0, the sample is a deep fake voice, p i is the probability that the classifier model predicts that the i-th sample is real speech.
8. An audio deep fake detection device based on a deep neural network, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the audio deep fake detection method based on deep neural network as described in any one of claims 1 to 7 is implemented.
9. An audio deepfake detection product based on a deep neural network, comprising a computer program, characterized in that: When the computer program is executed by a processor, the audio deep fake detection method based on deep neural network as described in any one of claims 1 to 7 is implemented.