Deep forged audio detection method based on attention and double features

Abstract

The invention discloses a deep forged audio detection method based on attention and double features, which comprises a training stage and a testing stage, and is characterized in that training data is acquired and preprocessed in the training stage, and a double-feature deep forged audio detection model is constructed; inputting the manual acoustic features and the original waveform features of the training data into respective detection branches for model training; attention blocks are added to feature extraction modules of the two models respectively, each attention block is composed of spatial local attention and spectrum global attention, the spatial local attention is used for focusing a local sensitive area of the features, the spectrum global attention is used for capturing cross-channel correlation between the features, namely, the long-range dependency relationship of the captured features, and the spatial local attention is used for focusing the local sensitive area of the features. Splicing, fusing and outputting the images, and inputting the images into a classification module for true and false classification; and finally, the detection model is trained in the test stage, authenticity detection of real voice and synthetic voice is completed, and high-quality deep forged audio detection is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of deep fake audio detection, and in particular to a deep fake audio detection method based on attention and dual features. Background Art

[0002] As the performance of speech synthesis systems continues to improve, the quality of synthesized speech is increasingly consistent with the auditory characteristics of the human ear, which has caused great trouble for people's safety in daily life. People have found that by using synthesized speech or replaying speech, it is easy to deceive the ASV (Automatic Speaker Verification) system and make it make wrong judgments. Therefore, deep fake audio detection has become a hot issue in ASV.

[0003] In order to build an efficient deep fake audio detection system, existing methods focus on two aspects: feature engineering and model design. In terms of feature engineering, 1. Extracting CQCC (Constant Q Cepstral Coefficients) features of speech is more effective in capturing signs of manipulation artifacts that indicate deception attacks; 2. Using fractional fusion of front-end features to detect synthetic speech, by focusing on three artifacts related to amplitude, phase and pitch changes to improve the forgery information contained in the features, the best performance was achieved in the model based on GMM (Gaussian Mixture Model) classifier; 3. Extracting information from the octave power spectrum of CQT (Constant Q Transform), using the MLT (Multilevel Transform) framework to capture the relevant information of the octave power spectrum from CQT in a multi-level manner. In terms of model design, 1. SENet (Squeeze-Excitation Network) and ResNet (Residual Network) are introduced into deep neural networks to achieve high-quality playback speech detection; 2. By designing a one-class Softmax (OC-Softmax), the real speech has a compact boundary while keeping a certain distance from the fake speech; 3. The GAT (Graph Attention Networks) structure is used to learn the relationship between clues in different sub-bands and time intervals, and the model-level graph fusion of spectral and temporal subgraphs and the graph pooling strategy are used to improve the distinction between fake and real speech; 4. The transfer learning scheme based on the wav2vec 2.0 pre-trained model with VIB (Variational Information Bottleneck) improves the model's performance in distinguishing unseen deceptions from real speech, and achieves good results in low-resource and cross-dataset tests.

[0004] The various deep fake audio detection methods mentioned above have evolved from detection in closed set situations to detection in open set situations, and can detect synthesized speech of any synthesis algorithm. Most of the methods are based on CNN (Convolutional Neural Network), but CNN lacks the ability to extract global features. For example, CNN cannot establish a connection between the beginning and end of the speech, so it will cause the problem of lack of "long-range dependence", resulting in insufficient model feature extraction. At the same time, most scholars only use one feature (manual acoustic feature or original waveform feature) to build a detection model, which will cause the detection performance of the model to decrease when the feature itself has defects. For example, the manual acoustic feature is not able to detect the synthesized speech after passing through the filter, and the original waveform feature is not able to detect the speech synthesized using only manual acoustic features. Summary of the invention

[0005] The problem to be solved by the present invention is: to provide a deep fake audio detection method based on attention and dual features. In view of the problem that traditional neural networks lack "long-range dependence", spatial local attention is used to extract and retain the local information of the features, and spectral global attention is used to extract the global information of the features, and the global information is spliced ​​to obtain features that contain both local and global information. In view of the problem that a single feature has detection defects, a dual detection branch is constructed, and manual acoustic features and original waveform features are used to detect deep fake audio at the same time, which enables the fake information in the features to be fully extracted and achieves high-quality deep fake audio detection in the open set case.

[0006] The present invention adopts the following technical solution: a deep fake audio detection method based on attention and dual features, including a training phase and a testing phase. The detection network and parameters required for deep fake audio detection are obtained in the training phase, and the detection of deep fake audio is realized in the testing phase.

[0007] The training phase includes the following steps:

[0008] Step 1: Obtain corpus as training data, where the training data consists of real speech and synthetic speech;

[0009] Step 2: preprocess the acquired training data to obtain original waveform features and manual acoustic features respectively;

[0010] Step 3: Build a detection model. Use the convolutional neural network structure to build a dual-feature deep fake audio detection model. Input the original waveform features and manual acoustic features into their respective detection branches for model training, obtain their respective classification prediction labels, and perform weighted average on the classification labels to obtain the final classification results.

[0011] Step 4: During the detection model training process, the hyperparameters of the detection model are set to minimize the objective function until the set number of training rounds is reached to obtain a trained detection model;

[0012] The testing phase includes the following steps:

[0013] Step 5: Obtain the corpus to be tested as test data, the test data includes real speech and synthesized speech, and extract the original waveform features and manual acoustic features of the test data respectively;

[0014] Step 6: Input the extracted original waveform features and manual acoustic features into the trained detection model for deep fake audio detection.

[0015] Furthermore, in step 2, the training corpus is preprocessed, and the specific steps are as follows: the training data is sampled at a sampling rate of 16kHz to obtain the original waveform features, which are expressed as X raw , and then further processed it with a frame length of 20ms and a frame overlap of 10ms, and then the number of frames is fixed to 750 to obtain the manual acoustic features, denoted as X lfcc .

[0016] Furthermore, in step 3, the detection branch of the original waveform feature includes: a Sinc module, a feature extraction module and a classification module, and the detection branch of the manual acoustic feature includes: a feature extraction module and a classification module.

[0017] For the original waveform features, first input them into the Sinc module for waveform feature processing to obtain the time-frequency features X rawTF , and then input it and the manual waveform features into their respective feature extraction modules to obtain their respective deep feature vectors X′ raw and X′ lfcc ; Then input them into their respective classification modules to obtain the classification prediction label y raw and lfcc , and finally the classification labels are weighted averaged to obtain the final classification result y.

[0018] Preferably, the feature extraction module of the manual acoustic features includes: 2 two-dimensional convolutional layers, 2 batch normalization layers, 2 ReLU activation functions, 4 residual blocks of manual acoustic features and 1 attention block; the feature extraction module of the original waveform features includes: 6 residual blocks of the original waveform features, 6 attention blocks, 1 batch normalization layer and 1 SELU activation function.

[0019] Among them, the residual block of the manual acoustic features consists of 5 2D convolutional layers, 4 batch normalization layers and 4 ReLU activation functions; the residual block of the original waveform features consists of 2 batch normalization layers, 2 SELU activation functions, 3 2D convolutional layers and 1 maximum pooling layer.

[0020] The attention block consists of 1 2D convolution layer, 1 batch normalization layer, 1 ReLU activation function, 1 spectral global attention and 1 spatial local attention. Its spectral global attention consists of 1 adaptive average pooling layer, 1 2D DCT layer, 2 linear layers, 1 ReLU activation function and 1 Sigmoid activation function, and its spatial local attention consists of 2 average pooling layers, 2 1D convolution layers, 2 group normalization layers and 2 Sigmoid activation functions.

[0021] Furthermore, the two-dimensional DCT layer in the spectral global attention first transforms the input feature X in Split into multiple parts along the channel dimension, and record the split parts as in n represents the number of channel splits, and C needs to be divisible by n. Indicates the size of each feature after splitting.

[0022] For each part, its corresponding two-dimensional DCT frequency component is assigned to be multiplied and summed, and the formula of its two-dimensional DCT layer can be expressed as:

[0023]

[0024] Among them, H represents the input feature The height of W represents the input feature The width of [u i ,v i ] represents the input feature The corresponding two-dimensional index of the frequency component, B represents the two-dimensional frequency component, Freq i Represents input features The compressed frequency factor is Indicates the size of each frequency factor.

[0025] Next, concatenate each frequency factor to get a frequency vector, the formula is:

[0026] Freq=cat(Freq 0 ,Freq 1 ,…,Freq n-1 )

[0027] Among them, Freq represents the frequency vector, and its size is represents the size of the entire frequency vector; cat(·) represents concatenation by channel dimension, and finally it is input into the linear layer and activation function in turn, and combined with the initial input feature X in Perform residual connection to get output feature X out , the formula is:

[0028]

[0029] Where fc(·) represents the linear layer, δ(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. Indicates channel addition operation, X out Represents the output of spectral global attention.

[0030] Furthermore, the spatial local attention will input feature X in Average pooling, one-dimensional convolution, group normalization and Sigmoid activation function operations are performed in the horizontal and vertical directions respectively to obtain the horizontal factor z h and the vertical factor z w , the formula is:

[0031] z h =σ(GN(Conv1d(AvgPool h (X in ))))

[0032] z w =σ(GN(Conv1d(AvgPool w (X in ))))

[0033] Among them, σ(·) represents the Sigmoid activation function, GN(·) represents group normalization, Conv1d(·) represents a one-dimensional convolutional layer, and AvgPool(·) represents an average pooling layer.

[0034] Finally, the obtained horizontal factor and vertical factor are multiplied with the input feature and residually connected with it to obtain the final output feature X′ out , the formula is:

[0035]

[0036] in, represents the channel addition operation, X′ out Represents the output of spatial local attention.

[0037] Furthermore, in step 3, the classification module of the original waveform features includes 1 GRU layer and 2 linear layers, and the deep feature vector X′raw Enter the classification label y raw The classification module of the handcrafted acoustic features includes 1 attention pooling layer and 2 linear layers, which transforms the deep feature vector X′ lfcc Enter the classification label y lfcc , and finally the classification labels are weighted averaged to obtain the final classification result y.

[0038] Furthermore, in step 4, the objective function of the detection network is expressed as:

[0039]

[0040] in, represents the model parameters corresponding to the original waveform feature detection branch, Indicates the parameters corresponding to the Sinc module, Represents the parameters corresponding to the feature extraction module, Indicates the parameters corresponding to the classification module; represents the parameters corresponding to the manual acoustic feature detection branch, Represents the parameters corresponding to the feature extraction module, Indicates the parameters corresponding to the classification module.

[0041] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0042] 1. The deep fake audio detection method of the present invention proposes an attention block composed of spectral global attention and spatial local attention in the feature extraction module: spectral global attention is used to extract cross-channel information of feature vectors, that is, global information, by grouping the input features with equal lengths, and using different two-dimensional frequency components to calculate their frequency factors, and then splicing them into a frequency vector, which can be multiplied with the input features to obtain information between different channels; spatial local attention is used to extract spatial area information of feature vectors, that is, local information, by pooling the input features in the horizontal and vertical directions, so that subsequent operations can be performed separately in these two dimensions, and the sensitive area information of the features can be obtained by multiplying the horizontal factor and the vertical factor with the input features; finally, the global information and the local information are spliced ​​to obtain a feature vector containing both global information and local information. At the same time, the parameters of the attention block proposed by the present invention are very few, which can effectively improve the performance of the detection model while ensuring the amount of model parameters.

[0043] 2. The deep fake audio detection method of the present invention also proposes to use dual features as the input of the model, namely the original waveform feature and the manual acoustic feature. Experiments have found that the original waveform feature has serious deficiencies in the detection ability of speech synthesized using only manual acoustic features, but has better detection ability for speech synthesized using end-to-end algorithms or after filtering; while the manual acoustic feature has serious deficiencies in the detection ability of synthesized speech after filtering and smoothing, but has better detection ability for speech synthesized using only manual acoustic features. Therefore, these two features are input into the detection model together to make up for each other's own deficiencies, thereby improving the detection ability of the model.

[0044] 3. The deep fake audio detection method of the present invention uses a large amount of real speech and synthetic speech of multiple algorithms in the training stage. Therefore, in the testing stage, the authenticity of the speech synthesized by the synthetic algorithm not used in training can be tested, which greatly expands the scope of application of the model and improves the performance of deep fake audio detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of a deep fake audio detection method of the present invention;

[0046] Figure 2 It is a network structure diagram of the residual block in the feature extraction module of the manual acoustic feature branch of the detection model of the present invention;

[0047] Figure 3 It is a network structure diagram of a residual block in a feature extraction module of an original waveform feature branch of a detection model of the present invention;

[0048] Figure 4 It is a network structure diagram of the attention block in the feature extraction module of the detection model of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely explain the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. At the same time, for the step numbers in the embodiments of the present invention, they are only set for the convenience of explanation, and the order between the steps is not limited. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] In one embodiment of the present invention, a deep fake audio detection method based on attention and dual features includes a training phase and a testing phase. The training phase is used to obtain a detection network and its parameters required for deep fake audio detection, while the testing phase is used to implement deep fake audio detection.

[0051] like Figure 1 As shown, the training phase includes the following steps:

[0052] Step 1: Get training data.

[0053] In this embodiment, the training data is the logical access track in the ASVspoof 2019 competition, that is, it is composed of real speech and speech synthesized by various speech synthesis algorithms; all the training corpus and verification corpus in the acquired corpus are taken out, with a total of 25,380 training corpus and 24,986 verification corpus, and each sentence lasts 2-8 seconds.

[0054] The selected training corpus is used as the input for model training. After each round of training, the verification corpus is input into the model for verification. This can make the training results of each round of the model more intuitive, thereby facilitating the adjustment of the model's training parameters.

[0055] Step 2: preprocess the acquired training data.

[0056] In this embodiment, the training data is sampled at a sampling rate of 16 kHz to obtain the original waveform feature X raw , and then further processed it with a frame length of 20ms and a frame overlap of 10ms, and then fixed the number of frames to 750 to obtain the manual acoustic feature X lfcc .

[0057] Step 3: The original waveform feature X obtained in step 2 raw and the handcrafted acoustic features X lfcc Input into the respective detection branch for training.

[0058] In this example, the network adopts the structure of convolutional neural network. First, the original waveform features are input into the Sinc module for waveform feature processing to obtain the time-frequency features X rawTF , and then input it and the manual waveform features into their respective feature extraction modules to obtain their respective deep feature vectors X′ raw and X′ lfcc .

[0059] Then input them into their respective classification modules to obtain the classification prediction label y raw and lfcc , and finally the classification labels are weighted averaged to obtain the final classification result y.

[0060] In this embodiment, the manual acoustic feature detection branch of the detection model mainly includes two parts: a feature extraction module and a classification module; the original waveform feature detection branch mainly includes three parts: a Sinc module, a feature extraction module and a classification module:

[0061] (1) In the Sinc module of the original waveform feature, the original waveform feature is processed by a one-dimensional convolution layer and a maximum pooling layer to obtain the time-frequency feature X rawTF .

[0062] (2) In the feature extraction module of the original waveform feature, the time-frequency feature X rawTF After passing through 6 groups of networks consisting of original waveform residual blocks and attention blocks, the deep feature vector X′ is obtained. raw ; In the feature extraction module of manual acoustic features, the feature vector X lfcc After four sets of manual acoustic feature residual blocks and one attention block, the deep feature vector X′ is obtained. lfcc ;

[0063] The feature extraction module of manual acoustic features includes: 2 two-dimensional convolutional layers, 2 batch normalization layers, 2 ReLU activation functions, 4 residual blocks of manual acoustic features and 1 attention block; the feature extraction module of raw waveform features includes: 6 residual blocks of raw waveform features, 6 attention blocks, 1 batch normalization layer and 1 SELU activation function.

[0064] Among them, the residual block of manual acoustic features, such as Figure 2 As shown in , it consists of 5 two-dimensional convolutional layers, 4 batch normalization layers and 4 ReLU activation functions; the residual block of the original waveform feature is as follows Figure 3 As shown in the figure, it consists of 2 batch normalization layers, 2 SELU activation functions, 3 2D convolutional layers and 1 maximum pooling layer.

[0065] Attention blocks, such as Figure 4 As shown, it consists of 1 2D convolution layer, 1 batch normalization layer, 1 ReLU activation function, 1 spectral global attention and 1 spatial local attention.

[0066] Preferably, the spectral global attention consists of 1 adaptive average pooling layer, 1 2D DCT layer, 2 linear layers, 1 ReLU activation function and 1 Sigmoid activation function, and its spatial local attention consists of 2 average pooling layers, 2 1D convolution layers, 2 group normalization layers and 2 Sigmoid activation functions.

[0067] Furthermore, the two-dimensional DCT layer in the spectral global attention first transforms the input feature X in Split into multiple parts along the channel dimension, and record the split parts as in n represents the number of channel splits, and C needs to be divisible by n. Indicates the size of each feature after splitting.

[0068] For each part, its corresponding two-dimensional DCT frequency component is assigned to be multiplied and summed, and the formula of its two-dimensional DCT layer can be expressed as:

[0069]

[0070] Among them, H represents the input feature The height of W represents the input feature The width of [u i ,v i ] represents the input feature The corresponding two-dimensional index of the frequency component, B represents the two-dimensional frequency component, Freq i Represents input features The compressed frequency factor is Indicates the size of each frequency factor.

[0071] Then, each frequency factor is concatenated to obtain a frequency vector, the formula of which is:

[0072] Freq=cat(Freq 0 ,Freq 1 ,…,Freq n-1 )

[0073] Among them, Freq represents the frequency vector, and its size is represents the size of the entire frequency vector; cat(·) represents concatenation by channel dimension.

[0074] Finally, it is input into the linear layer and activation function in turn, and combined with the initial input feature X in Perform residual connection to get output feature X out , the formula is:

[0075]

[0076] Where fc(·) represents the linear layer, δ(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. Indicates channel addition operation, X out Represents the spectral global attention output.

[0077] Furthermore, the spatial local attention will input feature X inAverage pooling, one-dimensional convolution, group normalization and Sigmoid activation function operations are performed in the horizontal and vertical directions respectively to obtain the horizontal factor z h and the vertical factor z w , the formula is:

[0078] z h =σ(GN(Conv1d(AvgPool h (X in ))))

[0079] z w =σ(GN(Conv1d(AvgPool w (X in ))))

[0080] Among them, σ(·) represents the Sigmoid activation function, GN(·) represents group normalization, Conv1d(·) represents a one-dimensional convolutional layer, and AvgPool(·) represents an average pooling layer.

[0081] Finally, the obtained horizontal factor and vertical factor are multiplied with the input feature and residually connected with it to obtain the final output feature X′ out , the formula is:

[0082]

[0083] in, represents the channel addition operation, X′ out Represents the final output of this attention.

[0084] (3) The classification module of the original waveform features includes 1 GRU layer and 2 linear layers. The deep feature vector X′ raw Enter the classification label y raw ; The classification module of the hand-crafted acoustic features includes 1 attention pooling layer and 2 linear layers, which transforms the deep feature vector X′ lfcc Enter the classification label y lfcc , and finally the classification labels are weighted averaged to obtain the final classification result y.

[0085] The objective function of the entire detection network is expressed as:

[0086]

[0087] in, represents the model parameters corresponding to the original waveform feature detection branch, Indicates the parameters corresponding to the Sinc module, Represents the parameters corresponding to the feature extraction module, Indicates the parameters corresponding to the classification module; represents the parameters corresponding to the manual acoustic feature detection branch, Represents the parameters corresponding to the feature extraction module, Indicates the parameters corresponding to the classification module.

[0088] Step 3-1: First input the original waveform features into the Sinc module to obtain the time-frequency features X rawTF , and then combine it with the manual acoustic feature X lfcc Input into their respective feature extraction modules, and obtain their respective deep feature vectors X′ through the feature extraction network raw and X′ lfcc .

[0089] Step 3-2: The deep feature vector X′ raw and X′ lfcc They are input into their respective classification modules for true and false classification. The classification label output by the classification module of the original waveform feature is y raw , the classification label output by the classification module of the manual acoustic features is y lfcc ; Classification label y raw and lfcc Perform weighted averaging to obtain the classification label y that is finally output by the detection model.

[0090] Step 4: Repeat steps 3-1 to 3-2 until the set number of iterations is reached, thereby obtaining a trained network. Due to different specific settings of the neural network and different performance of the experimental equipment, the number of iterations set is also different.

[0091] In this embodiment, the number of iteration rounds is particularly set to 120 rounds.

[0092] The testing phase includes the following steps:

[0093] Step 5: The test data includes real speech and synthesized speech. The original waveform features and manual acoustic features of the test data are extracted respectively, which are expressed as Y raw and Y lfcc ;

[0094] Step 6: Extract the original waveform feature Y raw and the handcrafted acoustic features Y lfcc Input them together into the trained detection model for deep fake audio detection.

[0095] In summary, the present invention adopts a deep fake audio detection method based on attention and dual features. Dual features, namely, original waveform features and manual acoustic features, are input into the model together, so that different features can make up for each other's defects and deficiencies; spectral global attention and spatial local attention are added to the feature extraction module. The spectral global attention is used to extract the cross-channel information of the feature vector, namely, the global information, and the spatial local attention is used to extract the spatial area information of the feature vector, namely, the local information, and splice them to obtain features containing both global and local information; through the above operations, the detection model can achieve high-quality deep fake audio detection in the open set case.

[0096] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A deep fake audio detection method based on attention and dual features, characterized in that: The total phase includes a training phase and a testing phase; The training phase includes the following steps: Step 1: Obtain corpus as training data, where the training data consists of real speech and synthetic speech; Step 2: preprocess the acquired training data to obtain original waveform features and manual acoustic features respectively; Step 3: Build a detection model. Use the convolutional neural network structure to build a dual-feature deep fake audio detection model. Input the original waveform features and manual acoustic features into their respective detection branches for model training, obtain their respective classification prediction labels, and perform weighted average on the classification labels to obtain the final classification results. Step 4: During the detection model training process, the hyperparameters of the detection model are set to minimize the objective function until the set number of training rounds is reached to obtain a trained detection model; The testing phase includes the following steps: Step 5: Obtain the corpus to be tested as test data, the test data includes real speech and synthesized speech, and extract the original waveform features and manual acoustic features of the test data respectively; Step 6: Input the extracted original waveform features and manual acoustic features into the trained detection model for deep fake audio detection.

2. The deep fake audio detection method based on attention and dual features according to claim 1 is characterized in that: In step 2, the training data is preprocessed. The specific steps are as follows: the training data is sampled at a sampling rate of 16kHz to obtain the original waveform features, which are expressed as X raw , X raw Further processing is performed with a frame length of 20ms and a frame overlap of 10ms, and the number of frames is fixed to 750 to obtain the manual acoustic features, denoted as X lfcc .

3. The deep fake audio detection method based on attention and dual features according to claim 2 is characterized in that: In step 3, the detection branch of the original waveform feature includes: a Sinc module, a feature extraction module and a classification module; the detection branch of the manual acoustic feature includes: a feature extraction module and a classification module; The dual-feature deep fake audio detection model is constructed using the convolutional neural network structure, which includes the following sub-steps: Step 3.1: Convert the original waveform feature X raw Input to the Sinc module, process the waveform features, and obtain the time-frequency feature X rawTF ; Step 3.2: In the feature extraction module of the original waveform feature, the time-frequency feature X rawTF The deep feature vector X is obtained by sequentially passing through the network composed of 6 residual blocks and attention blocks of the original waveform features. r ' aw ; In the feature extraction module of manual acoustic features, the manual acoustic feature X lfcc After passing through 4 residual blocks of manual acoustic features and 1 attention block, the deep feature vector X is obtained. l ' fcc ; Step 3.3: Transform the deep feature vector X r ' aw and X l ' fcc They are input into their respective classification modules for true and false classification. The classification label output by the classification module of the original waveform feature is y raw , the classification label output by the classification module of the manual acoustic features is y lfcc ; Step 3.4: Set the classification label y raw and lfcc Perform weighted averaging to obtain the classification label y that is finally output by the detection model.

4. The deep fake audio detection method based on attention and dual features according to claim 3 is characterized in that: The feature extraction module of the manual acoustic feature includes: 2 two-dimensional convolutional layers, 2 batch normalization layers, 2 ReLU activation functions, 4 residual blocks of manual acoustic features and 1 attention block; the residual block of the manual acoustic feature is composed of 5 two-dimensional convolutional layers, 4 batch normalization layers and 4 ReLU activation functions.

5. The deep fake audio detection method based on attention and dual features according to claim 4 is characterized in that: The feature extraction module of the original waveform features includes: 6 residual blocks of the original waveform features, 6 attention blocks, 1 batch normalization layer and 1 SELU activation function; the residual block of the original waveform features consists of 2 batch normalization layers, 2 SELU activation functions, 3 two-dimensional convolution layers and 1 maximum pooling layer.

6. The deep fake audio detection method based on attention and dual features according to claim 5 is characterized in that: The attention block consists of 1 2D convolution layer, 1 batch normalization layer, 1 ReLU activation function, 1 spectral global attention and 1 spatial local attention; The spectral global attention consists of 1 adaptive average pooling layer, 1 2D DCT layer, 2 linear layers, 1 ReLU activation function and 1 Sigmoid activation function; The spatial local attention consists of 2 average pooling layers, 2 one-dimensional convolutional layers, 2 group normalization layers and 2 Sigmoid activation functions.

7. The deep fake audio detection method based on attention and dual features according to claim 6 is characterized in that: The spectral global attention is used to extract cross-channel information of feature vectors as follows: The input feature X in Split into multiple parts along the channel dimension, each part after splitting is recorded as n represents the number of channel splits; for each part, the corresponding two-dimensional DCT frequency component is assigned to multiply it and sum it. The formula of the two-dimensional DCT layer is as follows: Among them, H represents the input feature The height of W represents the input feature The width of [u i ,v i ] represents the input feature The corresponding two-dimensional index of the frequency component, B represents the two-dimensional frequency component, Freq i Represents input features Frequency factor after compression; Concatenate each frequency factor to get a frequency vector. The formula is as follows: Freq=cat(Freq 0 ,Freq 1 ,…,Freq n-1 ) Among them, Freq represents the frequency vector, cat(·) represents concatenation by channel dimension; The frequency vector is input into the linear layer and activation function in turn, and combined with the initial input feature X in Perform residual connection to obtain output feature X out , the formula is as follows: Where fc(·) represents the linear layer, δ(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. Indicates channel addition operation, X out Represents the output of spectral global attention.

8. The deep fake audio detection method based on attention and dual features according to claim 6 is characterized in that: The spatial local attention is used to extract the spatial region information of the feature vector in the following way: The input feature X in Average pooling, one-dimensional convolution, group normalization and Sigmoid activation function operations are performed in the horizontal and vertical directions respectively to obtain the horizontal factor z h and the vertical factor z w , the formula is as follows: z h =σ(GN(Conv1d(AvgPool h (X in )))) z w =σ(GN(Conv1d(AvgPool w (X in )))) Among them, GN(·) represents group normalization, Conv1d(·) represents a one-dimensional convolutional layer, and AvgPool(·) represents an average pooling layer; The obtained horizontal factor and vertical factor are multiplied with the input feature, and residual connection is performed to obtain the final output feature X o ' ut , the formula is as follows: in, represents the channel addition operation, X′ out Represents the final output of spatial local attention.

9. The deep fake audio detection method based on attention and dual features according to claim 3 is characterized in that: In step 3.3, the classification module of the original waveform features includes: 1 GRU layer and 2 linear layers, and the input deep feature vector X r ' aw , get the classification label y raw ; The classification module of the manual acoustic features includes: 1 attention pooling layer and 2 linear layers, input deep feature vector X l ' fcc , get the classification label y lfcc , the classification label y raw and lfcc Perform weighted averaging to obtain the final classification result y.

10. The deep fake audio detection method based on attention and dual features according to claim 1 is characterized in that: The objective function of the detection model in step 4 is expressed as follows: in, represents the model parameters corresponding to the original waveform feature detection branch, Indicates the parameters corresponding to the Sinc module, Represents the parameters corresponding to the feature extraction module, Indicates the parameters corresponding to the classification module; represents the parameters corresponding to the manual acoustic feature detection branch, Represents the parameters corresponding to the feature extraction module, Indicates the parameters corresponding to the classification module.

Images