Low-quality compressed voice deep counterfeiting detection method based on frequency domain and time domain knowledge extraction
Through frequency and time domain distillation technology, the problem of poor low-quality compressed speech detection performance is solved, efficient forgery detection of low-quality speech is achieved, and the authenticity and security of speech information is improved.
Patent Information
- Application Number
- CN202411020089.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-29
AI Technical Summary
The existing voice forgery detection technology performs poorly in low-quality compressed speech, mainly due to the loss of high-frequency information in the frequency domain and time domain details, resulting in a degradation in detection performance.
Using knowledge distillation-based methods, through frequency and time domain distillation technology, student models can learn compressed data loss information from teacher models, improving the forgery detection performance of low-quality speech. Specific steps include using pre-trained models to extract speech features, training teacher models and student models, and optimizing student models through the weighted loss functions of frequency and time domain distillation modules.
It significantly improves the forgery detection performance of low-quality compressed voice, can effectively identify and combat forgery voice, and improves the authenticity and security of voice information.
Smart Images

Figure CN119049508A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimedia information security technology, and in particular to a detection method for detecting deep forgery in low-quality compressed speech. Background Art
[0002] With the rapid development of artificial intelligence technology, voice forgery technology has developed rapidly in recent years. New voice synthesis technology can generate highly realistic forged voices. Although voice forgery technology has entertainment applications, it has also been abused for fraud, spreading false information, etc., causing serious harm. In recent years, criminals have used voice forgery technology to commit fraud, causing a large amount of property losses. Voice forgery technology has also been used to create false information, causing serious impacts in political and military fields.
[0003] In order to meet the security challenges brought by voice forgery, the research on voice forgery detection technology has received widespread attention. In recent years, a lot of research work has been done on developing effective voice forgery detectors. In 2015, 2017, 2019 and 2021, the ASVspoof Challenge was successfully held, providing an important platform for the research and development of voice forgery detection technology. In 2022 and 2023, the Audio Deep Synthesis Detection Challenge (ADD 2022 and ADD 2023) was also held successively, further promoting the development of voice forgery detection technology.
[0004] Existing speech forgery detection technologies mainly have good detection performance on high-definition datasets. However, in real life, most of the voices on social media are low-definition voices that are stored after being processed by compression algorithms. Low-definition voice data contains less information than high-definition data, so speech forgery detection on low-definition voice is more challenging. Low-quality compressed data mainly has the following two problems: (1) Loss of high-frequency information in the frequency domain
[0005] The human auditory system is more sensitive to low-frequency components and relatively insensitive to high-frequency components. Based on this, many lossy compression encoding methods, such as MP3 and AAC, will intentionally remove some high-frequency information during the compression process to reduce the size of the audio file. In addition, a lower compression bit rate usually results in more high-frequency information loss. Figure 1The results of spectrum analysis of original and compressed speech are shown, where different colors represent the energy intensity of different frequency ranges. Warm colors represent high-energy areas, while cold colors represent low-energy areas. It can be seen that there are obvious differences between the original and compressed speech in the high-frequency part. Based on the fact that the human ear is insensitive to high-frequency information, studies have shown that the forged speech generated by speech synthesis and speech conversion is closer to the real speech in the low-frequency part, but because the high-frequency part is ignored, the forged speech leaves characteristic differences different from the real speech. Since the artificial traces of forged speech are mainly reflected in the high-frequency part, and the compressed low-quality speech usually loses this part of information, it is more difficult to detect forgery of compressed low-quality speech.
[0006] (2) Loss of time domain details
[0007] Compression coding usually aims to reduce the volume of audio data, which will lead to the loss of time domain information of speech signals. This includes the rapid amplitude changes of speech signals, the temporal relationship between speech features, and the perceptual characteristics of speech. Some compression algorithms discretize and represent the amplitude of speech signals. In this process, the signal amplitude may be limited to a certain range, resulting in a reduction in large intensity values, which may lead to the loss of time domain details and make the speech signal smoother, especially in the fast-changing parts. Figure 2 The difference between the original speech and the compressed speech in the time domain waveform is shown. It can be seen that the waveform of the compressed speech is smoother in some periods. Due to the loss of time domain information in compressed speech, the accuracy of the speech authentication system usually decreases, and the detection performance is reduced compared with high-definition speech.
[0008] In summary, existing voice authentication systems are mainly targeted at high-definition voices, while compressed voices widely available on social media lack some useful information for forgery detection. Compared with high-definition voices, forgery detection for compressed voices is more challenging. Therefore, how to accurately detect forgery with low-quality compressed voices is an urgent problem to be solved. Summary of the invention
[0009] In order to solve the problems existing in the above-mentioned prior art, and to address the problem of poor performance in detecting forgery of low-quality compressed speech, the present invention proposes a method for detecting deep forgery of low-quality compressed speech based on frequency domain and time domain knowledge extraction. The method uses knowledge distillation as a framework, adopts a data distillation strategy, uses high-quality data to train the teacher model, and uses low-quality data to train the student model. In addition, through frequency domain and time domain distillation, the student model can learn the frequency domain and time domain information lost by the compressed data from the teacher model, effectively improving the forgery detection performance of low-quality speech. It is worth mentioning that the high-quality data and low-quality data for training the distillation model are paired, and a lossy compression algorithm is used to compress the high-quality data set to generate a corresponding low-quality compressed data set.
[0010] To achieve the above object, the present invention provides the following technical solutions:
[0011] A method for detecting deep fakes of low-quality compressed speech based on frequency domain and time domain knowledge extraction, the steps are as follows:
[0012] S1. Compress the training data set: Six lossy compression algorithms are used to compress the training data set to obtain low-quality data that matches the high-quality training set. The specific steps are as follows:
[0013] S1.1, traverse each audio file in the ASVSpoof 2019 dataset;
[0014] S1.2, randomly select a compression algorithm to compress the audio file;
[0015] S1.3, save the compressed audio file to the specified path;
[0016] Repeat steps S1.1-S1.3 until all audio files are compressed;
[0017] S2. Speech feature extraction: Use the pre-trained large model XLS-R to extract features from speech data; the specific steps are as follows:
[0018] S2.1. Download the pre-trained speech feature extraction model XLS-R;
[0019] S2.2, directly input the read audio file into the XLS-R model to extract audio features;
[0020] S2.3, saving the extracted audio features for subsequent use;
[0021] S3. Teacher model training: Use high-quality raw data to train the teacher model; the specific steps are as follows:
[0022] S3.1. Input the audio features of the ASVSpoof 2019 training set into the ResNet18 network.
[0023] S3.2. Use the cross entropy loss function to train the ResNet18 network so that the model can learn the feature distribution and discriminant information of high-quality speech;
[0024] S3.3. Train until the model achieves a satisfactory fit on the ASVSpoof 2019 training set;
[0025] S4. Student model training: Use low-quality compressed data to train the student model through frequency domain distillation and time domain distillation.
[0026] Furthermore, the specific steps of step S4 are as follows:
[0027] S4.1, repeat steps S2.1 and S2.2, extract the compressed audio features of the ASVSpoof 2019 training set, and input the compressed audio features of the ASVSpoof 2019 training set into the ResNet18 network;
[0028] S4.2, input the intermediate layer features of the student model and the teacher model into the frequency domain distillation module and the time domain distillation module respectively;
[0029] S4.3, calculate the frequency domain distillation loss and the time domain distillation loss;
[0030] S4.4. Add the weighted classification loss, frequency domain distillation loss, and time domain distillation loss of the ResNet18 network to obtain the overall loss of the model; use the back propagation algorithm to train the student model and continuously update the model parameters; train until the model converges to a satisfactory state;
[0031]
[0032] in, represents the classification loss of the student model, represents the frequency domain distillation loss, represents the time domain distillation loss, α, β, and γ are the weights of the student model classification loss, frequency domain distillation loss, and time domain distillation loss, respectively.
[0033] Furthermore, the frequency domain distillation module performs the following steps:
[0034] The intermediate layer features of the teacher model and the student model are converted from the time domain to the frequency domain, and this conversion is achieved using the fast Fourier transform. Through FFT, the spectral characteristics of the speech data are examined and the frequency components that are most affected during the compression process are identified. The FFT formula is as follows:
[0035]
[0036] Where k represents the frequency index in the frequency domain, i represents the imaginary unit; and Frequency domain representations of the features of the intermediate layers of the student and teacher networks, respectively.
[0037] Furthermore, the frequency domain distillation module performs the following steps:
[0038] The square of the l2 norm is used to measure the difference between the frequency domain features of the teacher model and the student model. The difference calculation formula is as follows:
[0039]
[0040] Among them, Ft and Fs represent the frequency domain features of the teacher model and the student model respectively, and F represents the number of feature dimensions. The channel difference index is introduced as a weight in the work, which can amplify the difference part and weaken the similar part. The weight is defined as follows:
[0041]
[0042] Among them, lambda_FD is a hyperparameter used to scale weights. By introducing weights, the model pays more attention to the high-frequency information lost during compression. Combining the above content, the frequency domain distillation loss of the teacher and student models is defined as follows:
[0043]
[0044] Furthermore, the time domain distillation module performs the following steps:
[0045] Wasserstein distance spline and comparison loss are used to extract the time scale of the teacher model and the student model; SWD is a distance metric used to measure the similarity between two probability distributions, and the computational complexity is reduced by data projection; multiple projections are propagated from high-dimensional distributions to various one-dimensional marginal distributions, and the optimal transmission cost of each projection is calculated; let Ω be the probability space, u and v be two probability measures in P(Ω), then the q-Wasserstein distance between u and v in P(Ω) is defined as:
[0046]
[0047] Among them, q is the norm parameter used to control the distance calculation, which determines the measurement method in the calculation; Γ(u,v) represents the set of all possible couplings that can map u to v; c is the distance metric function defined on the probability space Ω; z1 and z2 represent samples from distributions u and v, respectively.
[0048] Furthermore, the time domain distillation module performs the following steps:
[0049] When calculating SWD, a direction vector is randomly drawn on the unit sphere, multiple projection directions are selected, and the two probability distributions Pu and Pv are projected onto each selected projection direction. For each distribution projection, the 1-Wasserstein distance is calculated, which is equivalent to setting the q in the q-Wasserstein distance to 1. Finally, the 1-Wasserstein distances of all projection directions are weighted averaged using appropriate weights to obtain the final SWD. The slice 1-Wasserstein distance (SWD) is defined as:
[0050]
[0051] Among them, R θμ and R θν represents the projection in the i-th or v-th projection direction, respectively, and θ is R d The weight assigned to each projection direction in Thus, computing the piecewise Wasserstein distance involves solving several one-dimensional optimal transport problems with closed-form solutions; in particular, when R θμ and R θν After the samples are sorted in ascending order, SWD can be approximated as:
[0052]
[0053] Where M is the number of uniform random samples θ, N represents the number of samples, and α and β are the θμ and R θν The samples are sorted in ascending order.
[0054] Furthermore, the time domain distillation module performs the following steps:
[0055] Use the Frobenius norm to normalize Fs and Ft; then normalize the feature F s / t Perform element-by-element power operation and finally divide the square of the power operation by the normalized square to get F s and F t The probability representation and The loss function of time domain distillation is defined as follows:
[0056]
[0057] in, express and The contrast loss, and is a hyperparameter that balances SWD and contrastive loss.
[0058] Furthermore, the time domain distillation module performs the following steps:
[0059] Introducing contrast loss in time domain distillation, this loss function aims to narrow the distance between samples of the same category and increase the distance between samples of different categories; The definition is as follows:
[0060]
[0061] Where N represents the number of sample pairs, y n is a label, d stands for and Euclidean distance, △ is a marginal threshold used to specify the minimum distance between negative sample pairs.
[0062] Beneficial effects of the present invention:
[0063] Compared with the prior art, the present invention proposes a method for detecting deep fakes of low-quality compressed speech based on frequency domain and time domain knowledge extraction. The method adopts knowledge distillation as the main framework, uses data distillation, uses high-quality data to train the teacher model, and uses low-quality data to train the student model. Then, frequency domain and time domain distillation are used, so that the student model can learn the frequency domain and time domain information lost by the compressed data from the teacher model, thereby improving the fake detection performance of low-quality data. The present invention can be applied to social media, voice customer service, forensic identification and other fields, effectively identifying and combating fake voices, and maintaining the authenticity and security of voice information. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:
[0065] Figure 1 It is the high frequency difference map of original and compressed speech;
[0066] Figure 2 It is the difference diagram of the original and compressed speech time domain waveforms;
[0067] Figure 3 This is the overall framework diagram of the low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction;
[0068] Figure 4 This is the framework diagram of the frequency domain distillation module;
[0069] Figure 5 This is the framework diagram of the time domain distillation module;
[0070] Figure 6 The EER result diagram for different time domain distillation loss weights. DETAILED DESCRIPTION
[0071] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Figure 1-6 The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction is further explained.
[0072] This paper proposes a low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction. The overall framework is as follows: Figure 3 As shown, the method is divided into 4 steps:
[0073] (1) Compressing the training data set: Six lossy compression algorithms are used to compress the training data set to obtain low-quality data that matches the high-quality training set.
[0074] (2) Speech feature extraction: Use the pre-trained large model XLS-R to extract features from speech data.
[0075] (3) Teacher model training: Use high-quality original data to train the teacher model.
[0076] (4) Student model training: Use low-quality compressed data to train the student model through frequency domain distillation and time domain distillation.
[0077] Furthermore, the specific steps of step (1) are as follows:
[0078] ① Traverse each audio file in the ASVSpoof 2019 dataset;
[0079] ② Randomly select a compression algorithm (MP3, MP2, AAC, OGG, GSM, Opus) to compress the audio file;
[0080] ③Save the compressed audio file to the specified path;
[0081] ④ Repeat steps ①-③ until all audio files are compressed;
[0082] The specific steps of step (2) are as follows:
[0083] ① Download the pre-trained speech feature extraction model XLS-R;
[0084] ② Input the read audio file directly into the XLS-R model to extract audio features;
[0085] ③Save the extracted audio features for subsequent use;
[0086] The specific steps of step (3) are as follows:
[0087] ① Input the audio features of the ASVSpoof 2019 training set into the ResNet18 network;
[0088] ② Use the cross entropy loss function to train the ResNet18 network so that the model can learn the feature distribution and discriminant information of high-quality speech;
[0089] ③ Train until the model achieves a satisfactory fit on the ASVSpoof 2019 training set;
[0090] Furthermore, the specific steps of step (4) are as follows:
[0091] ① Repeat steps 1 and 2 in step 2 to extract the compressed audio features of the ASVSpoof2019 training set, and input the compressed audio features of the ASVSpoof 2019 training set into the ResNet18 network.
[0092] ② Input the intermediate layer features of the student model and the teacher model into the frequency domain distillation module and the time domain distillation module respectively.
[0093] ③ Calculate the frequency domain distillation loss and time domain distillation loss;
[0094] ④ Add the weighted classification loss, frequency domain distillation loss, and time domain distillation loss of the ResNet18 network to get the overall loss of the model. Use the back propagation algorithm to train the student model and continuously update the model parameters. Train until the model converges to a satisfactory state.
[0095]
[0096] in, represents the classification loss of the student model, represents the frequency domain distillation loss, represents the time domain distillation loss, α, β, and γ are the weights of the student model classification loss, frequency domain distillation loss, and time domain distillation loss, respectively.
[0097] Furthermore, the above frequency domain distillation module framework is as follows Figure 4 As shown in Figure 2, the specific steps of the frequency domain distillation module are:
[0098] 1) In order to analyze the difference between high-quality speech data and compressed speech data in the frequency domain, the present invention converts the intermediate layer features of the teacher model and the student model (respectively denoted as H and S) from the time domain to the frequency domain, and uses the fast Fourier transform (FFT) to achieve this conversion. Through FFT, the present invention can examine the spectral characteristics of speech data and identify the frequency components that are most affected during the compression process. The FFT formula is as follows:
[0099]
[0100] Wherein, k represents the frequency index in the frequency domain, and i represents the imaginary unit. and Frequency domain representations of the features of the intermediate layers of the student and teacher networks, respectively.
[0101] 2) In order to allow the student model to learn the frequency domain information lost during the compression process, the present invention uses the square of the l2 norm to measure the difference between the frequency domain features of the teacher model and the student model. The difference calculation formula is as follows:
[0102]
[0103] Among them, Ft and Fs represent the frequency domain features of the teacher model and the student model respectively, and F represents the number of feature dimensions. In order to better highlight the difference in frequency domain features, the present invention introduces the channel difference index as a weight in the work, which can amplify the difference part and weaken the similar part. The weight is defined as follows:
[0104]
[0105] Among them, lambda_FD is a hyperparameter used to scale weights. By introducing weights, the model pays more attention to the high-frequency information lost during compression. Combining the above content, the frequency domain distillation loss of the teacher and student models is defined as follows:
[0106]
[0107] Furthermore, the above-mentioned time domain distillation module framework is as follows Figure 5 As shown in the figure, the specific steps of the time domain distillation module are:
[0108] 3) In order to enable the student model to effectively imitate the teacher model and better learn the missing time scale information from the compressed data, the present invention adopts Wasserstein distance spline and comparison loss to extract the time scale of the teacher model and the student model. SWD is a distance metric used to measure the similarity between two probability distributions. It is a variant of the Wasserstein distance and aims to reduce computational complexity through data projection. It propagates multiple projections from high-dimensional distributions to various one-dimensional marginal distributions and calculates the optimal transmission cost for each projection. Let Ω be the probability space, u and v be two probability measures in P(Ω), then the q-Wasserstein distance between u and v in P(Ω) is defined as:
[0109]
[0110] Where q is the norm parameter used to control the distance calculation, which determines the metric used in the calculation. Γ(u,v) represents the set of all possible couplings that can map u to v. c is the distance metric function defined on the probability space Ω. z1 and z2 represent samples from distributions u and v, respectively.
[0111] 4) To calculate SWD, it is necessary to randomly extract direction vectors on the unit sphere, select multiple projection directions, project the two probability distributions Pu and Pv onto each selected projection direction, and calculate the 1-Wasserstein distance for each distribution projection, which is equivalent to setting the q in the q-Wasserstein distance to 1. Finally, use appropriate weights to perform a weighted average of the 1-Wasserstein distances of all projection directions to obtain the final SWD. The slice 1-Wasserstein distance (SWD) is defined as:
[0112]
[0113] Among them, R θμ and R θν represents the projection in the i-th or v-th projection direction, respectively, and θ is R d The weight assigned to each projection direction in Thus, computing the piecewise Wasserstein distance involves solving several one-dimensional optimal transport problems with closed-form solutions. In particular, when R θμ and R θν After the samples are sorted in ascending order, SWD can be approximated as:
[0114]
[0115] Where M is the number of uniform random samples θ, N represents the number of samples, and α and β are the θμ and Rθν The samples are sorted in ascending order.
[0116] 5) In order to calculate the SWD (Wasserstein distance) between the intermediate layer features of the student model and the teacher model, the present invention needs to convert these features into probability distribution. The specific steps are as follows: Fs and Ft are normalized using the Frobenius norm. Then the normalized feature F s / t Perform element-by-element power operation and finally divide the square of the power operation by the normalized square to get F s and F t The probability representation and Figure 5 The overall time domain distillation process of the present invention is described. The loss function of time domain distillation is defined as follows:
[0117]
[0118] in, express and The contrast loss, and
[0119] is a hyperparameter that balances SWD and contrastive loss.
[0120] 6) In order to make the student model better imitate the teacher model, the present invention introduces contrast loss in time domain distillation. This loss function aims to narrow the distance between samples of the same category and increase the distance between samples of different categories. The definition is as follows:
[0121]
[0122] Where N represents the number of sample pairs, y n is a label, d stands for and Euclidean distance, △ is a marginal threshold used to specify the minimum distance between negative sample pairs.
[0123] Experimental dataset settings:
[0124] This experiment uses five datasets for speech forgery detection research, two of which are public original datasets without any processing, and the other three are public datasets, but their data are compressed to simulate the low-quality compression scenarios targeted by the model.
[0125] (1) ASVspoof 2019 LA dataset
[0126] The ASVspoof Challenge aims to promote research on forged speech detection, mainly targeting automatic speaker recognition deception attacks and defense countermeasures. The ASVspoof 2019 LA Database contains real and forged speech. The types of forged speech include converted speech and synthesized speech, which are mainly generated by 19 forged algorithms (A01-A9). The dataset is divided into training set, validation set and test set. The training set is used to train the model, the validation set is used to select the best performing model in training, and the test set is used to evaluate the model performance. The training set and validation set contain four speech synthesis algorithms and two speech conversion algorithms (A01-A06). In order to evaluate the generalization performance of the model, the deception attack of the test set consists of 13 unknown forged algorithms (A07-A19). The details of the ASVspoof 2019LA dataset are shown in Table 1.
[0127] Table 1 Detailed information of the ASVspoof2019LA dataset
[0128]
[0129] (2) ASVspoof 2021DF dataset
[0130] In order to realistically simulate the unpredictable situation of voice forgery in real life, ASVspoof2021DF Database only provides a test set. Since most of the voices on social media in real life are stored after compression, in order to better simulate the real situation, ASVspoof2021DF Database contains voices compressed by different lossy compression algorithms, which greatly increases the difficulty of detection. The dataset contains 611,829 voices, including real and forged voices, all processed by different lossy codecs for media storage. Table 2 lists the compression codecs used to generate the ASVspoof 2021DF Database, including three known compression codecs and two unknown compression codecs. The source data of ASVspoof 2021DF Database is taken from the test set of ASVspoof 2019LA and other datasets, so it contains forged voices generated using more than 100 different deception attack algorithms.
[0131] Table 2 ASVspoof2021 DF dataset compression details
[0132] Compression Algorithm Compression bit rate DF-C1 - - DF-C2 mp3 80-120 DF-C3 mp3 220-260 DF-C4 m4a 20-32 DF-C5 m4a 96-112 DF-C6 ogg 80-96 DF-C7 ogg 256-320 DF-C8 TBA TBA DF-C9 TBA TBA
[0133] (3) ASVspoof 2019LA-train-com dataset
[0134] The frequency-time domain distillation model proposed in this paper is trained using the teacher-student paradigm. The teacher model is trained using original speech data, and the student model is trained using compressed speech data. Since the student model needs to learn the information lost in the compressed data from the teacher model through knowledge distillation, the training data of the student and teacher models need to be paired, that is, the training data of the student model is compressed using the training data of the teacher model. Specifically, the present invention uses the training set of the ASVspoof2019LA Database to train the teacher model, and uses six lossy compression algorithms, namely MP3, MP2, AAC, OGG, GSM, and Opus, to compress the training set of the ASVspoof 2019LA Database for training the student model.
[0135] (4) Other datasets
[0136] In order to evaluate the generalization performance of the model, the present invention conducts cross-dataset tests on three datasets: In_the_wild, FOR, and Wavefake. The In_the_wild dataset contains 37.9 hours of audio clips, including 17.2 hours of fake audio clips and 20.7 hours of real audio clips. The real voice is collected from materials in real environments such as podcasts and speeches, and the fake voice is created by segmenting 219 publicly available video and audio files (these files are clearly advertised as audio deep fakes); the FOR dataset contains more than 198,000 voices from the latest deep learning speech synthesizers and real voices. It contains four different subsets. The present invention uses its for-2seconds subset test set to test the model. The voices in this subset are all 2s voice clips. The Wavefake dataset consists of 117,985 synthetic audios, which contains approximately 196 hours of synthetic audio files generated by six different speech synthesis models trained in two languages. The dataset is all generated audio, there is no real audio, and the dataset is not divided into training, verification, and test sets. The present invention selects 10,000 speech sounds from the data set for testing.
[0137] Since the only publicly available compressed dataset in the field of voice forgery detection is ASVspoof2021DF Database, In_the_wild, FOR, and Wavefake are all uncompressed raw datasets. In order to evaluate the performance of the model for compressed data, the present invention uses 10 lossy compression algorithms to compress these three datasets. Among them, 6 compression algorithms are known compression algorithms used in the above-mentioned compressed training sets. In order to test the robustness of the model, the present invention adds 4 compression algorithms that are unknown to the trained model: AC3, DTS, WMA, and RA.
[0138] Experimental results and analysis:
[0139] (1) Comparison with other counterfeit detection systems
[0140] In order to evaluate the performance of the frequency-time distillation model proposed in the present invention, the present invention compares it with six other baseline models on two datasets, ASVspoof 2021DF Database and ASVspoof 2019LA Database.
[0141] Table 3 shows the experimental results of the present invention on the ASVspoof 2021DF Database. It can be seen that the model of the present invention achieved an EER of 2.82% on this dataset, which is significantly better than the other six baseline models. Among them, STCAntispoofing Systems and Pindrop Labs'Submission are the models proposed by the first and second place winners of the ASVspoof 2021 competition, and the model of the present invention reduces the EER by 13.23% and 12.82% respectively. This shows that the model of the present invention can effectively identify forged content in low-quality compressed speech.
[0142] Table 3 Experimental results of the proposed method and other five baseline models on the ASVspoof 2021DF dataset
[0143] Model Equal Error Rate (EER) PindropLabs' Submission 16.05% STCAntispoofingSystems 15.64% LLGF+XLS-R 6.10% ATTResNet18+XLS-R+Logmet 5.34% AASIST+XLS-R 4.57% OURMETHOD 2.82%
[0144] Table 4 shows the experimental results of the present invention on the ASVspoof 2019LA Database. It can be seen that the model of the present invention achieved an EER of 0.30% on this dataset, which is also significantly better than the other six baseline models. Among them, RawNet, AASIST, and Dual-Branch Network are three models with better performance on the ASVspoof 2019LA Database in the past two years, but when these three models were evaluated on the ASVspoof 2021DF dataset, their detection performance dropped sharply, with EERs of 24.32%, 19.77%, and 30.28%, respectively. This shows that there is currently a problem in the field of forged speech detection that models with good performance for high-quality data cannot effectively detect low-quality data. The frequency domain-time domain model of the present invention solves this problem well, achieving performance close to SOTA on both high-quality and low-quality data.
[0145] Table 4 Experimental results of the proposed method and other three baseline models on the ASVspoof2019LA and ASVspoof2021DF datasets
[0146] Model 19_LAeval 21_DFeval RawNet2 1.12% 24.32% AASIST 0.83% 19.77% Dual-BranchNetwork 0.80% 30.28% OURMETHOD 0.30% 2.82%
[0147] (2) Cross-dataset experiments
[0148] In order to evaluate the generalization performance of the frequency-time domain distillation model proposed in the present invention, the present invention tests it on four different data sets, and the experimental results are shown in Table 5. The four data sets are ASVspoof 2019LA, In_the_wild, for-2seconds and WAVfake. Among them, the ASVspoof 2019LA and for-2seconds data sets contain real speech and fake speech, the In_the_wild data set only contains fake speech, and the WAVfake data set only contains synthetic speech. Since the only publicly available compressed data set is ASVspoof2021DF Database, and the compression algorithm of this data set is known, in order to more comprehensively evaluate the generalization performance and robustness of the model, the present invention compresses the four data sets of ASVspoof 2019LA, In_the_wild, for-2seconds and WAVfake to obtain four new compressed data sets, and tests the performance of the model on these four compressed data sets. In the compression process, in addition to the 6 known compression algorithms used to compress the training set data, the present invention also adds 4 unknown compression algorithms. The second column of Table 5 is the test results without adding the distillation module, and the third to fifth columns are the test results after adding the distillation module. Among them, the third column is the result on the entire test data set, the fourth column is the test result on the data compressed using 6 known compression algorithms, and the fifth column is the test result on the data compressed using 4 unknown compression algorithms. It is worth noting that since there is only synthetic speech but no real speech in the WAVfake data set, EER cannot be used as the evaluation indicator of this data set. Therefore, on this data set, the present invention uses accuracy (ACC) as the evaluation indicator.
[0149] It can be seen from Table 5 that on four different datasets, the performance of the model of the present invention has been improved to a certain extent after adding the distillation module. On the ASVspoof 2019LA, In_the_wild, and for-2seconds datasets, the EER of the model has decreased, and on the WAVfake dataset, the ACC of the model has also improved. This shows that the frequency-time domain distillation model of the present invention has good generalization performance and can achieve good detection results on datasets of different types and different compression levels.
[0150] In addition, it can be seen from Table 5 that the detection performance of the model of the present invention does not drop significantly when facing data compressed by an unknown compression algorithm. This shows that the model of the present invention has a certain robustness and can adapt to unknown compression algorithms.
[0151] Table 5 Experimental results of the method of the present invention on four different data sets
[0152]
[0153] (3) Ablation experiment
[0154] In order to evaluate the impact of each module in the frequency-time distillation model proposed in the present invention on the final result, the present invention conducted ablation experiments on the ASVspoof 2019LA Database (high-quality data) and the ASVspoof 2021DF Database (low-quality compressed data), and the results are shown in Table 6. As can be seen from Table 6, after using the feature extraction model XLS-R, the EER on the 19LA dataset and the 21DF dataset was reduced by 6.91% and 23.27% respectively compared with the use of lfcc features. This shows that the XLS-R model can learn robust feature representations from large-scale speech data and has good adaptability to various speech. After adding the frequency domain distillation module and the time domain distillation module, the detection performance of the model on the 19LA dataset and the 21DF dataset is further improved. This shows that the two distillation modules can effectively distill the knowledge of the teacher model to the student model, help the student model learn the information lost in the compressed data, and thus improve the detection performance of the model for high-quality and low-quality data. It is worth noting that when both the frequency domain distillation module and the time domain distillation module are added, the model performance reaches the best, with EER of 0.30% and 2.82% on the 19LA dataset and 21DF dataset, respectively. This shows that frequency-time domain distillation can make full use of frequency domain and time domain information to further improve the performance of the model.
[0155] The ablation experiment results of the present invention show that each module in the frequency-time distillation model proposed by the present invention has a significant contribution to the final result. Among them, the XLS-R feature extraction model can learn robust feature representation, and the frequency-domain distillation module and the time-domain distillation module can effectively distill the knowledge of the teacher model to the student model.
[0156] Table 6 Ablation experimental results of each module of the proposed method on ASVspoof2019LA and ASVspoof2021DF datasets
[0157]
[0158] Since the loss of frequency domain distillation is much greater than the loss of time domain distillation, in order to balance the two distillation modules, the present invention experiments on the hyperparameter α of time domain distillation. The present invention takes four different values (α = 500, 520, 540, 560) to test their impact on the performance of the model on the 21DF dataset. The results are as follows: Figure 6 As shown. Figure 6 It can be seen that when α = 520, the model performs best on the 21DF dataset. This shows that a reasonable α value can balance frequency domain distillation and time domain distillation, thereby further improving the performance of the model.
[0159] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for detecting deep fakes of low-quality compressed speech based on frequency domain and time domain knowledge extraction, It is characterized in that Here are the steps: S1. Compress the training data set: Six lossy compression algorithms are used to compress the training data set to obtain low-quality data that matches the high-quality training set. The specific steps are as follows: S1.1, traverse each audio file in the ASVSpoof 2019 dataset; S1.2, randomly select a compression algorithm to compress the audio file; S1.3, save the compressed audio file to the specified path; Repeat steps S1.1-S1.3 until all audio files are compressed; S2, speech feature extraction: Use the pre-trained large model XLS-R to extract features from speech data; The specific steps are as follows: S2.
1. Download the pre-trained speech feature extraction model XLS-R; S2.2, directly input the read audio file into the XLS-R model to extract audio features; S2.3, saving the extracted audio features for subsequent use; S3. Teacher model training: Use high-quality raw data to train the teacher model; the specific steps are as follows: S3.
1. Input the audio features of the ASVSpoof 2019 training set into the ResNet18 network. S3.
2. Use the cross entropy loss function to train the ResNet18 network so that the model can learn the feature distribution and discriminant information of high-quality speech; S3.
3. Train until the model achieves a satisfactory fit on the ASVSpoof 2019 training set; S4. Student model training: Use low-quality compressed data to train the student model through frequency domain distillation and time domain distillation.
2. The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction as claimed in claim 1, It is characterized in that The specific steps of step S4 are as follows: S4.1, repeat steps S2.1 and S2.2, extract the compressed audio features of the ASVSpoof 2019 training set, and input the compressed audio features of the ASVSpoof 2019 training set into the ResNet18 network; S4.2, input the intermediate layer features of the student model and the teacher model into the frequency domain distillation module and the time domain distillation module respectively; S4.3, calculate the frequency domain distillation loss and the time domain distillation loss; S4.
4. Add the weighted classification loss, frequency domain distillation loss, and time domain distillation loss of the ResNet18 network to obtain the overall loss of the model; use the back propagation algorithm to train the student model and continuously update the model parameters; train until the model converges to a satisfactory state; in, represents the classification loss of the student model, represents the frequency domain distillation loss, represents the time domain distillation loss, α, β, and γ are the weights of the student model classification loss, frequency domain distillation loss, and time domain distillation loss, respectively.
3. The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction as claimed in claim 1, It is characterized in that The frequency domain distillation module performs the following steps: The intermediate layer features of the teacher model and the student model are converted from the time domain to the frequency domain, and this conversion is achieved using the fast Fourier transform. Through FFT, the spectral characteristics of the speech data are examined and the frequency components that are most affected during the compression process are identified. The FFT formula is as follows: Where k represents the frequency index in the frequency domain, i represents the imaginary unit; and Frequency domain representations of the features of the intermediate layers of the student and teacher networks, respectively.
4. The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction as claimed in claim 1, It is characterized in that The frequency domain distillation module performs the following steps: The square of the l2 norm is used to measure the difference between the frequency domain features of the teacher model and the student model. The difference calculation formula is as follows: Among them, Ft and Fs represent the frequency domain features of the teacher model and the student model respectively, and F represents the number of feature dimensions. The channel difference index is introduced as a weight in the work, which can amplify the difference part and weaken the similar part. The weight is defined as follows: Among them, lambda_FD is a hyperparameter used to scale weights. By introducing weights, the model pays more attention to the high-frequency information lost during compression. Combining the above content, the frequency domain distillation loss of the teacher and student models is defined as follows:
5. The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction as claimed in claim 1, It is characterized in that The time domain distillation module performs the following steps: Wasserstein distance spline and comparison loss are used to extract the time scale of the teacher model and the student model; SWD is a distance metric used to measure the similarity between two probability distributions, and the computational complexity is reduced by data projection; multiple projections are propagated from high-dimensional distributions to various one-dimensional marginal distributions, and the optimal transmission cost of each projection is calculated; let Ω be the probability space, u and v be two probability measures in P(Ω), then the q-Wasserstein distance between u and v in P(Ω) is defined as: Among them, q is the norm parameter used to control the distance calculation, which determines the measurement method in the calculation; Γ(u,v) represents the set of all possible couplings that can map u to v; c is the distance metric function defined on the probability space Ω; z 1 and z 2 denote samples from distributions u and v respectively.
6. The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction as claimed in claim 1, It is characterized in that The time domain distillation module performs the following steps: When calculating SWD, a direction vector is randomly drawn on the unit sphere, multiple projection directions are selected, and the two probability distributions Pu and Pv are projected onto each selected projection direction. For each distribution projection, the 1-Wasserstein distance is calculated, which is equivalent to setting the q in the q-Wasserstein distance to 1. Finally, the 1-Wasserstein distances of all projection directions are weighted averaged using appropriate weights to obtain the final SWD. The slice 1-Wasserstein distance (SWD) is defined as: Among them, R θμ and R θν represents the projection in the i-th or v-th projection direction, respectively, and θ is R d The weight assigned to each projection direction in Thus, computing the piecewise Wasserstein distance involves solving several one-dimensional optimal transport problems with closed-form solutions; in particular, when R θμ and R θν After the samples are sorted in ascending order, SWD can be approximated as: Where M is the number of uniform random samples θ, N represents the number of samples, and α and β are the θμ and R θν The samples are sorted in ascending order.
7. The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction as claimed in claim 1, It is characterized in that The time domain distillation module performs the following steps: Use the Frobenius norm to normalize Fs and Ft; then normalize the feature F s / t Perform element-by-element power operation and finally divide the square of the power operation by the normalized square to get F s and F t The probability representation and The loss function of time domain distillation is defined as follows: in, express and The contrast loss is and is a hyperparameter that balances SWD and contrastive loss.
8. The low-quality compressed speech deep fake detection method based on frequency domain and time domain knowledge extraction as claimed in claim 1, It is characterized in that The time domain distillation module performs the following steps: Introducing contrast loss in time domain distillation, this loss function aims to narrow the distance between samples of the same category and increase the distance between samples of different categories; The definition is as follows: Where N represents the number of sample pairs, y n is a label, d stands for and Euclidean distance, △ is a marginal threshold used to specify the minimum distance between negative sample pairs.
Citation Information
Cited By
High-fidelity audio generation method and device and storage medium
CN119673199A
A high-fidelity audio generation method, apparatus and storage medium
CN119673199B
Method and system for identifying forged audio based on improved RawNet2
CN119811426A
Operation and maintenance anomaly detection method and system based on core set selection
CN121436976A
