A deepfake language detection method based on boundary perception

Abstract

The application discloses a kind of based on boundary perception's deep counterfeit language detection method.Real speech signal and the speech signal containing part counterfeit speech are collected, and each speech signal is marked in frame level counterfeit boundary, then all speech signals are processed to obtain language detection dataset with fixed frame number;Deep counterfeit speech detection model is constructed, language detection dataset is input into deep counterfeit speech detection model for training, and the trained deep counterfeit speech detection model is obtained;After the speech signal to be measured is processed with fixed frame number, it is input into the trained deep counterfeit speech detection model, and the binary classification detection result of the speech signal to be measured is obtained, and the binary classification detection result is real category or counterfeit category.The application effectively solves the information coupling problem caused by global feature fusion in partial counterfeit scene, improves the counterfeit boundary positioning accuracy and overall detection performance, and has stronger robustness when dealing with complex environmental factors.

Description

Technical Field

[0001] This invention belongs to the field of information security and speech signal processing, and specifically relates to a deep forgery language detection method based on boundary awareness. Background Technology

[0002] With the development of generative speech technologies such as text-to-speech (TTS) and voice conversion (VC), deepfake audio attacks are becoming increasingly diverse. In addition to forging entire audio segments, "partial forgery" attacks, such as replacing specific paragraphs and inserting key sentences, are becoming more and more common, posing a serious challenge to judging the authenticity of audio content.

[0003] Current mainstream audio forgery detection methods mostly employ self-supervised pre-trained models (such as Wav2Vec and WavLM) as front-end feature extractors. These models can learn general acoustic features from large-scale unlabeled speech data, reducing reliance on forged samples. However, existing methods have significant shortcomings when dealing with some forged audio: for example, global pooling-based methods average the features of real and forged segments, leading to a significant drop in detection performance when the proportion of forged segments is low; while some forgery localization methods can identify forgery boundaries, they do not effectively filter the multi-layer features of the self-supervised model and lack the ability to adaptively select sensitive layers.

[0004] Therefore, there is an urgent need for an audio forgery detection scheme that can simultaneously achieve accurate localization of forgery boundaries and on-demand filtering of multi-layer features of a self-supervised model in certain forgery scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a boundary-aware deep language forgery detection method. It aims to solve the problem that existing audio forgery detection methods, in certain forgery scenarios, do not fully utilize the hidden layers of the self-supervised pre-trained model, leading to the coupling of feature information in the forged audio and resulting in low recognition rates for some forged audio.

[0006] The technical solution adopted in this invention is: This invention includes the following steps: S1. Collect the entire real speech signal and the speech signal containing some fake speech, and perform frame-level fake boundary annotation on each speech signal. Then, process all speech signals with a fixed number of frames to obtain the language detection dataset. S2. Construct a deep fake speech detection model. Input the language detection dataset into the deep fake speech detection model for training to obtain a trained deep fake speech detection model. S3. After processing the speech signal to be tested for a fixed number of frames, input it into the trained deep fake speech detection model to obtain the binary classification detection result of the speech signal to be tested, which is either the real category or the fake category.

[0007] The fixed frame number processing is as follows: if the number of frames of the voice signal is greater than or equal to the preset frame number, the tail is truncated so that the number of frames of the voice signal is equal to the preset frame number; otherwise, zero values ​​are filled at the tail until the number of frames of the voice signal reaches the preset frame number.

[0008] The deepfake speech detection model includes a self-supervised feature extraction module, a boundary recognition module, and a sensitive layer selection and classification module. The speech signal processed with a fixed number of frames is input into the self-supervised feature extraction module to extract features and obtain multi-layer deep features. Then, the multi-layer deep features are input into the boundary recognition module to locate the fake boundary and obtain a boundary frame label sequence. Subsequently, the multi-layer deep features and the boundary frame label sequence are input into the sensitive layer selection and classification module to perform segmented sensitive layer weighted fusion and classification to obtain the binary classification detection result of the speech signal under test.

[0009] The self-supervised feature extraction module specifically adopts the pre-trained Wav2Vec2-large model.

[0010] The boundary recognition module includes an average pooling layer, an AGRES2Net module, a linear mapping layer, a sigmoid activation function, and a binarization operation. Multi-layer deep features are input into the average pooling layer, and average pooling is performed along the layer dimensions to obtain fused features. The fused features are then input into the AGRES2Net module for boundary feature enhancement processing to obtain boundary fused features. Subsequently, the boundary fused features are sequentially passed through the linear mapping layer and the sigmoid activation function to obtain the forged boundary probability of each frame. The forged boundary probability of each frame is binarized according to a preset threshold to obtain a boundary frame label sequence.

[0011] The sensitivity layer selection and classification module includes a sensitivity weight generation module, a weighted fusion module, an average pooling layer, and a classification head. Based on the boundary frame label sequence, the multi-layer deep features are divided into multiple audio segment features along the time axis. Each audio segment feature is input into the sensitivity weight generation module to generate sensitivity weights for each audio segment feature in each hidden layer. Each audio segment feature and its corresponding sensitivity weight are input into the weighted fusion module, which performs weighted fusion on the features of each audio segment in each hidden layer to obtain the segment feature vector of each audio segment. Then, all segment feature vectors are concatenated along the time axis and sequentially processed by the average pooling layer and the classification head to obtain the binary classification detection result of the speech signal under test. The classification head is specifically a multilayer perceptron.

[0012] The sensitive weight generation module processes the data according to the following formula: in, This represents the sensitivity weight vector of the b-th audio segment. This represents the Sigmoid activation function. This represents the mapping of fully connected layers. This indicates the average pooling operation. This represents the feature of the b-th audio segment.

[0013] The loss function of the deepfake speech detection model is a weighted sum of the forgery classification loss and the boundary prediction loss, specifically set according to the following formula: in, Represents the loss function. Indicates falsified classification losses, Indicates the boundary prediction loss. This represents the balance parameter used to balance the spoofing classification loss and the boundary prediction loss. This represents the output of the final classification head, where y represents the true label and i represents the sequence number of audio frames. Represents the boundary probability. Indicates the true label of the boundary. This represents the boundary true label of the i-th frame. This represents the boundary probability of the i-th frame. Indicates the category label number, , =1 indicates a real sample. =0 indicates a fake sample. This represents the output score of the true class of the a-th sample. This represents the model's output score for the a-th sample of category c. Let 'a' represent the batch sample number, and 'a' represent the batch sample sequence. This represents the weight corresponding to the true class label of the a-th sample.

[0014] The beneficial effects of this invention are: This invention constructs a boundary recognition module and utilizes gated Res2Net and attention mechanisms to locate frame-level boundaries of forged segments. This effectively avoids the coupling and dilution of real and forged segment features by global pooling operations, improving detection accuracy in low-proportion forgery scenarios in some forgery identification. Simultaneously, it segments audio features based on predicted boundaries and independently calculates the sensitive weights of the hidden layer of the self-supervised model for each segment. This enables on-demand adaptive weighted fusion of multi-layer features from different audio regions, overcoming the shortcomings of existing methods in utilizing information from different feature layers of the pre-trained model. This invention can simultaneously achieve forgery boundary localization and whole-speech authenticity determination in some forgery scenarios, making it suitable for high-security scenarios such as voice evidence collection and content authenticity verification. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the boundary-aware deep fake speech detection method in this embodiment.

[0016] Figure 2 This is a schematic diagram of the boundary recognition module structure in this embodiment.

[0017] Figure 3 This is a schematic diagram of the sensitive layer selection and classification module structure in this embodiment.

[0018] Figure 4 This is the boundary probability map generated by the model in this embodiment. Detailed Implementation

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0021] This embodiment includes the following steps: S1. Collect the entire real speech signal and the speech signal containing some fake speech, and perform frame-level fake boundary annotation on each speech signal. Then, process all speech signals with a fixed number of frames to obtain the language detection dataset. In this embodiment, the dataset used is the PartialSpoof dataset, which contains a large amount of real speech (from ASVspoof2019) and some fake speech, and provides frame-level fake labels. To facilitate batch processing, all audio in the dataset is fixed to the same length.

[0022] In this embodiment, the maximum audio length is set to 4 seconds, corresponding to 200 frames. If the audio length exceeds 200 frames, the end is truncated; if it is less than 200 frames, zero values ​​are padded to bring the length to 200 frames, and a mask is added. Since the ratio of fake data to real data in the PartialSpoof dataset is 9:1, data augmentation is performed during training, setting the weight of real audio to 9 and the weight of some fake audio to 1.

[0023] S2. Construct a deep fake speech detection model. Input the language detection dataset into the deep fake speech detection model for training to obtain a trained deep fake speech detection model. S3. After processing the speech signal to be tested for a fixed number of frames, input it into the trained deep fake speech detection model to obtain the binary classification detection result of the speech signal to be tested, which is either the real category or the fake category.

[0024] The fixed frame number processing is as follows: if the number of frames of the speech signal is greater than or equal to the preset frame number, the tail is truncated so that the number of frames of the speech signal is equal to the preset frame number; otherwise, zero values ​​are filled at the tail until the number of frames of the speech signal reaches the preset frame number, and a mask is generated for marking valid frames and filling frames.

[0025] The deepfake speech detection model includes a self-supervised feature extraction module, a boundary recognition module, and a sensitive layer selection and classification module. The speech signal processed with a fixed number of frames is input into the self-supervised feature extraction module to extract features and obtain multi-layer deep features. Then, the multi-layer deep features are input into the boundary recognition module to locate the fake boundary and obtain a boundary frame label sequence. Subsequently, the multi-layer deep features and the boundary frame label sequence are input into the sensitive layer selection and classification module to perform segmented sensitive layer weighted fusion and classification to obtain the binary classification detection result of the speech signal under test.

[0026] In step S2, the training of the deepfake speech detection model adopts a two-stage training strategy: In the first stage, the parameters of the self-supervised feature extraction module are frozen, and only the parameters of the boundary recognition module and the sensitive layer selection and classification module are trained until the preset number of rounds is reached. In the second stage, the parameters of the self-supervised feature extraction module are unfrozen, and the learning rate of the self-supervised feature extraction module is set lower than that of the boundary recognition module and the sensitivity layer selection and classification module for joint training. That is, the Wav2Vec2 parameters are first frozen, and only the boundary recognition and classification modules are trained; then all parameters are unfrozen for joint fine-tuning.

[0027] In this embodiment, the learning rate of the Wav2Vec2 layer is set to 1 / 10 of that of the custom layer (in this embodiment, the learning rate of the custom layer is 1×10). -4 Wav2Vec2 learning rate 1×10 -5 The optimizer uses AdamW with a weight decay of 1×10⁻⁶. -5 Cosine annealing is used for learning rate scheduling, with a minimum learning rate of 1×10⁻⁶. -6 The gradient clipping threshold is set to 1.0. Training is performed for 30 epochs with a batch size of 16. The model is saved every 5 epochs on the validation set, and the best model is saved based on the validation set EER (equal error rate).

[0028] The self-supervised feature extraction module specifically adopts the pre-trained Wav2Vec2-large model.

[0029] This embodiment uses a pre-trained Wav2Vec2-large model as the front-end feature extractor. Input speech signal First, the sequence is converted into a frame-level feature sequence by a feature encoder (multiple convolutional layers), and then input into a module containing... In a multi-layer Transformer network, the feature output matrix of each layer is obtained, totaling... layer, , ,in This represents the number of frames in the audio. For feature dimension, For the output of the convolutional layer, .

[0030] The boundary recognition module includes an average pooling layer, an AGRES2Net module, a linear mapping layer, a sigmoid activation function, and a binarization operation. Multi-layer deep features are input into the average pooling layer, and average pooling is performed along the layer dimensions to obtain fused features. The fused features are then input into the AGRES2Net module for boundary feature enhancement processing to obtain boundary fused features. Subsequently, the boundary fused features are sequentially passed through the linear mapping layer and the sigmoid activation function to obtain the forged boundary probability of each frame. The forged boundary probability of each frame is binarized according to a preset threshold to obtain a boundary frame label sequence.

[0031] The core of the boundary recognition module is to use the AGRes2Net module to capture boundary features. The specific process is as follows: First, multi-layer features are processed... Average pooling along the layer dimension yields the fused features. Then input it into the GRes2Net module.

[0032] GRes2Net inputs along the channel dimension Evenly divided into Individual feature map Then, these sub-feature maps are passed through multiple one-dimensional convolutional layers. Processing is performed along frame number T (time). In each... Residual connections were used between them, along with a gating mechanism. This effectively captures the potential relationships between feature maps, avoids invalid features from contaminating subsequent scales, and thus enables more accurate boundary identification.

[0033] The gating network is implemented using 1×1 convolutions, and the output weights are used to adjust the contribution of the previous group's features to the current group. Adding attention allows for the capture of dependencies between groups with large intervals. Assuming each... Containing the same number of channels, let's say J channels, i.e., D = s * j. yes The feature map of the Jth channel. Three fully connected layers are used to learn... The query, key, and value (respectively using...) , , (Indicated). Similarly, using and express Queries, keys and values, and The Middle n Feature maps of each channel. Given two distinct feature maps. and We calculate channel attention as follows: Finally, the boundary fusion features are obtained: Boundary fusion features are achieved through a linear mapping layer ( ). x This is followed by a sigmoid activation function to obtain the boundary probabilities. Then apply the threshold operation to the... Binarization is performed to obtain frame-level boundary prediction. , Based on the timeline, the suspected boundary is set to 1, and the others are set to 0.

[0034] The sensitivity layer selection and classification module includes a sensitivity weight generation module, a weighted fusion module, an average pooling layer, and a classification head. Based on the boundary frame label sequence, the multi-layer deep features are divided into multiple audio segment features along the time axis. Each audio segment feature is input into the sensitivity weight generation module to generate sensitivity weights for each audio segment feature in each hidden layer. Each audio segment feature and its corresponding sensitivity weight are input into the weighted fusion module, which performs weighted fusion on the features of each audio segment in each hidden layer to obtain the segment feature vector of each audio segment. Then, all segment feature vectors are concatenated along the time axis and sequentially processed by the average pooling layer and the classification head to obtain the binary classification detection result of the speech signal under test. The classification head is specifically a multilayer perceptron.

[0035] The entire audio According to the predicted boundary Split into M+1 audio segments M is The number of frames containing 1 is the number of boundary frames, calculated independently. weights of hidden layers .definition Let the weight of the l-th layer be the weight of the b-th audio segment. Where F(a:b) represents the features Perform a slicing operation along T, selecting all rows from a to b, retaining all features of T and D, given... By splicing and merging, the segmented recalibration feature can be obtained. : Finally, for Temporal average pooling is performed to obtain a global representation of the entire speech segment. This representation is then input into a classification head (two-layer MLP) for binary classification of fake / real speech, and the output is logits. .

[0036] The sensitive weight generation module processes the data according to the following formula: in, This represents the sensitivity weight vector of the b-th audio segment in each hidden layer. This represents the Sigmoid activation function. This represents the mapping of fully connected layers. This represents the average pooling operation, used to compress audio segment features along the time dimension. This represents the feature of the b-th audio segment.

[0037] During the inference phase, the boundary probabilities of the model output are... A one-dimensional moving average smoothing process (kernel size 5) is performed, and then local peaks are detected as fake boundaries. Peaks must meet the following criteria: probability greater than 0.6 times the global maximum, distance from adjacent peaks not less than 5 frames, and peak prominence greater than 0.05. Finally, a binary peak mask is output for segmentation.

[0038] This embodiment uses a joint loss function to optimize the model. The loss function of the deep forgery speech detection model consists of the forgery classification loss. and boundary prediction loss The weighted summation is calculated using the following formula: To address the imbalance between real samples (label 1) and fake samples (label 0) in the dataset, a cross-entropy loss function with class weights is employed. Boundary loss: Employs masked binary cross-entropy loss, calculating the loss only for valid frames (non-padded regions): in, Represents the loss function. Indicates falsified classification losses, Indicates the boundary prediction loss. This represents the balance parameter used to balance the spoofing classification loss and the boundary prediction loss. This represents the output of the final classification header, where y represents the true label. In this context, 'i' represents the sequence number of audio frames. Describing boundary probabilities , Indicates the true label of the boundary. This represents the boundary true label of the i-th frame. This represents the boundary probability of the i-th frame. Indicates the category label number, , =1 indicates a real sample. =0 indicates a fake sample. and These are the weights for real samples and fake samples, respectively. This represents the output score of the true class of the a-th sample. This represents the model's output score for the a-th sample of category c. Let 'a' represent the batch sample number, and 'a' represent the batch sample sequence. This represents the weight corresponding to the true class label of the a-th sample. and In this context, 'i' represents the frame number, the i-th frame, which is a 4-second audio file. Sampling occurs every 20ms, resulting in 200 frames. This represents the number of audio entries, specifically entry a.

[0039] This embodiment sets the parameters based on training set statistics. =9.0, =1.0, which allows the minority class (real samples) to receive a higher gradient contribution, thus alleviating the class imbalance problem.

[0040] In this embodiment, boundary label The values ​​∈{0,1} are generated as follows: First, based on the frame-level real / fake labels provided by the dataset, the label abrupt change positions (0→1 or 1→0) are detected, with 20ms frame lengths corresponding to 4 seconds of audio (200 frames). Considering the ambiguity of boundary regions and annotation errors, a tolerance mechanism is introduced: each abrupt change position is then... Frames within the frame range are marked as strong boundaries (label value set to 1), and will then be expanded outwards. Frames (excluding strong boundary regions) are marked as weak boundaries (label value set to 0.5), and other frames are marked as 0. This embodiment sets... =2 frames, =1 frame.

[0041] This invention utilizes a pre-trained self-supervised model to extract multi-layer deep features from the input speech. A boundary recognition module is constructed using a gated Res2Net and an attention mechanism to capture multi-scale acoustic features and predict frame-level forgery boundary probabilities. Based on the detected boundaries, the audio features are divided into multiple segments. The sensitivity weights of each hidden layer of the self-supervised model are independently calculated for each segment, and then layer-wise weighted fusion is performed. Finally, the spliced ​​global temporal features are input into a classifier to achieve forgery / realism discrimination of the entire audio segment.

[0042] The above detailed embodiments illustrate the technical solution and beneficial effects of the present invention. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A boundary-aware deep forgery language detection method, characterized in that, The method includes the following steps: S1. Collect the entire real speech signal and the speech signal containing some fake speech, and perform frame-level fake boundary annotation on each speech signal. Then, process all speech signals with a fixed number of frames to obtain the language detection dataset. S2. Construct a deepfake speech detection model. Input the language detection dataset into the deepfake speech detection model for training to obtain a trained deepfake speech detection model. S3. After processing the speech signal to be tested for a fixed number of frames, input it into the trained deep fake speech detection model to obtain the binary classification detection result of the speech signal to be tested, which is either the real category or the fake category.

2. The boundary-aware deep forgery language detection method according to claim 1, characterized in that: The fixed frame number processing is as follows: if the number of frames of the voice signal is greater than or equal to the preset number of frames, the tail is truncated so that the number of frames of the voice signal is equal to the preset number of frames; otherwise, zero values ​​are filled at the tail until the number of frames of the voice signal reaches the preset number of frames.

3. The boundary-aware deep forgery language detection method according to claim 1, characterized in that: The deepfake speech detection model includes a self-supervised feature extraction module, a boundary recognition module, and a sensitive layer selection and classification module. The speech signal processed with a fixed number of frames is input into the self-supervised feature extraction module to extract features and obtain multi-layer deep features. Then, the multi-layer deep features are input into the boundary recognition module to locate the fake boundary and obtain a boundary frame label sequence. Subsequently, the multi-layer deep features and the boundary frame label sequence are input into the sensitive layer selection and classification module to perform segmented sensitive layer weighted fusion and classification to obtain the binary classification detection result of the speech signal under test.

4. The boundary-aware deep forgery language detection method according to claim 3, characterized in that: The self-supervised feature extraction module specifically adopts the pre-trained Wav2Vec2-large model.

5. The boundary-aware deep forgery language detection method according to claim 3, characterized in that: The boundary recognition module includes an average pooling layer, an AGRES2Net module, a linear mapping layer, a sigmoid activation function, and a binarization operation. Multi-layer deep features are input into the average pooling layer, and average pooling is performed along the layer dimensions to obtain fused features. The fused features are then input into the AGRES2Net module for boundary feature enhancement processing to obtain boundary fused features. Subsequently, the boundary fused features are sequentially passed through the linear mapping layer and the sigmoid activation function to obtain the forged boundary probability of each frame. The forged boundary probability of each frame is binarized according to a preset threshold to obtain a boundary frame label sequence.

6. The boundary-aware deep forgery language detection method according to claim 3, characterized in that: The sensitivity layer selection and classification module includes a sensitivity weight generation module, a weighted fusion module, an average pooling layer, and a classification head. Based on the boundary frame label sequence, the multi-layer deep features are divided into multiple audio segment features along the time axis. Each audio segment feature is input into the sensitivity weight generation module to generate sensitivity weights for each audio segment feature in each hidden layer. Each audio segment feature and its corresponding sensitivity weight are input into the weighted fusion module, which performs weighted fusion on the features of each audio segment in each hidden layer to obtain the segment feature vector of each audio segment. Then, all segment feature vectors are concatenated along the time axis and sequentially processed by the average pooling layer and the classification head to obtain the binary classification detection result of the speech signal under test. The classification head is specifically a multilayer perceptron.

7. The boundary-aware deep forgery language detection method according to claim 6, characterized in that: The sensitive weight generation module processes the data according to the following formula: in, This represents the sensitivity weight vector of the b-th audio segment. This represents the Sigmoid activation function. Represents the mapping of fully connected layers. This indicates the average pooling operation. This represents the feature of the b-th audio segment.

8. The deep forgery language detection method based on boundary awareness according to claim 1, characterized in that: The loss function of the deepfake speech detection model is a weighted sum of the forgery classification loss and the boundary prediction loss, specifically set according to the following formula: in, Represents the loss function. Indicates falsified classification loss, Indicates the boundary prediction loss. This represents the balance parameter used to balance the spoofing classification loss and the boundary prediction loss. This represents the output of the final classification head, where y represents the true label and i represents the sequence number of audio frames. Represents the boundary probability. Indicates the true label of the boundary. This represents the boundary true label of the i-th frame. This represents the boundary probability of the i-th frame. Indicates the category label number. , =1 indicates a real sample. =0 indicates a fake sample. This represents the output score of the true class of the a-th sample. This represents the model's output score for the a-th sample of category c. Let 'a' represent the batch sample number, and 'a' represent the batch sample sequence. This represents the weight corresponding to the true class label of the a-th sample.

Images