Multitask learning-based forged voice detection method and system

Abstract

The invention provides a forged voice detection method and system based on multi-task learning, and the method comprises the steps: carrying out the preprocessing of original audio data, and extracting a primary feature through a pre-training model; then, inputting the primary features into a forged voice detection model, and carrying out multi-task learning; thirdly, jointly optimizing classification loss and reconstruction loss, and training the forged voice detection model until convergence; and finally, performing authenticity detection on to-be-detected audio data by using the trained forged voice detection model, and distinguishing whether the to-be-detected audio data is forged voice or not. The method is suitable for acoustic scenes in which the number of real voices and the number of counterfeit voices are unbalanced, is also suitable for detecting counterfeit voices of unknown counterfeit types, and has good universality and generalization.

Description

Technical Field

[0001] The present invention relates to the field of speech processing and biometrics technology, and in particular to a forged speech detection method and system based on multi-task learning. Background Art

[0002] With the development of deep learning technologies, such as AI-generated content, text-to-speech, and voice conversion, artificially generated voices have reached a level that is indistinguishable from the real thing. While bringing convenience to areas such as voice interaction, it also poses a serious threat to judicial evidence collection, social stability, and the safety of people's lives and property. Forged voice detection aims to determine whether a given voice is forged, and it plays a key role in protecting user privacy, preventing telecommunications fraud, and enhancing the reliability of voice systems.

[0003] In the prior art, forged speech detection based on deep neural networks usually consists of manual feature calculation, embedding extraction using neural networks, classifiers, and loss functions. Among them, manual feature calculation relies on the theoretical and empirical understanding of speech signals, and uses the characteristics of human auditory perception to effectively extract key information from speech. Inspired by the speaker verification task, forged speech detection also attempts to use manual features as model input, such as MFCC, CQCC, FBank, etc. Using neural networks to extract embeddings can learn more complex features and capture high-level semantic information of speech. This automatic feature learning can achieve efficient results even with limited data resources. Finally, the embedded features extracted by the model are input into the classifier and the constrained loss function. Commonly used loss functions include binary cross entropy (BCE) loss and mean square error (MSE) loss.

[0004] Based on deep neural network-based forged speech detection, the current mainstream methods focus on learning time-frequency (time-frequency) representations. For example, in order to capture the relationship between different sub-bands, Tak et al. used graph neural networks (GNNs) to model non-Euclidean data manifolds across different sub-bands and time periods; Liu et al. proposed Rawformer, which uses position-related local-global dependencies, and uses Transformer to more effectively capture global dependencies through self-attention mechanisms. However, the above methods all rely on specific data sets for supervised learning, and the training data types are single and identically distributed, which leads to poor forged speech detection performance when facing unknown forgery types or new application scenarios. Summary of the invention

[0005] In view of the shortcomings in the prior art, the present invention provides a method and system for detecting forged speech based on multi-task learning, which utilizes a pre-trained model to obtain primary features of speech, and uses two tasks to share the primary features to improve the learning efficiency and generalization performance of the model. At the same time, different tasks provide complementary information so that the model can understand the data more comprehensively.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] Forged speech detection method based on multi-task learning:

[0008] Preprocess the raw audio data and then extract primary features through the pre-trained model;

[0009] The extracted primary features are input into the forged speech detection model to complete the classification and generation tasks:

[0010] For classification tasks, the primary features are downsampled and then divided into frequency dimensions. The local downsampling module is used to extract the marker feature A with high local discriminability. The marker feature A is connected with a learnable classification marker and then the position code is added, which is recorded as the marker feature B. The marker feature B is sent to the Conformer encoder to obtain multi-scale high-level features. The multi-scale high-level features are sent to the classifier to determine whether it is real speech or forged speech.

[0011] For the generation task, the primary features are downsampled, then divided into blocks in the frequency dimension and randomly masked at a ratio of γ. Only the unmasked blocks pass through the local downsampling module to obtain the marker features of the unmasked blocks. After passing through the Conformer encoder, the high-level marker features of the unmasked blocks are obtained. A shared and learnable mask marker is repeated and interpolated into the high-level marker features of the unmasked blocks, and the masked parts are interpolated and repeated. Finally, the reconstructed primary features are obtained through the Conformer decoder.

[0012] Training the fake speech detection model, combining the binary cross entropy loss in the classification task and the mean square error loss in the generation task, until convergence;

[0013] The trained forged speech detection model is used to detect the audio data to be tested to determine whether it is forged speech.

[0014] Furthermore, the pre-trained model uses the speech recognition model Wav2Vec2-XLS-R for primary feature extraction.

[0015] Furthermore, the speech recognition model Wav2Vec2-XLS-R is trained unsupervisedly using speech data from different real complex domains, and the speech data from different real complex domains is specifically a large-scale multilingual dataset containing 436,000 hours and a total of 128 types of speech.

[0016] Furthermore, the marker feature A is connected with a learnable classification marker, that is, an initialized vector is attached to the front of the marker feature A for transmitting and aggregating information in the network.

[0017] Furthermore, the frequency dimension segmentation specifically includes: processing the primary features through adaptive maximum pooling, downsampling the time step, and then segmenting in the frequency dimension.

[0018] Furthermore, a shared convolutional neural network is used in the local downsampling module to downsample each block, retaining the key spectral information in the input features and obtaining tokenized block-level features, namely, labeled features.

[0019] Furthermore, the Conformer encoder parameters in the classification task and the generation task are shared.

[0020] Furthermore, the binary cross entropy loss in the joint classification task and the mean square error loss in the generation task, wherein the binary cross entropy loss is:

[0021]

[0022] Where N is the number of samples in the original audio data, and f i represents the multi-scale high-level features extracted by the Conformer encoder, y i represents the label of the sample in the original audio data, and g represents the probability distribution extracted by the classifier;

[0023] The mean square error loss is:

[0024]

[0025] Among them, X i Represents the primary characteristics, represents the primary features of the reconstruction;

[0026] The overall loss function is expressed as follows:

[0027] L=L BCE +λL MSE

[0028] Here, λ is a hyperparameter of the balance loss.

[0029] A forged speech detection system based on multi-task learning, comprising:

[0030] The primary feature extraction module is used to preprocess the raw audio data and extract primary features through the pre-trained model;

[0031] The multi-task learning module inputs the extracted primary features into the forged speech detection model to complete the classification task and generation task;

[0032] The training module jointly optimizes the binary cross entropy loss in the classification task and the mean square error loss in the generation task to train the fake speech detection model;

[0033] The result identification module uses the trained forged speech detection model to detect the authenticity of the audio data to be tested.

[0034] A computer device comprises a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor and executes the forged speech detection method based on multi-task learning.

[0035] The beneficial effects of the present invention are:

[0036] (1) The primary features extracted by the pre-trained model in the present invention have rich semantic information, which improves the generalization ability of the model in downstream tasks;

[0037] (2) The local downsampling module in the present invention can capture the fine-grained features within the local frequency window, effectively retaining important spectral features. At the same time, in the generation task, by reconstructing the masked part, the model's ability to discriminate local frequency correlation is further improved;

[0038] (3) The multi-task learning framework in the present invention utilizes different tasks to provide complementary information, which not only improves the generalization ability and robustness of the model, but also better captures the detailed features in the speech signal, thereby improving the performance of forged speech detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the forged voice detection method of the present invention;

[0040] Figure 2 The figure is a schematic diagram of the overall framework of the forged speech detection method of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0042] like Figure 1 As shown, the forged voice detection method in the embodiment of the present invention includes steps S10 to S30.

[0043] S10: preprocess the original audio data and extract primary features through a pre-trained model.

[0044] Specifically, the original audio data is downsampled to 16000 Hz. Since the durations of the sampled voices are different, each sampled voice is cut or copied and completed to obtain a voice with a fixed duration of 4 seconds.

[0045] The pre-trained model uses the speech recognition model Wav2Vec2-XLS-R to extract primary features; the speech recognition model Wav2Vec2-XLS-R is trained unsupervised by speech data in different real complex domains. The speech data in different real complex domains is a large-scale multilingual data set containing 436,000 hours and a total of 128 kinds of speech; the primary features extracted by this model can effectively improve the robustness of the forged speech detection model in different scenarios, facing different noises and encoding and decoding channels. After freezing the gradient of the feature extractor of the speech recognition model Wav2Vec2-XLS-R, the preprocessed audio data is input to obtain 1024-dimensional latent variables as primary features; among them, the feature extractor contains a 7-layer convolutional network and a 24-layer Transformer structure.

[0046] The original audio data comes from the ASVspoof2019LA dataset; the ASVspoof2019LA dataset uses a total of 107 speakers (61 females and 46 males), and consists of 12,456 real voices and 108,978 forged voices; the real voice data comes from the VCTK database, which contains the voices of speakers with a variety of accents and voice styles, and the forged voice data is generated by 11 speech synthesis methods and 8 speech conversion methods.

[0047] S20, inputting the primary features into the forged speech detection model for multi-task learning.

[0048] Figure 2 FIG. 4 is a schematic diagram showing the principle of a forged speech detection system based on multi-task learning of the present invention; Figure 2As shown, the forged speech detection model of the present invention includes a fully supervised forged speech detection classification branch and a self-supervised mask feature modeling branch, and each branch contains a local downsampling module. The primary features extracted by S10 are passed to the forged speech detection classification branch and the mask feature modeling branch respectively; in the forged speech detection classification branch, the primary features are first downsampled, then divided into blocks in the frequency dimension, and then a marker feature A with high local discriminability is extracted through a local downsampling module, and the marker feature A is connected with a learnable classification tag, that is, an initialized vector is attached to the front of the marker feature to transmit and aggregate information in the network; after the marker feature is connected with the classification tag, the position code is added, recorded as the marker feature B, and passed to the Conformer encoder and classifier in turn, and the probability distribution is output to judge the authenticity of the input speech, thereby achieving the goal of forged speech detection; in the mask feature modeling branch, the primary features are first downsampled, then divided into blocks in the frequency dimension and randomly masked, and then a marker feature C with high local discriminability is extracted through a local downsampling module. Finally, the primary features of the original input are reconstructed through the Conformer encoder and decoder, with the aim of exploring local correlations in the frequency domain.

[0049] Specifically, the shape of the primary feature is (4, 201, 1024), where 4 is the batch size, which refers to the number of samples passed by the model in one training iteration; 201 is the time step of the speech, and 1024 is the frequency dimension of the speech. Then, in order to retain the key information in each time window, the primary feature is processed by adaptive maximum pooling, and the time step is downsampled to 64. The feature (4, 64, 1024) is divided into 128 blocks in the frequency dimension, where each block contains 8 consecutive frequency frames; then it is sent to the local downsampling module, in which each block is downsampled using a shared CNN (convolutional neural network), retaining the key spectral information in the input feature and obtaining the tokenized block-level feature, i.e., the token feature, with a shape of (4, 128, 128). The kernel size of the shared CNN is 3, the padding is 1, and the stride is 1. In each convolution layer of the shared CNN, the shared CNN uses 3 times 2-fold average pooling on the time step dimension of the primary features after segmentation, and 6 times 2-fold average pooling on the frequency dimension of the primary features after segmentation. The last convolution layer passes through 128 filters, so that the output channel is 128.

[0050] The Conformer encoder combines the advantages of CNN (convolutional neural network) and Transformer to capture local detail features and model global dependencies. In the Conformer encoder, the labeled features B / C are first converted into high-dimensional embedding representations through a linear layer, then layer normalization is used to stabilize the training process, and finally sent to 6 Conformer Blocks. The Conformer Block consists of a feedforward network, a multi-head self-attention mechanism, a convolution module, a residual connection, and layer normalization; among them, the multi-head self-attention enhances the expressiveness of the model by introducing multiple attention heads, each of which can capture different features and dependencies, so that the model can focus on different parts of the input sequence at the same time.

[0051] The main idea of ​​the self-attention mechanism is to calculate the attention weight through query, key and value. Assume that the input sequence is Y∈R n×d , where n is the sequence length and d is the dimension of each input. Then, through linear transformation, we get the query matrix Q, key matrix K and value matrix V. The calculation process is as follows:

[0052] Q=YW Q , K=YW K , V=YW V

[0053] Among them, W Q , W K and W V is a trainable weight matrix;

[0054] Then the dot product is used to calculate the similarity between the query and the key and normalized by the Softmax function. The calculation process is as follows:

[0055]

[0056] Among them, A is the attention weight matrix, d k is the dimension of the key, used to scale the dot product;

[0057] Finally, the attention weights are used to perform weighted summation on the values ​​to obtain the attention output O = AV.

[0058] The multi-head attention mechanism calculates multiple attention heads in parallel and concatenates their outputs together. First, h query, key, and value matrices are obtained through different linear transformations. The calculation process is as follows:

[0059]

[0060] Then calculate the attention output for each attention head separately and get Oi =Attention(Q i , K i , V i ), and finally concatenate the outputs of all attention heads together and pass a linear transformation to get the final output O = [O1; O2; ...; O h ]W O , where W O is the linear transformation matrix after concatenation.

[0061] Finally, the multi-scale high-level features obtained by the Conformer encoder combine local and global information and can identify forgery traces at different frequency scales. The multi-scale high-level features containing classification tags are then input into the classifier to output the prediction results of forged speech detection and determine whether the input speech is real or forged.

[0062] In addition, the Conformer encoder in the self-supervised mask feature modeling branch shares the encoder parameters with the fully supervised forged speech detection branch. Specifically, the primary features after block division are randomly masked at a ratio of 0.5 in the frequency dimension, and then only the unmasked blocks are downsampled through the local downsampling module, and the marker features C of the downsampled unmasked blocks are embedded in the position encoding, and the position encoding of the unmasked parts is consistent with their absolute positions in the original blocks; after passing through the Conformer encoder, the high-level marker features of the unmasked blocks are obtained, with a shape of (4, 64, 128); finally, in order to reconstruct the original primary features, a shared and learnable mask marker is inserted into the high-level marker features of the unmasked blocks, and the masked parts are interpolated and repeated, and finally the primary features are reconstructed through the Conformer decoder. The Conformer decoder also contains 6 Conformer Blocks. Unlike the encoder, it uses multiple upsampling layers and linear layers after the Conformer Block to restore the original primary features.

[0063] Traditional self-supervised learning of masked spectrum modeling only reconstructs the masked part to reduce the computational cost of pre-training, and then fine-tunes the model for subsequent downstream tasks. However, only reconstructing the masked part will lead to insufficient feature learning. Therefore, in the present invention, multi-task learning (multi-tasks include classification tasks and generation tasks, corresponding to the forged speech detection classification branch and mask feature modeling branch, respectively) is used to simultaneously optimize the mask feature modeling task and the downstream forged speech detection task. This joint optimization method allows the model to learn features more comprehensively, so that it can perform both downstream tasks and mask feature modeling.

[0064] S30, jointly optimizing the classification loss and the reconstruction loss, and training the forged speech detection model until convergence.

[0065] The task of forged speech detection aims to distinguish whether the input speech is real speech or forged speech. This is a typical binary classification problem. The output of the model is usually a probability value between 0 and 1, indicating the possibility that the input speech is judged to be forged. Binary Cross-Entropy (BCE) is a loss function commonly used in binary classification problems. It can effectively process the probability value output by the model. Binary cross entropy has a large penalty for misclassification, especially for samples with high confidence but misclassification, which helps the model adjust the weights more accurately, thereby improving classification accuracy. Therefore, for the fully supervised forged speech detection classification branch, the binary cross entropy loss is used to measure the gap between the probability distribution predicted by the model and the true label. The formula is as follows:

[0066]

[0067] Where N is the number of samples in the original audio data, and f i represents the multi-scale high-level features extracted by the Conformer encoder, y i represents the label of the sample in the original audio data, and g represents the probability distribution extracted by the classifier.

[0068] For the self-supervised mask feature modeling branch, the mean square error loss is used to measure the gap between the original input and the reconstructed output, and its formula is as follows:

[0069]

[0070] Among them, X i represents the original input (primary features), Represents the reconstruction output (the reconstructed primary features).

[0071] Therefore, the overall loss function of the proposed forged speech detection method based on multi-task learning is as follows:

[0072] L=L BCE +λL MSE

[0073] Among them, λ is a hyperparameter that balances the classification loss and the reconstruction loss so that the model does not focus too much on the generation task and thus harms the performance of the classification task.

[0074] S40: Use the trained forged speech detection model to detect the audio data to be tested to identify whether it is forged speech.

[0075] In this embodiment, by locally downsampling the primary features, fine-grained frequency change information can be effectively extracted, thereby retaining the key spectral information in the input features and obtaining marker features with high local discriminability; in the forged speech detection classification branch, the Conformer encoder is used to extract multi-scale high-level features that combine local and global information, which can identify forgery traces at different frequency scales; in the self-supervised mask feature modeling branch, the frequency information of the masked block is reconstructed through the frequency information of the unmasked block, so as to learn the local frequency correlation between adjacent frequencies; finally, the mask feature modeling task and the downstream forged speech detection task are simultaneously optimized through multi-task learning, so that the model learns features more comprehensively, improves the generalization of the forged speech detection model, and improves the performance of forged speech detection.

[0076] The forged speech detection system based on multi-task learning includes:

[0077] The primary feature extraction module is used to preprocess the raw audio data and extract primary features through the pre-trained model;

[0078] The multi-task learning module inputs the extracted primary features into the forged speech detection model to complete the classification task and generation task;

[0079] The training module jointly optimizes the binary cross entropy loss in the classification task and the mean square error loss in the generation task to train the fake speech detection model;

[0080] The result identification module uses the trained forged speech detection model to detect the authenticity of the audio data to be tested.

[0081] The embodiment of the present application also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor of a computer device to implement the operations performed by the computer device in the forged voice detection method based on multi-task learning in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0082] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A forged speech detection method based on multi-task learning, characterized by: Preprocess the raw audio data and then extract primary features through the pre-trained model; The extracted primary features are input into the forged speech detection model to complete the classification and generation tasks: For classification tasks, the primary features are downsampled and then divided into frequency dimensions. The local downsampling module is used to extract the marker feature A with high local discriminability. The marker feature A is connected with a learnable classification marker and then the position code is added, which is recorded as the marker feature B. The marker feature B is sent to the Conformer encoder to obtain multi-scale high-level features. The multi-scale high-level features are sent to the classifier to determine whether it is real speech or forged speech. For the generation task, the primary features are downsampled, then divided into blocks in the frequency dimension and randomly masked with a ratio of γ. Only the unmasked blocks pass through the local downsampling module to obtain the marker features of the unmasked blocks. After passing through the Conformer encoder, the high-level marker features of the unmasked blocks are obtained. Repeat and interpolate a shared and learnable mask signature into the high-level signature features of the unmasked blocks, interpolate and repeat the masked parts, and finally get the reconstructed primary features through the Conformer decoder; Training the fake speech detection model, combining the binary cross entropy loss in the classification task and the mean square error loss in the generation task, until convergence; The trained forged speech detection model is used to detect the audio data to be tested to determine whether it is forged speech.

2. The forged speech detection method based on multi-task learning according to claim 1 is characterized in that: The pre-trained model uses the speech recognition model Wav2Vec2-XLS-R for primary feature extraction.

3. The forged speech detection method based on multi-task learning according to claim 2 is characterized in that: The speech recognition model Wav2Vec2-XLS-R is trained unsupervisedly using speech data from different real complex domains, where the speech data from different real complex domains is specifically a large-scale multilingual dataset containing 436,000 hours and a total of 128 types of speech.

4. The forged speech detection method based on multi-task learning according to claim 1 is characterized in that: The label feature A is connected with a learnable classification label, that is, an initialized vector is attached to the front of the label feature A for transmitting and aggregating information in the network.

5. The forged speech detection method based on multi-task learning according to claim 1 is characterized in that: The frequency dimension segmentation is specifically as follows: the primary features are processed by adaptive maximum pooling, the time step is downsampled, and then the frequency dimension is segmented.

6. The method for detecting forged speech based on multi-task learning according to claim 4, characterized in that: A shared convolutional neural network is used in the local downsampling module to downsample each block, retaining the key spectral information in the input features and obtaining tokenized block-level features, namely, labeled features.

7. The forged speech detection method based on multi-task learning according to claim 1 is characterized in that: The Conformer encoder parameters in the classification task and the generation task are shared.

8. The method for detecting forged speech based on multi-task learning according to claim 1, characterized in that: The binary cross entropy loss in the joint classification task and the mean square error loss in the generation task, where the binary cross entropy loss is: Where N is the number of samples in the original audio data, and f i represents the multi-scale high-level features extracted by the Conformer encoder, y i represents the label of the sample in the original audio data, and g represents the probability distribution extracted by the classifier; The mean square error loss is: Among them, X i Represents the primary characteristics, represents the primary features of the reconstruction; The overall loss function is expressed as follows: L=L BCE +λL MSE Among them, λ is a hyperparameter of the balance loss.

9. A system based on the forged speech detection method based on multi-task learning according to any one of claims 1 to 8, characterized in that: include: The primary feature extraction module is used to preprocess the raw audio data and extract primary features through the pre-trained model; The multi-task learning module inputs the extracted primary features into the forged speech detection model to complete the classification task and generation task; The training module jointly optimizes the binary cross entropy loss in the classification task and the mean square error loss in the generation task to train the fake speech detection model; The result identification module uses the trained forged speech detection model to detect the authenticity of the audio data to be tested.

10. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor and executes the forged speech detection method based on multi-task learning according to any one of claims 1 to 8.

Images