Method for detecting a fake voice based on a visual transformer

Abstract

The application relates to the technical field of voice detection, in particular to a fake voice detection method based on a visual Transformer, wherein the method comprises the following steps: fine-tuning a target pre-training model by using marked data to obtain a self-supervised learning front-end feature extractor, and inputting intermediate voice features of a voice to be detected into a preset rear-end classification network to cut the voice to be detected into a plurality of overlapping blocks of a target size for random dropout processing, and adding classification tokens and distillation tokens on the plurality of overlapping blocks after processing; inputting average values of the classification tokens and the distillation tokens into a classification layer of a Transformer encoder for classification prediction to obtain artifacts in the voice to be detected, and outputting a final voice detection result according to the artifacts. The application still has good detection performance under mismatched scenes that may be encountered in various actual applications, is high in robustness, and is good in interpretability.

Description

Technical Field

[0001] The present application relates to the field of speech detection technology, and in particular to a forged speech detection method based on visual Transformer. Background Art

[0002] Among related technologies, speech forgery technology based on deep neural networks poses a severe challenge to the credibility and authenticity of public opinion. Both human users and Automatic Speaker Verification (ASV) systems face high-risk security threats. Therefore, the development of forged speech detection technology is crucial. Mainstream forged speech detection research can be roughly divided into two categories: pipeline detection methods and end-to-end detection methods. The former usually includes a front-end feature extractor and a back-end classifier, and uses different data augmentation techniques for different scenarios; the latter, which has received increasing attention in recent years, uses a model to jointly optimize feature extraction and classification by directly operating on the original speech waveform.

[0003] However, the forged speech detection methods in related technologies perform well when facing in-domain data. When there are various mismatches between the data to be detected and the training data, such as forged speech generation algorithm mismatch, speech coding technology mismatch, low quality mismatch, and partial forgery mismatch, the detection performance of the model will drop sharply, and the robustness is poor and lacks interpretability. How to effectively detect out-of-domain speech data and various low-quality speech data and enhance the interpretability of the model to improve forged speech detection technology needs to be solved urgently. Summary of the Invention

[0004] This application provides a forged speech detection method based on visual Transformer to solve the problem that the forged speech detection method in the related art performs well when facing in-domain data, but when there are various mismatches between the data to be detected and the training data, such as forged speech generation algorithm mismatch, speech coding technology mismatch, low quality mismatch and partial forgery mismatch, the detection performance of the model will drop sharply, the robustness is poor and the interpretability is lacking. How to effectively detect out-of-domain speech data and various low-quality speech data and enhance the interpretability of the model to improve the forged speech detection technology.

[0005] The first aspect of the present application provides a method for detecting forged speech based on a visual Transformer, comprising the following steps: fine-tuning a target pre-trained model using labeled data to obtain a front-end feature extractor for self-supervised learning, and extracting intermediate speech features of the speech to be detected through the front-end feature extractor; inputting the intermediate speech features into a preset back-end classification network to divide them into multiple overlapping blocks of a target size, performing random dropout processing on the multiple overlapping blocks, and adding classification tokens and distillation tokens to the processed multiple overlapping blocks; inputting the average value of the classification tokens and distillation tokens of the added overlapping blocks into the classification layer of the Transformer encoder for classification prediction, obtaining artifacts in the speech to be detected, and outputting the final speech detection result based on the artifacts.

[0006] Optionally, in one embodiment of the present application, the labeled data includes real training data and forged training data in a preset downstream forged speech detection task.

[0007] Optionally, in one embodiment of the present application, it also includes: when using the labeled data to fine-tune the target pre-trained model, based on the labeled data, optimizing the target pre-trained model and the back-end classification network through a backpropagation strategy and a weighted cross entropy objective function.

[0008] Optionally, in one embodiment of the present application, the weighted cross entropy objective function is:

[0009]

[0010] Where N is the number of samples in each training batch, w1 is the weight of the real sample, w0 is the weight of the fake sample, and y i is the actual label, indicating the true category of the i-th sample, is the predicted output of the model, indicating the probability that the i-th sample belongs to category 1.

[0011] Optionally, in one embodiment of the present application, before inputting the average value of the classification token and the distillation token into the classification layer of the Transformer encoder for classification prediction, it also includes: performing linear projection processing on the overlapping blocks to obtain corresponding feature vectors; inputting the feature vectors into the initial Transformer encoder to feed at least one batch normalization layer, optimizing the classification layer of the Transformer encoder to generate the Transformer encoder.

[0012] The second aspect embodiment of the application provides a device for detecting fake speech based on visual Transformer, comprising: an extraction module configured to fine-tune a target pre-training model using labeled data to obtain a self-supervised learning front-end feature extractor, and extract intermediate speech features of a speech to be detected through the front-end feature extractor; a first processing module configured to input the intermediate speech features into a preset back-end classification network to split the intermediate speech features into a plurality of overlapping blocks of a target size, perform random dropout processing on the plurality of overlapping blocks, and add classification tokens and distillation tokens to the plurality of overlapping blocks after processing; and a detection module configured to input average values of the classification tokens and the distillation tokens of the overlapping blocks after adding to a classification layer of a Transformer encoder to perform classification prediction and obtain artifacts in the speech to be detected, and output a final speech detection result according to the artifacts.

[0013] Optionally, in an embodiment of the application, the labeled data includes real training data and fake training data in a preset downstream fake speech detection task.

[0014] Optionally, in an embodiment of the application, further comprising a first optimization module configured to optimize the target pre-training model and the back-end classification network based on the labeled data by using a back propagation strategy and a weighted cross-entropy objective function when fine-tuning the target pre-training model using the labeled data.

[0015] Optionally, in an embodiment of the application, the weighted cross-entropy objective function can be represented as:

[0016]

[0017] wherein N is the number of samples in each training batch, w1 is the weight of real samples, w0 is the weight of fake samples, y i is an actual label, indicating the real class of the i-th sample, is the prediction output of the model, indicating the probability that the i-th sample belongs to class 1.

[0018] Optionally, in an embodiment of the application, further comprising a second processing module configured to perform linear projection processing on the overlapping blocks to obtain corresponding feature vectors before inputting the average values of the classification tokens and the distillation tokens to the classification layer of the Transformer encoder for classification prediction; and a second optimization module configured to input the feature vectors to an initial Transformer encoder to feed at least one batch normalization layer, optimize the classification layer of the Transformer encoder, and generate the Transformer encoder.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the forged speech detection method based on visual Transformer as described in the above embodiment.

[0020] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned visual Transformer-based forged speech detection method.

[0021] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned visual Transformer-based forged speech detection method.

[0022] The embodiment of the present application can detect the speech to be detected by a front-end feature extractor based on self-supervised learning and a back-end classification network based on visual Transformer, and obtain a forged speech detection result of the speech to be detected. Thus, it is achieved that the pre-trained model of the sound event detection task is fine-tuned using the data set of the forged speech detection task, and finally an effective forged speech detection result is output. It still has good detection performance in mismatch scenarios that may be encountered in various practical applications, and has high robustness. In addition, the self-attention mechanism can be used to perform specific judgment analysis on the detection results of some forged speech, and has good interpretability. Thus, the forged speech detection method in the related art performs well when facing in-domain data. When there are various mismatches such as forged speech generation algorithm mismatch, speech coding technology mismatch, low quality mismatch and partial forged mismatch between the data to be detected and the training data, the detection performance of the model will drop sharply, the robustness is poor and there is a lack of interpretability. How to effectively detect out-of-domain speech data and various low-quality speech data and enhance the interpretability of the model to improve the forged speech detection technology and other issues.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 This is a schematic diagram of a framework for a robust forged speech detection technology based on a visual Transformer according to an embodiment of the present application;

[0026] Figure 2 This is a flowchart of a method for detecting forged speech based on a visual Transformer according to an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a typical forged voice detection system according to an embodiment of the present application is shown;

[0028] Figure 4 This is a schematic diagram of a framework of a front-end feature extractor technology based on self-supervised learning according to an embodiment of the present application;

[0029] Figure 5 This is a schematic diagram of the framework of a backend classification network based on a visual Transformer according to an embodiment of the present application;

[0030] Figure 6 This is a schematic diagram of the framework of the Transformer encoder and decoder according to one embodiment of the present application;

[0031] Figure 7 This is a schematic diagram comparing the attention map details of some forged speech in one embodiment of the present application;

[0032] Figure 8 Schematic diagram of the structure of a forged speech detection device based on a visual Transformer according to an embodiment of the present application;

[0033] Figure 9 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0034] Reference numerals:

[0035] 10-Forged speech detection device based on visual Transformer: 100-extraction module, 200-first processing module and 300-detection module; 901-memory, 902-processor and 903-communication interface. DETAILED DESCRIPTION

[0036] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the forged speech detection method based on a visual transformer in an embodiment of the present application with reference to the accompanying drawings. The forged speech detection method in the related art mentioned in the above background technology performs well when facing in-domain data. However, when there are various mismatches between the data to be detected and the training data, such as forged speech generation algorithm mismatch, speech coding technology mismatch, low-quality mismatch, and partial forgery mismatch, the detection performance of the model will drop sharply, and the robustness will be poor and the interpretability will be lacking. How to effectively detect out-of-domain speech data and various low-quality speech data and enhance the interpretability of the model to improve the forged speech detection technology? The present application provides a forged speech detection method based on a visual transformer, in which the speech to be detected can be detected by a front-end feature extractor based on self-supervised learning and a back-end classification network based on a visual transformer to obtain a forged speech detection result of the speech to be detected. As a result, the pre-trained model for the sound event detection task is fine-tuned using the dataset of the forged speech detection task, ultimately outputting effective forged speech detection results. This method still has good detection performance in mismatch scenarios that may be encountered in various practical applications, with high robustness. Furthermore, the self-attention mechanism can be used to perform specific judgment analysis on the detection results of some forged speech, which has good interpretability. This solves the problem that the forged speech detection method in the related art performs well when facing in-domain data, but when there are various mismatches between the data to be detected and the training data, such as forged speech generation algorithm mismatch, speech coding technology mismatch, low-quality mismatch, and partial forgery mismatch, the detection performance of the model will drop sharply, resulting in poor robustness and lack of interpretability. This also addresses the issue of how to effectively detect out-of-domain speech data and various low-quality speech data and enhance the interpretability of the model to improve forged speech detection technology.

[0038] Before explaining the forged voice detection method based on visual Transformer in the embodiment of the present application, the technical framework diagram of robust forged voice detection based on visual Transformer in the embodiment of the present application is first explained.

[0039] Figure 1 This is a schematic diagram of the framework of a robust forged speech detection technology based on visual Transformer according to an embodiment of the present application. Figure 1 As shown, the robust forged speech detection technology based on visual Transformer in the embodiment of the present application mainly but not limited to includes two parts: a front-end feature extractor based on self-supervised learning and a back-end classification network based on visual Transformer.

[0040] Specifically, Figure 2This is a flowchart of a method for detecting forged speech based on a visual Transformer provided in an embodiment of the present application.

[0041] like Figure 2 As shown in FIG, the forged speech detection method based on visual Transformer includes the following steps:

[0042] In step S201, the target pre-trained model is fine-tuned using labeled data to obtain a front-end feature extractor for self-supervised learning. The front-end feature extractor then extracts intermediate speech features of the speech to be detected. The labeled data includes both real and fabricated training data from a pre-defined downstream fabricated speech detection task.

[0043] It is understood by those skilled in the art that a typical voice forgery detection system usually includes a front-end feature extractor and a back-end classifier, and uses different data enhancement techniques for different scenarios, such as Figure 3 The figure shows a schematic diagram of the framework of a typical speech forgery detection system in one embodiment of the present application. The feature extractor can typically extract hand-designed acoustic features from the original speech waveform, such as constant-Q cepstral coefficients (CQCCs), linear-frequency cepstral coefficients (LFCCs), and Mel-scale short-time Fourier transform (MSTFT), all of which have shown excellent effectiveness in forged speech detection tasks.

[0044] Based on this, the embodiment of the present application can construct a front-end feature extractor based on self-supervised learning, which can fine-tune the target pre-trained model using labeled data to learn more robust intermediate speech features in the speech to be detected. Especially when facing forgery algorithms that have not appeared before, it can help related forgery algorithms reduce overfitting, thereby improving the generalization and robustness of the present application.

[0045] It should be noted that the target pre-training model here can be understood as an existing general pre-training model or a pre-training model selected by professional and technical personnel in this field according to specific circumstances or obtained by training on a large data set through simulation experiments. The embodiments of this application are only for illustrative purposes and are not specifically limited.

[0046] For example, the present application may, but is not limited to, use the Wav2vec 2.0 pre-trained model and perform fine-tuning to use it as a front-end feature extractor based on self-supervised learning, and finally extract the intermediate speech features of the speech to be detected through the front-end feature extractor based on sub-supervised learning.

[0047] Among them, the Wav2vec2.0 pre-trained model can be used to convert the original input waveform x 1:L Extract a series of feature representations o 1:N, where L is the number of samples and N is the number of features. Figure 4 This is a schematic diagram of the front-end feature extractor technology framework based on self-supervised learning in one embodiment of the present application. Figure 4 As shown in (a), the front-end feature extractor based on self-supervised learning consists of a convolutional neural network (CNN) and a Transformer network. Among them, the convolutional neural network can be used to transform the input x 1:L Convert to hidden feature sequence z 1:N , and the Transformer network can be used to transform z 1:N Convert to output sequence o 1:N The ratio between L and N can be determined by the CNN network step size (20ms).

[0048] Furthermore, the fine-tuning process is as follows Figure 4 As shown in (b), unlike the pre-training process, the fine-tuning process in the embodiment of the present application may not be performed on the hidden feature z 1:N Furthermore, the embodiment of the present application can output the Wav2vec 2.0 Transformer encoder. 1:N A fully connected layer is then added to reduce the final feature representation dimension to better match the input dimension of the backend classification network.

[0049] Since the pre-training process of the Wav2vec2.0 model only uses real data, the embodiments of the present application can, but are not limited to, fine-tune the model by using real training data and forged training data in certain downstream forged speech detection tasks as labeled data, thereby improving the performance of the model.

[0050] Optionally, in one embodiment of the present application, when fine-tuning the target pre-trained model using labeled data, the target pre-trained model and the backend classification network are optimized based on the labeled data using a backpropagation strategy and a weighted cross entropy objective function. The weighted cross entropy objective function can be expressed as:

[0051]

[0052] Wherein, N is the number of samples in each training batch, w1 is the weight of the real sample, which can be but not limited to 0.9 in the embodiment of the present application, and w0 is the weight of the forged sample, which can be but not limited to 0.1 in the embodiment of the present application, for class balance; y i is the actual label, indicating the true category of the i-th sample. In the embodiment of the present application, it can be, but is not limited to, 0 or 1. is the predicted output of the model, indicating the probability that the i-th sample belongs to category 1.

[0053] Those skilled in the art will appreciate that backend classifiers can typically determine whether audio is forged or authentic based on extracted features. For example, architectures such as Lightweight Convolutional Neural Networks (LCNN), Residual Networks (ResNet), RawNet2, and Graph Attention Networks (GAT) have demonstrated excellent performance in detecting forged speech.

[0054] In actual implementation, when fine-tuning the target pre-trained model using labeled data, this application can first freeze the parameters of the target pre-trained model, such as the wav2vec2.0 model, and then use the fully connected layer and back-end classification model (visual transformer) following wav2vec2.0 as trainable parameters. Before training begins, the parameters of wav2vec2.0 and visual transformer are initialized using the pre-trained model.

[0055] During the training process, the target pre-trained model and backend classification network can be optimized based on the labeled data through the back-propagation strategy and the weighted cross-entropy objective function. Among them, the weighted cross-entropy objective function is mainly used to minimize the training loss during the fine-tuning process and can be expressed as:

[0056]

[0057] Wherein, N is the number of samples in each training batch, w1 is the weight of the real sample, which can be but not limited to 0.9 in the embodiment of the present application, and w0 is the weight of the forged sample, which can be but not limited to 0.1 in the embodiment of the present application, for class balance; y i is the actual label, indicating the true category of the i-th sample. In the embodiment of the present application, it can be, but is not limited to, 0 or 1. is the predicted output of the model, indicating the probability that the i-th sample belongs to category 1.

[0058] In step S202, the intermediate speech features are input into a preset back-end classification network to divide them into multiple overlapping blocks of a target size, and random dropout processing is performed on the multiple overlapping blocks to add classification tokens and distillation tokens to the processed multiple overlapping blocks.

[0059] Based on the relevant descriptions of other embodiments, it can be understood that in the process of forged voice detection, this application not only utilizes a feature extractor based on self-supervised learning, but also a back-end classification network based on visual Transformer. Specifically, Figure 5 This is a schematic diagram of the framework of the back-end classification network based on the visual Transformer in one embodiment of the present application. Figure 5As shown, the embodiment of the present application can view the speech spectrogram as an image with a channel number of 1, a width of frequency bins, and a height of time frames. Since the speech spectrogram has two different dimensions, frequency and time, the embodiment of the present application can decompose the original position encoding into two dimensions, frequency and time.

[0060] In some embodiments, the main operating process of the back-end classification network based on the visual Transformer is as follows: After the input speech passes through the front-end feature extractor based on self-supervised learning, intermediate speech features are obtained. Their dimensionality is reduced through a fully connected layer to match the input dimension of the back-end classification network. After the intermediate speech features are fed into the back-end classification network, they are divided into overlapping blocks of fixed size and positional encodings are added to them in both the frequency and time dimensions. These overlapping blocks are then subjected to random dropout processing, and classification tokens and distillation tokens are added.

[0061] Among them, random dropout here can be understood as a regularization technique used to prevent overfitting of neural networks during training. In the visual Transformer, Dropout can be applied at multiple levels, including in the layers before or after the attention mechanism. Random Dropout sets the output of the Transformer's embedding or attention weights to zero with a certain probability (such as 0.1, 0.5, etc.), thereby training a different "sub-network" in each iteration. The main purpose is to reduce the impact of sequence length on the computational complexity of training the Transformer model. At the same time, discarding part of the input sequence during training can encourage the model to perform classification using incomplete sequences, which can effectively regularize the training process and enhance the generalization of the model.

[0062] Furthermore, in visual Transformers, a special classification token (class token) is often added to the beginning or end of the input sequence. The embedding of this classification token is propagated throughout the Transformer model and ultimately used to generate the output of the classification task. When performing speech detection, the final embedding of the classification token can be used as the basis for classification prediction.

[0063] The visual Transformer in the embodiment of the present application uses a target pre-training model for the sound event detection task. The target pre-training model can be implemented by, but is not limited to, a model distillation method, and a distillation token is added to the input sequence, wherein the prediction target of the distillation token is consistent with the output label of the teacher model. It should be noted that the method in the embodiment of the present application does not include a model distillation process, but still adds a distillation token, thereby achieving consistency between the visual Transformer and the target pre-training model and ensuring the effectiveness of the pre-training model. The embedding of the distillation token will also be propagated throughout the Transformer model and ultimately used to generate the output of the classification task.

[0064] It should be noted that specific implementation details such as the probability of Dropout and the initialization method of classification tokens may vary depending on different training tasks. They can be adjusted by professional and technical personnel in this field according to actual conditions or needs. This is only an illustrative explanation without specific limitations.

[0065] Optionally, in one embodiment of the present application, before inputting the average value of the classification token and the distillation token into the classification layer of the Transformer encoder for classification prediction, it also includes: performing linear projection processing on the overlapping blocks to obtain corresponding feature vectors; inputting the feature vectors into the initial Transformer encoder to feed at least one batch normalization layer, optimizing the classification layer of the Transformer encoder, and generating a Transformer encoder.

[0066] In other embodiments, after adding classification tokens and distillation tokens to overlapping blocks, the present application can also linearly project these overlapping blocks to feature vectors and input them into the Transformer encoder. Figure 6 This is a schematic diagram of the framework of the Transformer encoder and decoder of an embodiment of the present application. Figure 6 As shown, only the encoder structure is used in the embodiment of the present application, which contains a total of 12 basic blocks. Each block consists of two sub-layer connection structures. The first sub-layer connection structure includes a multi-head self-attention layer and a residual normalization layer, and the second sub-layer connection structure includes a feedforward network layer and a residual normalization layer. Each multi-head self-attention layer contains 12 self-attention heads. These self-attention heads will focus on different blocks in the input spectrogram during training, capture the relationship between them, and thus process long-distance dependent speech sequences.

[0067] Furthermore, after the corresponding feature vectors are input into the Transformer encoder, the high-level representations further extracted by the Transformer encoder are fed into a batch normalization layer (BN). These batch normalization layers can accelerate the training process of the neural network and improve the stability of the training process. Moreover, by normalizing each small batch of data, the distribution of the input of the network middle layer can be made more stable, thereby optimizing the classification layer of the Transformer encoder in the embodiment of the present application, generating a more stable Transformer encoder for subsequent use.

[0068] In step S203, the average value of the classification token and the distillation token of the added overlapping block is input into the classification layer of the Transformer encoder for classification prediction, so as to obtain the artifacts in the speech to be detected, and output the final speech detection result based on the artifacts.

[0069] As a possible implementation method, after adding classification tokens and distillation tokens to the overlapping blocks, the embodiment of the present application can input the average value of the classification tokens and distillation tokens into the classification layer for classification prediction, obtain the final detection result of the target pre-trained model for the speech to be detected and output it.

[0070] Additionally, the embodiments of the present application can also verify the final detection results.

[0071] For example, this application presents three types of experiments: performance analysis for intra-domain and speech codec mismatch scenarios, performance analysis for combined mismatch scenarios, and performance analysis for partial forgery mismatch scenarios. In each experiment, the performance of this application's embodiment is compared with that of other related forged speech detection methods, using the equal error rate (EER) to evaluate detection performance.

[0072] First, the experimental results under the intra-domain and voice codec mismatch scenarios are shown in Table 1, which is a table of experimental results under the intra-domain and voice codec mismatch scenarios according to an embodiment of the present application, as shown below:

[0073] Table 1

[0074]

[0075] As can be seen from Table 1, the embodiment of the present application achieved an EER of 0.19% on the in-domain dataset 2019LA, and achieved relatively advanced performance on the speech codec mismatch datasets (2021LA and 2021DF). Among them, when the data enhancement method is not used, the embodiment of the present application is 2.22% and 3.38% lower than the fake speech detection method in the related art on the 2021LA and 2021DF datasets in terms of EER (Equal Error Rate), respectively. When using the same data enhancement method, the embodiment of the present application is 0.14% and 1.20% lower than the best existing method on the 2021LA and 2021DF datasets in terms of EER, respectively. It can be seen that the embodiment of the present application has achieved better detection performance in the in-domain and speech codec mismatch scenarios, has good robustness, and can be combined with existing data enhancement methods to further improve detection performance.

[0076] Next, the experimental results under the combination mismatch scenario are shown in Table 2, which is an experimental result table under the combination mismatch scenario of an embodiment of the present application, and can be expressed as follows:

[0077] Table 2

[0078]

[0079] As can be seen from Table 2, the detection EER of the two compared baseline models on the In-the-Wild dataset in real scenarios is around 50%. This shows that the performance of the baseline model drops sharply when faced with mismatched data that is widely present in the real world. Its performance is basically equivalent to random guessing, and it cannot reliably distinguish between true and false categories, and its robustness is poor. The embodiment of the present application achieved an EER of 17.78% on the In-the-Wild dataset, and the other relevant best technical results in the comparison table reduced the EER by nearly 20%. This result proves that the embodiment of the present application can still maintain good robustness when faced with a variety of mismatch scenario combinations in the real world.

[0080] Next are the experimental results under the partial forgery mismatch scenario. Table 3 is an experimental result table under the partial forgery mismatch scenario of one embodiment of the present application, which can be expressed as follows:

[0081] Table 3

[0082]

[0083] As can be seen from Table 3, the detection EER of the two compared baseline models on the PartialSpoof evaluation dataset both reached more than 20%, while the embodiment of the present application can reach an EER of 13.57%, which improves the detection performance and shows good robustness against partial forgery mismatch scenarios.

[0084] Finally, the interpretability analysis of the detection results of the embodiments of this application is presented. Figure 7 This is a comparison diagram of the attention map details of two forged speech examples in one embodiment of the present application. Figure 7 As shown in the figure, the upper layer is the attention map, the middle layer is the segment labels, and the lower layer is the Mel-level spectrogram features. The upper layer's attention map is mapped using the 10th self-attention head in the 11th layer of the model's multi-head self-attention layer. Darker areas indicate greater attention paid by the model. The middle layer shows the segment labels of the speech. The resolution of these labels is 160ms. Both speech segments are 64,600 samples long (after cropping) with a sampling rate of 16,000. Therefore, the two speech segments are approximately 4,037ms long, with a total of 25 segment-level labels. Label "1" (blue) indicates that the segment is from real speech, while label "0" (red) indicates that the segment is from forged speech. The correspondence between the upper and middle layers shows that the 10th head of the multi-head self-attention layer pays more attention to the forged segments in the speech segments, providing the primary basis for the model to infer that the speech is forged. It should be noted that the relationship shown in the figure does not correspond exactly. This is mainly because STFT transformation or convolution operations are performed when extracting features. Both operations cause overlap between time frames, while segment labels do not overlap at all. In summary, the forged speech detection method proposed in the embodiment of the present application can use the self-attention mechanism to perform specific judgment analysis on the detection results of some forged speech, and has good interpretability.

[0085] According to the forged voice detection method based on visual transformer proposed in the embodiment of the present application, the voice to be detected can be detected by a front-end feature extractor based on self-supervised learning and a back-end classification network based on visual transformer, and a forged voice detection result of the voice to be detected can be obtained. Thus, it is achieved that the pre-trained model of the sound event detection task is fine-tuned using the data set of the forged voice detection task, and finally an effective forged voice detection result is output. It still has good detection performance in mismatch scenarios that may be encountered in various practical applications, and has high robustness. In addition, the self-attention mechanism can be used to perform specific judgment analysis on the detection results of some forged voices, and has good interpretability. Thus, the forged voice detection method in the related art performs well when facing in-domain data. When there are various mismatches such as forged voice generation algorithm mismatch, voice coding technology mismatch, low quality mismatch and partial forged mismatch between the data to be detected and the training data, the detection performance of the model will drop sharply, the robustness is poor and there is a lack of interpretability. How to effectively detect out-of-domain voice data and various low-quality voice data and enhance the interpretability of the model to improve the forged voice detection technology and other issues.

[0086] Next, a forged speech detection device based on a visual Transformer according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0087] Figure 8 2 is a schematic diagram of the structure of a forged speech detection device based on a visual Transformer according to an embodiment of the present application.

[0088] like Figure 8 As shown, the forged speech detection device 10 based on visual Transformer includes: an extraction module 100 , a first processing module 200 and a detection module 300 .

[0089] Among them, the extraction module 100 is used to fine-tune the target pre-training model using labeled data to obtain a front-end feature extractor for self-supervised learning, and extract the intermediate speech features of the speech to be detected through the front-end feature extractor.

[0090] The first processing module 200 is used to input the intermediate speech features into a preset back-end classification network to divide them into multiple overlapping blocks of a target size, perform random dropout processing on the multiple overlapping blocks, and add classification tokens and distillation tokens to the processed multiple overlapping blocks.

[0091] The detection module 300 is used to input the average value of the classification token and the distillation token of the added overlapping block into the classification layer of the Transformer encoder for classification prediction, obtain the artifacts in the speech to be detected, and output the final speech detection result based on the artifacts.

[0092] Optionally, in one embodiment of the present application, the labeled data includes real training data and forged training data in a preset downstream forged speech detection task.

[0093] Optionally, in one embodiment of the present application, it further includes: a first optimization module.

[0094] Among them, the first optimization module is used to optimize the target pre-training model and the back-end classification network based on the labeled data through the back propagation strategy and the weighted cross entropy objective function when fine-tuning the target pre-training model using labeled data.

[0095] Optionally, in one embodiment of the present application, the weighted cross entropy objective function can be expressed as:

[0096]

[0097] Where N is the number of samples in each training batch, w1 is the weight of the real sample, w0 is the weight of the fake sample, and y i is the actual label, indicating the true category of the i-th sample, is the predicted output of the model, indicating the probability that the i-th sample belongs to category 1.

[0098] Optionally, in one embodiment of the present application, it further includes: a second processing module and a second optimization module.

[0099] The second processing module is used to perform linear projection processing on the overlapping blocks to obtain the corresponding feature vector before inputting the average value of the classification token and the distillation token into the classification layer of the Transformer encoder for classification prediction;

[0100] A second optimization module is used to input the feature vector into the initial Transformer encoder to feed it into at least one batch normalization layer, and optimize the classification layer of the Transformer encoder to generate a Transformer encoder.

[0101] It should be noted that the aforementioned explanation of the embodiment of the forged voice detection method based on visual Transformer is also applicable to the forged voice detection device based on visual Transformer in this embodiment, and will not be repeated here.

[0102] According to the forged voice detection device based on visual transformer proposed in the embodiment of the present application, the voice to be detected can be detected by a front-end feature extractor based on self-supervised learning and a back-end classification network based on visual transformer, and a forged voice detection result of the voice to be detected can be obtained. Thus, it is achieved that the pre-trained model of the sound event detection task is fine-tuned using the data set of the forged voice detection task, and finally an effective forged voice detection result is output. It still has good detection performance in mismatch scenarios that may be encountered in various practical applications, with high robustness, and can use the self-attention mechanism to perform specific judgment analysis on the detection results of some forged voices, with good interpretability. Thus, the forged voice detection method in the related art performs well when facing in-domain data. When there are various mismatches such as forged voice generation algorithm mismatch, voice coding technology mismatch, low quality mismatch and partial forged mismatch between the data to be detected and the training data, the detection performance of the model will drop sharply, the robustness is poor and the interpretability is lacking. How to effectively detect out-of-domain voice data and various low-quality voice data and enhance the interpretability of the model to improve the forged voice detection technology and other issues.

[0103] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0104] A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .

[0105] When the processor 902 executes the program, the forged voice detection method based on visual Transformer provided in the above embodiment is implemented.

[0106] Furthermore, the electronic device further includes:

[0107] The communication interface 903 is used for communication between the memory 901 and the processor 902 .

[0108] The memory 901 is used to store computer programs that can be run on the processor 902 .

[0109] The memory 901 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0110] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the communication interface 903, the memory 901 and the processor 902 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 9 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0111] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can complete communication between each other through an internal interface.

[0112] The processor 902 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0113] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the above-mentioned method for detecting a fake voice based on a visual Transformer.

[0114] The embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program can run computer instructions, and the computer instructions are executed by a processor to implement the method for detecting a fake voice based on a visual Transformer provided by the embodiment of the present application.

[0115] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0116] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0117] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0118] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0119] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are each well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0120] Those of skill in the art would understand that the steps of the methods carried out above can be carried out wholly or partly by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and when executed, includes one or a combination of the steps of the method embodiments.

[0121] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0122] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A forged speech detection method based on visual Transformer, characterized in that: The following steps are involved: Fine-tune the target pre-trained model using the labeled data to obtain a front-end feature extractor for self-supervised learning, and extract intermediate speech features of the speech to be detected through the front-end feature extractor; Inputting the intermediate speech features into a preset backend classification network to divide them into multiple overlapping blocks of a target size, performing random dropout processing on the multiple overlapping blocks, and adding classification tokens and distillation tokens to the processed multiple overlapping blocks; Inputting the average value of the classification token and the distillation token of the added overlapping block into the classification layer of the Transformer encoder for classification prediction, obtaining the artifact in the speech to be detected, and outputting the final speech detection result based on the artifact; The labeled data includes real training data and forged training data in a preset downstream forged speech detection task; When fine-tuning the target pre-trained model using the labeled data, the target pre-trained model and the backend classification network are optimized based on the labeled data through a back-propagation strategy and a weighted cross-entropy objective function.

2. The method according to claim 1, characterized in that The weighted cross entropy objective function is: ， in, is the number of samples in each training batch, is the weight of the real sample, is the weight of the fake sample, is the actual label, indicating the The true category of the samples, is the predicted output of the model, indicating the The probability that a sample belongs to class 1.

3. The method according to claim 1, characterized in that Before inputting the average of the classification token and the distillation token into the classification layer of the Transformer encoder for classification prediction, it also includes: Performing linear projection processing on the overlapping blocks to obtain corresponding feature vectors; The feature vector is input into an initial Transformer encoder to be fed into at least one batch normalization layer, and a classification layer of the Transformer encoder is optimized to generate the Transformer encoder.

4. A forged speech detection device based on visual Transformer, characterized in that: include: An extraction module, configured to fine-tune the target pre-trained model using the labeled data to obtain a front-end feature extractor for self-supervised learning, and extract intermediate speech features of the speech to be detected through the front-end feature extractor; a processing module, configured to input the intermediate speech features into a preset backend classification network to divide the intermediate speech features into a plurality of overlapping blocks of a target size, perform random dropout processing on the plurality of overlapping blocks, and add classification tokens and distillation tokens to the processed plurality of overlapping blocks; A detection module, configured to input the average of the classification tokens and the distillation tokens of the added overlapping blocks into the classification layer of the Transformer encoder for classification prediction, obtain artifacts in the speech to be detected, and output a final speech detection result based on the artifacts; The labeled data includes real training data and forged training data in a preset downstream forged speech detection task; The first optimization module is used to optimize the target pre-training model and the backend classification network based on the labeled data through a back propagation strategy and a weighted cross entropy objective function when fine-tuning the target pre-training model using the labeled data.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the forged speech detection method based on visual Transformer according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the forged speech detection method based on visual Transformer according to any one of claims 1 to 3.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed, it is used to implement the forged speech detection method based on visual Transformer according to any one of claims 1 to 3.

Images