End-to-end deep counterfeit voice detection system and method

Abstract

The invention discloses an end-to-end deep counterfeit voice detection system and method, and belongs to the technical field of voice processing. The detection system is divided into a front-end embedded extraction network and a rear-end classification network. And the front-end embedded extraction network adopts a front-end pre-training model to extract global semantic features of different scales of the voice signals, and adopts a feature fusion module to perform deep fusion on the extracted global semantic features of different scales, so that the expression ability of feature embedding is further enhanced. And the rear-end classification network adopts a single-layer long-short-term memory classification network to classify the fusion features, performs joint optimization by using cross entropy loss and center loss, and finally judges whether the voice is real or forged. The method is applied to scenes of voice security, identity authentication, voice assistants and the like, a more accurate and robust deep counterfeit voice detection scheme with good generalization ability can be provided, and the method has important significance for improving the security and credibility of a voice recognition system.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to an end-to-end deep fake speech detection system and method. Background Art

[0002] Deep fake voice detection is the process of using advanced machine learning, signal processing and voice analysis technologies to identify fake voices generated by artificial intelligence. Fake voices can realistically imitate the voiceprint of a specific individual, and even have the intonation and expression of natural voices. The rapid development of fake voices has brought serious security risks to people's work and life. For example, attackers can use fake voices to simulate sounds, thereby implementing fund transfers and committing fraud; especially in recent years, the rapid development of voice conversion and speech synthesis technology based on deep learning has made deep fake voices more and more "realistic". Once abused without effective detection methods, the consequences will be very serious.

[0003] At present, research on forged speech detection has made some progress. Some traditional methods rely on manually extracted speech features (such as Mel-frequency cepstral coefficients, pitch, and time-frequency graphs) combined with machine learning classifiers to detect forged speech. However, when dealing with complex forgery techniques or diverse scenarios, the detection performance of these methods often fails to meet actual needs. In addition, deep learning methods are gradually introduced into the field of forged speech detection, among which models based on convolutional neural networks or recursive neural networks have achieved certain results, but their performance depends on the support of large-scale labeled data, and the generalization ability of the model still needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and propose an end-to-end deep fake voice detection system and method. By introducing a pre-trained model as a feature extractor, combined with a feature fusion module and a classification network, an end-to-end detection system is constructed. It aims to effectively identify a variety of deep fake voices and improve the accuracy and robustness of detection.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides an end-to-end deep fake voice detection system, comprising:

[0007] The front-end feature embedding extraction network is used to extract and fuse the global semantic features of the original speech signal at different scales to obtain the fused features;

[0008] A back-end classification network is used to make classification decisions on the fused features output by the front-end feature embedding and extraction network, and generate a prediction result of whether the original speech signal is a fake speech or a real speech;

[0009] The front-end feature embedding extraction network includes:

[0010] The front-end pre-training model is used to extract global semantic features of different scales of the original speech signal;

[0011] The feature fusion module is used to perform weighted fusion on the global semantic features of different scales extracted by the front-end pre-training model.

[0012] Preferably, the front-end feature embedding extraction network further includes:

[0013] The data preprocessing module is used to normalize the original speech signal, pad and trim the original speech signal to 4s, and obtain the sample value of the normalized original speech signal through the pytorch toolkit; the sample value is the amplitude of the normalized original speech signal at each moment.

[0014] Preferably, the front-end pre-training model adopts a fine-tuned self-supervised pre-training model Wav2Vec2.0.

[0015] Preferably, the front-end pre-training model includes a convolutional layer, a quantizer and at least one Transformer encoder connected in sequence.

[0016] The convolution layer is used to perform preliminary feature extraction on the sample values ​​extracted by the data preprocessing module to obtain low-level features; the convolution layer has 7 layers, and the convolution kernel sizes are 10, 3, 3, 3, 3, 2, 2, and the step sizes are 5, 2,2, 2, 2, 2, 2, respectively;

[0017] The quantizer is used to map the acquired low-level features into a discrete symbol space to obtain discretized features, and input them into the Transformer encoder;

[0018] The Transformer encoder is used to capture the temporal relationship and contextual information of the input features to obtain global semantic features;

[0019] Multiple Transformer encoders are set up in sequence to extract global semantic features of different scales.

[0020] Preferably, the feature fusion module adopts a convolutional attention module.

[0021] Preferably, the back-end classification network includes a single-layer bidirectional LSTM network, an average pooling layer and a linear layer.

[0022] The single-layer bidirectional LSTM network is used to extract the time series features in the input data through bidirectional transmission and generate a time series feature representation;

[0023] The average pooling layer is used to average the features of all time steps output by the single-layer bidirectional LSTM network to generate a feature vector of fixed dimension;

[0024] The linear layer is used to classify the input feature vector and output the prediction scores of the samples belonging to the two categories of real speech and fake speech.

[0025] Preferably, the backend classification network further includes:

[0026] The dimension transformation layer is used to adjust the dimension of the fused features output by the front-end feature embedding extraction network to adapt to the single-layer bidirectional LSTM network.

[0027] Preferably, the front-end feature embedding extraction network and the back-end classification network are pre-trained, and the pre-training process is as follows:

[0028] Obtain a training data set and a verification data set of speech signals;

[0029] In each round of training, the training set data is traversed and input into the front-end feature embedding extraction network to be trained in batches. The prediction results are output through the back-end classification network. The loss value is calculated based on the prediction results, the gradient is calculated through back propagation, and the network parameters are updated using the Adam optimizer. After the entire training data set is traversed, the average training loss of this round is calculated.

[0030] Traverse the validation data set, input it in batches into the front-end feature embedding extraction network after the network parameters are updated, output the prediction results through the back-end classification network, and calculate the loss value based on the prediction results; after completing the traversal of the entire validation data set, calculate the average validation loss of this round;

[0031] Continue the next round of training in the above manner until the preset number of training times is reached, save the network parameters with the lowest average verification loss, and obtain the pre-trained front-end feature embedding extraction network and back-end classification network;

[0032] During the training process, the weighted combination of cross entropy loss and center loss is used to calculate the loss value. The loss function is expressed as:

[0033] ,

[0034] in, is the loss function, is a hyperparameter, is the binary cross entropy loss, The center loss.

[0035] The present invention also provides an end-to-end deep fake voice detection method, which is implemented based on the above-mentioned end-to-end deep fake voice detection system, and the method includes:

[0036] Obtaining the original speech signal;

[0037] Using a front-end pre-trained model to extract features from the original speech signal, and obtaining global semantic features of the original speech signal at different scales;

[0038] The feature fusion module is used to perform weighted fusion of global semantic features of different scales to obtain fused features;

[0039] The back-end classification network is used to make classification decisions on the fused features and generate a prediction result of whether the original speech signal is a fake speech or a real speech.

[0040] Preferably, after obtaining the original voice signal, the method further includes:

[0041] The original speech signal is normalized, padded and trimmed to 4s, and the sample value of the normalized original speech signal is obtained through the pytorch toolkit; the sample value is the amplitude of the normalized original speech signal at each moment.

[0042] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0043] The present invention provides an end-to-end deep fake voice detection system and method, which significantly improves the performance of deep fake voice detection by introducing a pre-training model based on self-supervised learning and combining a feature fusion mechanism. The pre-training model can learn rich voice feature representations through its pre-training on a large-scale data set, so that the model has higher expression and recognition capabilities when processing complex voice data. By fusing the features of different levels in the pre-training model, the multi-level information in the voice signal can be effectively captured, further enhancing the model's ability to distinguish forged voices. Based on the above technical solution, the present invention can provide a more accurate, robust and well-generalized deep fake voice detection solution in application scenarios such as voice security, identity authentication, and voice assistants. The present invention can effectively prevent the abuse of forged voices, and has important practical significance for improving the security and credibility of voice recognition systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Shown is a schematic diagram of the structure of an end-to-end deep fake voice detection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0046] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0047] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0048] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection with an intermediate.

[0049] It should be emphasized here that the step marks mentioned below are not intended to limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0050] The first embodiment of the present invention provides an end-to-end deep fake speech detection system, including two parts: a front-end feature embedding extraction network and a back-end classification network.

[0051] The front-end feature embedding extraction network includes:

[0052] The data preprocessing module is used to normalize the original speech signal, uniformly pad and trim the original speech signal to about 4s, and then obtain the sample value of the normalized original speech signal through the pytorch toolkit; for speech with a sampling rate of 16kHz, there are 64,600 sample values. Each sample value corresponds to the amplitude of the audio signal at a certain moment. It should be noted that pytorch is an open source deep learning framework for machine learning and deep learning, released by Facebook in 2016.

[0053] The front-end pre-trained model is used to use the pre-trained model as a feature extractor, take the sample values ​​obtained by the data preprocessing module as input, extract the global semantic features of the original speech signal at different scales, and capture the fine-grained and contextual information in the speech.

[0054] The feature fusion module is used to perform weighted fusion of global semantic features of different scales extracted by the front-end pre-training model to obtain a richer speech feature embedding representation.

[0055] The back-end classification network is used to make classification decisions on the fused features output by the front-end feature embedding extraction network to determine whether the input speech is forged speech.

[0056] In an embodiment of the present invention, the front-end pre-training model adopts a fine-tuned self-supervised pre-training model Wav2Vec2.0. The self-supervised pre-training model Wav2Vec2.0 processes the input original speech signal and extracts a high-dimensional feature representation of the original speech signal. Through the fine-tuning mechanism, the pre-training model can better adapt to the detection requirements of forged speech in the task, significantly improving the accuracy and robustness of feature extraction. It should be noted that Wav2Vec2.0 is a work proposed by Facebook (META) on June 20, 2020, called a self-supervised learning framework for speech representation.

[0057] Specifically, see Figure 1 , the front-end pre-trained model includes sequentially connected convolutional layers, quantizers, and multiple Transformer encoders,

[0058] The convolution layer is used to perform preliminary feature extraction on the sample values ​​extracted by the data preprocessing module to obtain low-level features;

[0059] Quantizer, used to map the acquired low-level features into a discrete symbol space to obtain discretized features;

[0060] Transformer encoder is used to capture long-distance dependencies in the discretized features of the input and extract global semantic features. In this stage, the model further learns the temporal relationship and global context information in the audio signal;

[0061] In this embodiment, multiple Transformer encoders are arranged in sequence to extract global semantic features of different scales respectively.

[0062] Specifically, in this embodiment, there are 7 convolution layers, the convolution kernel sizes are 10, 3, 3, 3, 3, 2,2, and the step sizes are 5, 2, 2, 2, 2, 2, 2, 2, and the number of output channels of each convolution layer is 512. After the sample values ​​with a batch length of 64600 pass through the above convolution layers, the dimension of the output feature map is batch*201*512.

[0063] It should be noted that the Transformer encoder is composed of several stacked layers, each of which is usually composed of two main sub-modules: a multi-head self-attention mechanism and a feedforward neural network. The input and output feature dimensions of each layer are usually the same, but different operations are used to learn representations.

[0064] In this embodiment, the number of heads in the multi-head self-attention mechanism is 12, which means that each self-attention layer has 12 heads. Each head generates an attention matrix and performs calculations, and finally concatenates the outputs of each head; the hidden layer (hidden_size) dimension of each self-attention head is 768; the number of layers of the encoder stack (num_hidden_layers) is 12, which means there are 12 layers of encoders.

[0065] Through the above processing, the output of each layer of Transformer is batch*201*768. The output of the 12-layer stacked Transformer encoder is consistent with the output of each layer, and the final output feature map size is batch*201*768.

[0066] The front-end pre-trained model generates high-dimensional audio feature representations without relying on traditional manual feature engineering. These features gradually capture the detailed information of speech signals from shallow to deep layers to higher-level semantic representations. Shallow features focus on local speech signal patterns, while deep features extract more abstract speech features.

[0067] In an embodiment of the present invention, the feature fusion module is implemented using a convolutional block attention module (CBAM), which weights the input features by calculating channel attention and spatial attention. Specifically, the input features first pass through the channel attention module to generate channel weighting coefficients, and then adjust the input features by the channel weighting coefficients to obtain channel-weighted features. Next, the spatial weighting coefficients are calculated using the channel-weighted features, and finally the features are further adjusted by the spatial weighting coefficients. In this way, the feature fusion module can effectively enhance the attention to important features in the forged speech detection task and suppress irrelevant background information, as shown below:

[0068] ,

[0069] in, is the input feature, is the channel weighted feature, is the spatially weighted feature, is the channel weighting coefficient, is the spatial weighting coefficient.

[0070] Specifically, see Figure 1 ,The feature fusion module includes channel attention module and spatial attention module,

[0071] The data processing process of the channel attention module is:

[0072] The global semantic features of different scales output by the front-end pre-trained model are stacked into a multi-dimensional feature matrix as the input of the channel attention module. The features of each layer represent different levels of speech information. This process ensures that both shallow and deep feature information are retained, so that each layer of features can provide valuable input for subsequent detection tasks.

[0073] Global average pooling (GAP) and global maximum pooling (GMP) are performed on the input features to extract the global information of each channel. Through these two pooling operations, the feature vectors obtained reflect the average response and maximum response of each channel respectively. Then, the two pooling results are connected and input into a multi-layer perceptron to generate the weighting coefficients of each channel. Then, the channel weighting coefficients are applied to the input features by channel-by-channel multiplication to obtain the channel-weighted features. The channel-weighted features will highlight important channel information and suppress irrelevant channel information. The channel weighting coefficient calculation is expressed as follows:

[0074] ,

[0075] in, represents the input features, is the channel weighting coefficient, , , Indicates the number of channels, time dimension and feature dimension of the input features, represents the activation function, represents the global average pooling operation, represents the global maximum pooling operation, Represents the result of the global average pooling operation, Represents the result of the global maximum pooling operation, represents a multi-layer perceptron, Weight and Shared for both inputs.

[0076] The data processing process of the spatial attention module is as follows:

[0077] The spatial attention module extracts global information at each spatial position by performing maximum pooling and average pooling on the channel-weighted features in the channel dimension. Then, the two pooled feature maps are connected and the spatial weighting coefficients are obtained through convolution operations. The spatial weighting coefficients are applied to the channel-weighted features by position-by-position multiplication to obtain the final weighted features. This operation further emphasizes the important areas in space and suppresses the unimportant areas. The calculation of the spatial weighting coefficient is expressed as follows:

[0078] ,

[0079] in, , represents the feature aggregation graph obtained by average pooling and maximum pooling operations, The filter size is The convolution operation.

[0080] Through this fusion method, the features not only retain complete information from shallow to deep layers, but also, after weighted processing by CBAM, can highlight the features that are most critical for forged speech detection. This comprehensive and accurate feature embedding expression provides more abundant and discernible feature information for subsequent classification tasks, significantly improving the performance of deep forged speech detection.

[0081] In the embodiment of the present invention, the back-end classification network uses a single-layer long short-term memory (LSTM) classification network to classify the fused features to determine whether the input voice is a forged voice.

[0082] Specifically, see Figure 1 The backend classification network includes a single-layer bidirectional LSTM network, a mean pooling layer and a linear layer.

[0083] The data processing process of the backend classification network is as follows:

[0084] The fused features output by the front-end feature embedding extraction network are dimensionally adjusted to fit the back-end classification network. Specifically, the high-dimensional features extracted by the front-end feature embedding extraction network usually have high dimensions and complex structures. These features need to be dimensionally adjusted appropriately so that they can be effectively input into the subsequent classification network for processing.

[0085] The dimensionally adjusted features are input into a single-layer bidirectional LSTM network. LSTM processes the input timing information in the forward and backward propagation processes respectively, and the output dimension is (B, H, hidden_dim*2). Among them, hidden_dim is the hidden layer dimension of LSTM. After experimental comparison, hidden_dim=512 is taken here. *2 means that the output of the bidirectional LSTM is the concatenation of the forward and reverse outputs. B represents the batch size, and H is the time dimension. LSTM effectively extracts the timing features in the speech signal by transmitting information in both directions and generates a time-series feature representation.

[0086] On the output after LSTM processing, global pooling is performed in the time dimension, using the average pooling operation in the time step dimension. The purpose of this operation is to aggregate the feature information of all time steps into a vector of fixed dimension. The output shape of this average pooling operation is (batch, hidden_dim * 2), that is, the features of all time steps are averaged to obtain the global feature representation of each sample, and batch is the batch size.

[0087] These features are then input into a linear layer for classification, and the output dimension is (batch, 2), which indicates the predicted score of each sample belonging to the two categories of real speech and fake speech.

[0088] In the embodiment of the present invention, the front-end feature embedding extraction network and the back-end classification network are pre-trained jointly. The joint training process is as follows:

[0089] Obtain training data sets, verification data sets, and test data sets for speech signals: Specifically, download the processed data sets directly from the official resources. The data sets have been divided into training data sets, verification data sets, and test data sets according to task requirements. No additional processing is required. After downloading, they can be directly used for model training, verification, and testing;

[0090] Run the training data set: traverse the training data set and input the front-end feature embedding extraction network in batches. For each batch of data, the back-end classification network obtains the prediction result, calculates the loss value based on the prediction result, calculates the gradient through back propagation, and uses the optimizer (Adam) to update the network parameters. After completing the traversal of the entire training data set, calculate the average training loss of the batch;

[0091] Run the validation data set: traverse the validation data set, batch input the front-end feature embedding extraction network after the network parameters are updated; for each batch of data, obtain the prediction results through the back-end classification network, and calculate the loss value based on the prediction results. After completing the traversal of the entire validation data set, calculate the average validation loss of the batch;

[0092] Save the best model: During the training process, the loss value on the validation dataset is continuously monitored. After each training, the performance of the current model on the validation dataset is evaluated. If the loss value on the current validation dataset is lower than the previous optimal loss value, the current network parameters are saved as the best model until the maximum number of iterations is reached. The best model finally obtained is the model composed of the network parameters with the lowest loss value on the validation dataset.

[0093] Run the test data set: After saving the best model, evaluate the model on the test data set and calculate performance indicators such as Equal Error Rate (EER) and Minimum Detection Cost Function (min t-DCF). By analyzing the performance of performance indicators, the classification accuracy and generalization ability of the model are evaluated, thus confirming the effectiveness of the model in actual application scenarios.

[0094] Preferably, in the embodiment of the present invention, a weighted combination of cross entropy loss and center loss is used to calculate the loss, and the loss function is expressed as:

[0095] ,

[0096] in is the loss function, is a hyperparameter used to balance the weight of cross entropy loss and center loss.

[0097] is the binary cross entropy loss, which is used to measure the difference between the category probability distribution predicted by the model and the true category distribution. By minimizing the cross entropy loss, the model can better fit the training data and accurately classify real and fake speech.

[0098] It is the center loss, which is used to calculate the Euclidean distance between the feature of each sample and its category center. The formula is as follows:

[0099] ,

[0100] Indicates The features of the samples are the features of the second-to-last layer before the input fully connected layer; Indicates the feature center of the category to which it belongs. The categories here refer to the two categories of fake speech and real speech. Indicates the sample size.

[0101] The center loss is used to constrain the feature distribution of the same category, making the features of samples of the same category more compact, thereby enhancing the model's ability to distinguish.

[0102] Based on the above end-to-end deep fake voice detection system, a second embodiment of the present invention provides an end-to-end deep fake voice detection method, which includes:

[0103] Obtaining the original speech signal;

[0104] Use the front-end pre-trained model to extract features from the original speech signal and obtain global semantic features of the original speech signal at different scales;

[0105] The feature fusion module is used to perform weighted fusion of global semantic features of different scales to obtain fused features;

[0106] The back-end classification network is used to make classification decisions on the fused features and generate a prediction result of whether the original speech signal is a fake speech or a real speech.

[0107] It should be noted that after obtaining the original speech signal, the original speech signal is normalized by using the data preprocessing module, and the original speech signal is padded and trimmed to about 4s. The sample value of the normalized original speech signal is obtained through the pytorch toolkit. The sample value refers to the amplitude of the speech signal at a certain moment.

[0108] It should be noted that this method is implemented based on the above-mentioned deep fake voice detection system, and the specific data processing process of each step is consistent with the above, which will not be repeated here.

[0109] The end-to-end deep fake voice detection method provided by the present invention is verified in combination with specific experimental cases as follows:

[0110] The datasets used in the experiment are ASVspoof LA and ASVspoof DF datasets. The training dataset and verification dataset are both from LA2019, and the test dataset is from 21LA and 21DF. The sampling rate of all speech is 16kHz.

[0111] In order to verify the effectiveness of the method of the present invention, the following deep fake voice detection scheme was designed for comparison.

[0112] Solution 1: The Wav2Vec2.0 pre-trained model was used as the front-end feature extractor, but no fine-tuning was performed (W2V2(fixed) in Table 1). Then the LCNN (Lightweight Convolutional Neural Network) module was added to further extract features. In the back-end classification stage, BLSTM (Bi-directional Long Short-Term Memory) was used, and the loss function was cross entropy loss.

[0113] Solution 2: Similar to Solution 1, the only difference is that Wav2Vec2.0 is fine-tuned (W2V2 (finetuned) in Table 1).

[0114] Solution 3: The front-end features are extracted using Linear Frequency Cepstral Coefficients (LFCC), and the back-end classification stage adopts the enhanced time delay neural network (ECAPA-TDNN) module based on the channel attention mechanism, and uses the Softmax (OC-Softmax) loss function based on category center optimization.

[0115] Table 1 below shows the experimental results of the end-to-end deep fake voice detection method based on feature fusion proposed by the present invention and the above three solutions on the same data set. The last row in Table 1 is the end-to-end deep fake voice detection solution based on feature fusion proposed by the present invention.

[0116] Table 1 Comparison of experimental results of different deep fake voice detection methods

[0117] According to the analysis of experimental results, the method of the present invention achieved an EER of 3.42% on the 21LA dataset, showing a significant performance advantage. Compared with the comparison schemes 1 and 2, the present invention uses the pre-trained model as a feature embedding extractor and further optimizes the feature expression by introducing CBAM for feature fusion. In contrast, although schemes 1 and 2 also use pre-trained models, they do not use feature fusion, and the EERs are 10.97% and 7.18%, respectively. The present invention further reduces the EER to 3.42% by combining a fine-tuned pre-trained model and a feature fusion mechanism, which fully demonstrates the key role of the fusion strategy in improving model performance.

[0118] In addition, the EER of the third comparison scheme, which did not use the pre-trained model, was 5.46%. The present invention significantly reduced the EER to 3.42% by introducing the pre-trained model for feature extraction. This improvement shows that the pre-trained model can effectively improve the quality of feature extraction, enabling the system to more accurately capture the forged features in the voice signal, thereby significantly improving the performance of deep forged voice detection. The experimental results fully verify the importance of the Wav2Vec2.0 pre-trained model in this task.

[0119] This embodiment further conducts an ablation experiment. In the experiment, first, the last layer output features of the pre-trained model are directly used for classification; secondly, different feature fusion methods are used, including the convolutional attention module CBAM and the squeeze-excitation module SE to fuse the extracted features to improve the representation ability of the features. In addition, different classification networks, including LSTM and ECAPA-TDNN, are used to classify the features.

[0120] Finally, the experimental results of different fusion methods and different classification networks are compared and analyzed. The specific experimental results are shown in Table 2 below, which further proves the superior performance of the method of the present invention in the deep fake speech detection task.

[0121] Table 2 Experimental performance comparison of different fusion mechanisms and classification networks

[0122] From the experimental results, it can be seen that different feature fusion methods and classification networks have a significant impact on the final performance. The specific analysis is as follows:

[0123] Although using the last layer of features of the pre-trained model directly for classification (corresponding to the Last Layer+LSTM in Table 2) can achieve certain results, its performance is limited and it cannot fully explore the deep relationship between features. The use of feature fusion can effectively enhance the representation ability of features and significantly improve the performance of the model. This is because the fusion mechanism strengthens the weight of important features and suppresses irrelevant information through methods such as attention mechanism or channel compression, thereby obtaining a more discriminative feature representation.

[0124] Further comparison of the two feature fusion methods (corresponding to SE fusion + LSTM and CBAM fusion + LSTM in Table 2) shows that the performance of CBAM fusion is better than that of SE fusion. CBAM fusion can capture effective features more comprehensively and improve the feature distinguishing ability by considering the importance information of spatial dimension and channel dimension at the same time; while SE fusion only compresses and re-weights the channel dimension, and the feature enhancement effect is relatively insufficient.

[0125] In terms of the selection of classification networks (corresponding to CBAM fusion + ECAPA-TDNN and CBAM fusion + LSTM in Table 2), the experimental results show that the performance of the LSTM network is better than that of the ECAPA-TDNN network. As a recurrent neural network, LSTM can effectively capture long dependencies in the time dimension and is particularly suitable for modeling sequence data, which enables it to demonstrate stronger feature learning and classification capabilities in deep fake voice detection tasks. Although the ECAPA-TDNN network has strong feature extraction capabilities in its original design, in this experiment, its core module TDNN was removed and used only as a classifier, which may have led to its failure to fully exert its advantages in the classification task of fusion features.

[0126] In summary, the present invention uses a pre-trained model for feature extraction at the front end and combines it with CBAM for feature fusion, which significantly improves the model performance; in the back-end classification stage, an LSTM classification network is selected. Experimental verification shows that the overall design combining front-end feature embedding fusion and back-end classification can effectively improve the accuracy and generalization of forged voice detection.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An end-to-end deep fake voice detection system, characterized in that: include: The front-end feature embedding extraction network is used to extract and fuse the global semantic features of the original speech signal at different scales to obtain the fused features; A back-end classification network is used to make classification decisions on the fused features output by the front-end feature embedding and extraction network, and generate a prediction result of whether the original speech signal is a fake speech or a real speech; The front-end feature embedding extraction network includes: The front-end pre-training model is used to extract global semantic features of different scales of the original speech signal; The feature fusion module is used to perform weighted fusion on the global semantic features of different scales extracted by the front-end pre-training model.

2. An end-to-end deep fake voice detection system according to claim 1, characterized in that: The front-end feature embedding extraction network also includes: The data preprocessing module is used to normalize the original speech signal, pad and trim the original speech signal to 4s, and obtain the sample value of the normalized original speech signal through the pytorch toolkit; the sample value is the amplitude of the normalized original speech signal at each moment.

3. An end-to-end deep fake voice detection system according to claim 2, characterized in that: The front-end pre-training model adopts the fine-tuned self-supervised pre-training model Wav2Vec2.

0.

4. An end-to-end deep fake voice detection system according to claim 3, characterized in that: The front-end pre-training model includes a convolutional layer, a quantizer and at least one Transformer encoder connected in sequence. The convolution layer is used to perform preliminary feature extraction on the sample values ​​extracted by the data preprocessing module to obtain low-level features; the convolution layer has 7 layers, and the convolution kernel sizes are 10, 3, 3, 3, 3, 2, 2, and the step sizes are 5, 2, 2, 2, 2, 2, 2, respectively; The quantizer is used to map the acquired low-level features into a discrete symbol space to obtain discretized features, and input them into the Transformer encoder; The Transformer encoder is used to capture the temporal relationship and contextual information of the input features to obtain global semantic features; Multiple Transformer encoders are set up in sequence to extract global semantic features of different scales.

5. An end-to-end deep fake voice detection system according to claim 4, characterized in that: The feature fusion module adopts a convolutional attention module.

6. An end-to-end deep fake voice detection system according to claim 5, characterized in that: The backend classification network includes a single-layer bidirectional LSTM network, an average pooling layer and a linear layer. The single-layer bidirectional LSTM network is used to extract the time series features in the input data through bidirectional transmission and generate a time series feature representation; The average pooling layer is used to average the features of all time steps output by the single-layer bidirectional LSTM network to generate a feature vector of fixed dimension; The linear layer is used to classify the input feature vector and output the prediction scores of the samples belonging to the two categories of real speech and fake speech.

7. An end-to-end deep fake voice detection system according to claim 6, characterized in that: The backend classification network also includes: The dimension transformation layer is used to adjust the dimension of the fused features output by the front-end feature embedding extraction network to adapt to the single-layer bidirectional LSTM network.

8. An end-to-end deep fake voice detection system according to claim 6, characterized in that: The front-end feature embedding extraction network and the back-end classification network are pre-trained, and the pre-training process is as follows: Obtain a training data set and a verification data set of speech signals; In each round of training, the training set data is traversed and input into the front-end feature embedding extraction network to be trained in batches. The prediction results are output through the back-end classification network. The loss value is calculated based on the prediction results, the gradient is calculated through back propagation, and the network parameters are updated using the Adam optimizer. After the entire training data set is traversed, the average training loss of the current round is calculated. Traverse the validation data set, input it in batches into the front-end feature embedding extraction network after the network parameters are updated, output the prediction results through the back-end classification network, and calculate the loss value based on the prediction results; after completing the traversal of the entire validation data set, calculate the average validation loss of the current round; Continue the next round of training in the above manner until the preset number of training times is reached, save the network parameters with the lowest average verification loss, and obtain the pre-trained front-end feature embedding extraction network and back-end classification network; During the training process, the weighted combination of cross entropy loss and center loss is used to calculate the loss value. The loss function is expressed as: ， in, is the loss function, is a hyperparameter, is the binary cross entropy loss, The center loss.

9. An end-to-end deep fake voice detection method, characterized in that: Based on the end-to-end deep fake voice detection system according to any one of claims 1 to 8, the method comprises: Obtaining the original speech signal; Using a front-end pre-trained model to extract features from the original speech signal, and obtaining global semantic features of the original speech signal at different scales; The feature fusion module is used to perform weighted fusion of global semantic features of different scales to obtain fused features; The back-end classification network is used to make classification decisions on the fused features and generate a prediction result of whether the original speech signal is a fake speech or a real speech.

10. An end-to-end deep fake voice detection method according to claim 9, characterized in that: After obtaining the original voice signal, the method further includes: The original speech signal is normalized, padded and trimmed to 4s, and the sample value of the normalized original speech signal is obtained through the pytorch toolkit; the sample value is the amplitude of the normalized original speech signal at each moment.

Images