Pseudo voice detection method based on RawNet2 model and Res2Net model

Abstract

The invention discloses a pseudo voice detection method based on a RawNet2 model and a Res2Net model, is used for improving the accuracy and robustness of pseudo voice detection, and belongs to the technical field of voiceprint fraud detection. The method comprises the following steps: firstly, preparing original data, and then, carrying out the following steps: step 1, firstly, extracting manual features from preprocessed voice signals, then inputting the manual features of each voice into a Res2Net model, and finally, outputting a score of each voice signal by the Res2Net model; step 2, inputting the voice signals into a RawNet2 model, then performing end-to-end processing on the input voice signals by the RawNet2 model, and outputting a score of each voice signal at the same time; step 3, data normalization: carrying out normalization processing on scores of the RawNet2 model and the Res2Net model; 4, performing score fusion, and outputting the score of each voice of the fusion model; and step 5, EER calculation: calculating the EER of the fusion model.

Description

Technical Field

[0001] The present invention belongs to the technical field of voiceprint fraud detection, and specifically is a pseudo-voice detection method based on a RawNet2 model and a Res2Net model. Background Art

[0002] The Gaussian Mixture Model-Universal Background Model (GMM-UBM) was proposed in 2000. It is trained with a large-scale speaker corpus, so it can describe the common characteristics of speakers. However, since it is trained from mixed data of a large number of different speakers, it does not have the ability to represent the identity of a specific speaker. Then, using UBM as the initial model, the target speaker data is used for adaptive training based on the maximum a posteriori probability (MAP), and the Gaussian mixture model (GMM) of the target speaker is obtained after fine-tuning the parameters. Speakers are scored by calculating the log-likelihood value, and then recognition decisions are made.

[0003] Joint Factor Analysis (JFA): Due to the mismatch between the training environment and the test environment, the system performance of the GMM-UBM model is unstable. The JFA model assumes that the speaker space and the channel space are independent and uncorrelated, and models the speaker space, channel space, and residual noise separately, thereby obtaining a more accurate description of the speaker's characteristics.

[0004] i-vector method: Proposed in 2011, it is essentially an improvement on the JFA model. A space is used to replace the speaker space defined by the eigenvoice space matrix and the channel space defined by the eigenchannel space matrix. This space contains both differences between speakers and differences between channels. In the test phase, the consine distance between the i-vector of the test speech and the i-vector of the model is calculated, which is the final score. The introduction of i-vector simplifies the research on speaker recognition into a problem of numerical analysis and data analysis: any audio segment of different lengths and contents will eventually be mapped to a low-dimensional fixed-length i-vector.

[0005] d-vector method: In the back-end classification stage, the DNN is trained to classify speakers at the frame level. In the input stage of the speaker corpus, the trained DNN is used to extract speech features from the hidden layer, and these features are merged and averaged to obtain a sentence-level representation. This utterance-level representation is the deep speaker vector, or d-vector for short. In the evaluation stage, the cosine distance between the d-vector extracted from each utterance and the recorded speaker model d-vector is calculated, and finally a judgment is made and scored. The DNN-based system shows robust performance to added noise and outperforms the previous i-vector system in terms of false rejection rate.

[0006] x-vector method: It can accept speech input of any length and convert variable-length speech into fixed-length feature expressions. When calculating speech similarity, the classic probabilistic linear discriminant analysis (PLDA) algorithm is used. At the same time, data enhancement strategies including noise and reverberation are introduced during model training to increase the robustness of the model.

[0007] Problems and shortcomings: (1) High requirements for voice signal quality: Traditional signal processing methods are very sensitive to the quality of voice signals and are easily affected by factors such as environmental noise and channel changes.

[0008] (2) Difficulty in dealing with complex forgery methods: With the continuous development of voice forgery technology, especially the emergence of deep forgery technology, traditional signal processing methods are unable to cope with complex forged voices.

[0009] (3) Feature design relies on experience: Feature design in traditional methods often relies on the experience of researchers and lacks automated feature extraction and selection mechanisms.

[0010] (4) High data dependence: A large amount of training data is required. The performance of machine learning models is highly dependent on the diversity and quality of training data. However, fake speech data is often difficult to obtain. Machine learning models require a large amount of labeled data for training, but in the field of fake speech detection, high-quality labeled data is often difficult to obtain. The labeling process is time-consuming and laborious, and there may be subjectivity and inconsistency, and the labeling cost is high.

[0011] (5) Poor generalization ability: When faced with new counterfeiting techniques, machine learning models tend to overfit the training data, resulting in poor performance on unseen test data. Improving the generalization ability of the model is an urgent problem to be solved, which leads to limited generalization ability.

[0012] (6) High consumption of computing resources: Although the performance of machine learning models is better than traditional signal processing methods, their training and inference processes usually require a lot of computing resources.

[0013] (7) Using a single voiceprint fraud detection model for pseudo-speech detection, the model's detection results have a high error rate. Summary of the invention

[0014] The object of the present invention is to provide a pseudo speech detection method based on the RawNet2 model and the Res2Net model, so as to improve the accuracy and robustness of pseudo speech detection.

[0015] The present invention is achieved by adopting the following technical solutions: A pseudo-speech detection method based on the RawNet2 model and the Res2Net model first prepares the raw data and then performs the following steps: Step 1: First, extract manual features from the preprocessed speech signal, then input the manual features of each speech into the Res2Net model, and finally the Res2Net model outputs the score of each speech signal. It is divided into the following steps: S11, data preprocessing; S12, feature extraction, Extracting manual features; S13, model training; S14, model scoring, The model scores each speech.

[0016] Step 2: First, input the speech signal into the RawNet2 model, and then the RawNet2 model processes the input speech signal end-to-end and outputs the score of each speech signal. It is divided into the following steps: S21, data preprocessing; S22, feature extraction and scoring, The RawNet2 model processes the input speech signal end-to-end and scores each speech.

[0017] Step 3: Data normalization. Normalize the scores of the RawNet2 model and the Res2Net model.

[0018] Step 4: score fusion, output the score of each speech in the fusion model.

[0019] Step 5: EER calculation: calculate the EER of the fusion model.

[0020] Further preferably, the S11 data preprocessing method is to divide the voice file into real voice and spoofed voice, and the metadata file into protocol file and CM protocol file, and generate spk2utt file, utt2index file, utt2spk file and utt2systemID through the protocol file.

[0021] Further preferably, S21, the data preprocessing method is to organize each voice data into a file containing speaker_id, file_name, path, sys_id and key, and save the preprocessed voice data into a .npy file.

[0022] Further preferably, in step one, the manual feature types extracted from the speech signal include cqt, spec, and mfcc, and the manual features are extracted by inputting corresponding codes into the Python interpreter. Before extracting the manual features, the input speech signal is pre-emphasized, framed, and windowed.

[0023] Further preferably, in step three, the scores of the RawNet2 model and the Res2Net model are mapped to the interval (0, 1), and the data normalization method is maximum and minimum normalization, sigmoid change or inverse tangent function normalization.

[0024] Further preferably, the three methods of data normalization are used to perform calculations respectively, and after obtaining the normalized data scores, the EERs of the original data and the normalized data are calculated respectively, and the data normalization method with the closest results of the original data EER and the normalized data EER is taken as the final data normalization algorithm.

[0025] Further preferably, in step 4, Z i =β1X i +β2Y i Fusion of the Res2Net model and the RawNet2 model scores.

[0026] The score of each voice in the RawNet2 model is X i , the score of each speech in the Res2Net model is Y i , assign weights of β1 and β2 to the two models respectively, and β1+β2=1.

[0027] Further preferably, step one further comprises S15, calculating the EER of the model, the EER calculated by the models under different parameters are different, and the model with the lowest EER is selected to proceed to step three.

[0028] Further preferably, step 2 further comprises S23, calculating the EER of the model, the EER calculated by the models under different parameters is different, and the model with the lowest EER is selected to proceed to step 3.

[0029] Compared with the prior art, the present invention has the following beneficial effects: 1. Complementary features: The RawNet2 model is good at directly extracting low-dimensional embedded features from raw speech signals, while the Res2Net model is known for its residual learning and multi-scale feature fusion mechanism. By combining the two, the features can be complemented and the model's comprehensive representation ability of speech signals can be improved.

[0030] 2. Performance improvement: The fusion model can utilize the original signal processing capabilities of the RawNet2 model and the residual learning and multi-scale feature fusion advantages of the Res2Net model, showing higher accuracy and robustness in the pseudo-speech detection task.

[0031] 3. Enhance generalization ability: Through model fusion, we can combine the advantages of multiple models, enhance the model's defense capabilities against unknown attack methods, and improve the generalization performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 Represents the network architecture diagram of the ResNet50 model.

[0035] Figure 2 Schematic diagram showing the comparison of BottleNeckblock in ResNet and Res2Net.

[0036] Figure 3 A network architecture diagram showing BTNK1 and BTNK2. BTNK stands for BottleNeckblock.

[0037] Figure 4 A diagram showing the network structure of a residual block.

[0038] Figure 5 Represents the RawNet2 architecture for voiceprint fraud detection.

[0039] Figure 6 Represents the preliminary feature extraction logic diagram.

[0040] Figure 7 Represents the algorithm implementation logic diagram.

[0041] Figure 8 Represents the voiceprint fraud detection process. DETAILED DESCRIPTION

[0042] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0044] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0045] With the improvement and breakthrough of computer hardware resources, voiceprint fraud detection algorithms based on deep learning have become increasingly popular. The two algorithm models used below, Res2Net model and RawNet2 model, are both voiceprint fraud detection algorithms based on deep learning.

[0046] Most existing speaker recognition algorithms require manually extracted features in the feature extraction stage, which is greatly affected by prior knowledge and has strong limitations. The end-to-end recognition system combines the feature extraction and evaluation stages into one, and uses deep neural networks to autonomously learn deep speech features, which greatly avoids the influence of manual features.

[0047] The present invention uses deep learning technology to analyze sound features, identify the voices of different individuals, analyze the voice content, identify abnormal patterns or features therein, and detect the difference between the sound produced by speech synthesis and the real sound. It can help identify fraudulent activities such as telephone fraud and fake customer service, protect communication security, protect users from financial losses, and enhance the security of accounts and devices.

[0048] In the first embodiment, raw data preparation is first performed: a large number of real speech and forged speech data sets are obtained, and the diversity and representativeness of the data sets are ensured.

[0049] Then manual feature extraction is performed: features that can be distinguished from real speech are extracted from the fake speech and input into the back-end for judgment, such as constructing features by taking advantage of the fact that fake speech usually lacks spectral and temporal detail information.

[0050] We use three types of feature extraction: mfcc, cqt and spec. The common operations in the process are pre-emphasis, framing and windowing, which convert the original speech signal into a feature vector representation.

[0051] Pre-emphasis: Pre-emphasis refers to compensating the high-frequency components in the input signal at the transmitting end. During the transmission of audio, the higher the rate of the audio signal, the more susceptible it is to damage and attenuation, which may cause the resonance peaks of the phonemes to be unclear and weaken the acoustic model's ability to model these phonemes. The main idea of ​​pre-emphasis is to pass the signal through a first-order high-pass filter, thereby increasing the energy of the high-frequency part of the speech signal, but pre-emphasis will not affect the noise, so it can effectively improve the output signal-to-noise ratio.

[0052] Framing: The speech signal is a non-steady-state, time-varying signal, and it is easily affected by many factors during its formation, including the external environment and the human vocal organs. However, it can be roughly considered that the speech signal is time-invariant and steady within a short time range, and this short time is generally 10-30ms. Therefore, when processing the speech signal, in order to reduce the impact of the overall non-steady-state and time-varying speech signal, the speech signal can be segmented, where each segment is defined as a frame, and the frame length is generally 25ms. In order to make the transition between adjacent speech frames smooth, when framing, the overlapping segmentation method can be used to make the tail of each frame partially overlap with the head of the next frame.

[0053] Windowing: The purpose of windowing the framed speech is to reduce the edge effect of the speech frame so that the signals at both ends of the speech frame can smoothly fade to 0. When selecting the window function, both the length and shape of the window function need to be considered. The selected window should change slowly and the window edge shape should be smooth to reduce the truncation effect on the speech frame. In the field of voiceprint recognition, the Hamming window is generally used as the window function.

[0054] The types of manual features extracted are as follows: cqt, uses a set of filters with constant Q value to filter the signal. The center frequency of the filter group is based on log2 and is distributed according to the exponential law. The ratio of the center frequency to the filter bandwidth is a constant Q. The filter bandwidth changes accordingly according to the signal frequency, thereby achieving better performance.

[0055] spec, Fast Fourier Transform: A general term for a class of methods for quickly calculating DFT (Discrete Fourier Transform), which is essentially DFT. Its main function is to convert speech signals from time domain to frequency domain, so as to further study the changing laws and spectral structure of speech signals. This method can reduce the number of multiplications required when calculating discrete Fourier transform. The larger the number of sampling points N, the more obvious the amount of calculation saved. FFT (Fast Fourier Transform) is used to convert speech signals from time domain to frequency domain, so as to further study the changing laws and spectral structure of speech signals. spec (spectrogram) is a three-dimensional spectrum that can simultaneously display the changes of speech spectrum over time in time and frequency, which is very beneficial for analyzing the time-frequency characteristics of speech signals.

[0056] mfcc, Mel filtering: The spectrum is passed through a Mel-scale triangular bandpass filter group, the purpose of which is to smooth the spectrum, eliminate the influence of harmonics, and highlight the resonance peaks of the speech. Discrete Cosine Transform (DCT): Data compression of speech. The advantages are that the signal spectrum has concentrated energy, rich components, and no need to estimate the speech phase, which can achieve better speech enhancement effects at a lower computational complexity. mfcc is a representative of amplitude domain features in the field of speech signal processing. It takes into account the auditory characteristics of the human ear and has a higher resolution in the low-frequency part of the speech. The higher the frequency, the lower the resolution. The main operation is to map the linear spectrum to a nonlinear Mel spectrum and then perform cepstrum analysis.

[0057] Then conduct back-end model analysis and modeling: The RawNet2 model is an end-to-end deep neural network model that operates directly on the raw speech waveform, almost completely avoiding reliance on handcrafted features. This model is an improvement on RawNet, combining the original RawNet method with the SincNet method.

[0058] The first layer of the RawNet2 model operates directly on the raw waveform and contains a set of bandpass filters parameterized in the form of sinc functions.

[0059] The model uses a restricted first layer with fewer learnable parameters, with fixed rectangular filter responses, learning only the cut-in and cut-off frequencies, thereby learning a more meaningful filter bank structure and output.

[0060] The upper layer consists of residual blocks and GRU layers. The residual block is used to extract frame-level representations. The use of skip connections can effectively reduce the problems of gradient disappearance and network model degradation. The network structure of the residual block is as follows: Figure 4As shown in the figure, assuming that the input is x, after propagation through layers A and B, even if the model has the problem of gradient attenuation, the gradient at x can be directly passed to the ReLU activation function, thus realizing the cross-layer propagation of the gradient.

[0061] Gated recurrent unit (GRU) is used to aggregate speech-level representations. Like LSTM (Long-Short Term Memory), GRU is also designed to solve problems such as long-term memory and gradient vanishing in back propagation. However, compared with LSTM, GRU is easier to train and can improve training efficiency.

[0062] Compared with RawNet, the embedding dimension of the RawNet2 model has also increased greatly, from the original 128 dimensions to 1024 dimensions.

[0063] A new feature added to the RawNet2 model is the use of a sigmoid function, which is applied to the filter feature map scaling (FMS) of the residual block output. FMS is an attention mechanism whose goal is to obtain a more discriminative representation.

[0064] Figure 5 The parameters and architecture components of the RawNet2 model are shown, where the numbers in the CONV brackets represent the filter length, step size, and number of filters, respectively; the numbers in the Maxpooling brackets represent the number of maximum pooling layers.

[0065] The characteristic of the Res2Net model is that the features in the same block are divided into multiple channel groups, different groups are transformed at different scales, and residual-like connections are designed between groups. This structure can learn multi-scale feature representations.

[0066] Depending on the number of model network layers, the Res2Net model can be divided into multiple structures.

[0067] This embodiment uses the Res2Net50 model. It should be noted that the main difference between Res2Net and ResNet is that Res2Net proposes a new multi-scale feature representation module, such as Figure 2 As shown in , a hierarchical residual connection is constructed in a residual block, so that features of different scales can be integrated at a finer granularity, thereby enhancing the model's ability to express multi-scale features. They are similar in the overall network architecture. The network structure diagram of ResNet50 is used here, as shown in Figure 1 shown.

[0068] from Figure 1 It can be seen that the network structure of the Res2Net50 model is divided into five stages, among which the structures of btnk1 and btnk2 are as follows Figure 3 shown.

[0069] The Res2Net50 model training process is as follows: Stage 0, performs preliminary feature preprocessing on the input manual feature map, which is divided into two layers; The first step of the first layer is CONV: convolution operation, which uses convolution kernels to extract the basic features of the input image.

[0070] The second step of the first layer, BN: batch normalization, subtracts a mean from the input data and divides it by a standard deviation to reduce the value, which can effectively speed up the calculation; The third step of the first layer, ReLU: activation function, adds nonlinear factors, allowing the model to learn more complex feature representations; The second layer is the maximum pooling layer, which reduces the dimension of the data by selecting the maximum value in the local area while retaining important feature information; Stage 1, using three BottleNeck blocks for further feature extraction; Stage 2, uses 4 BottleNeck blocks for further feature extraction; Stage 3 uses 6 BottleNeck blocks to deeply extract features to enhance the model's ability to understand complex images; Stage 4 uses three BottleNeck blocks to extract features and prepare for the final classification task.

[0071] The BottleNeck block includes BTNK1 and BTNK2. BTNK1 has one more convolutional layer than BTNK2. Stage1 includes 1 BTNK1 and 2 BTNK2s; Stage2 includes 1 BTNK1 and 3 BTNK2s; Stage3 includes 1 BTNK1 and 5 BTNK2s; Stage4 includes 1 BTNK1 and 2 BTNK2s.

[0072] Specifically, Stage 0: Preprocessing of the input, divided into two layers. (3, 224, 224) represents the number of channels, height and width of the input data, represented by (C, H, W). Assuming that the width and height of the input data are equal, it can be represented as (C, W, W). The first layer of this stage is divided into three steps: CONV: convolution operation, 7×7 represents the size of the convolution kernel, 64 represents the number of output channels after the convolution operation, and / 2 represents the step size.

[0073] BN: Batch Normalization. Subtracting a mean from the input data, or dividing it by a variance, reduces the value, which can effectively speed up the calculation; ReLU: ReLU activation function adds nonlinear factors to improve the expressiveness of the model.

[0074] The second layer of this stage is the maximum pooling layer (MAXPOOL). 3×3 represents the size of the pooling kernel, and / 2 represents the step size. Its functions include: reducing the dimension of the information extracted by the convolution layer, reducing the amount of calculation, and strengthening the invariance of features.

[0075] The last four stages of the Res2Net50 model are all composed of BottleNeck blocks, and the structures are similar. The four stages include 3, 4, 6, and 3 BottleNeck blocks respectively. Stage 1 will be introduced in detail here, and the last three stages have the same principle as Stage 1, so they will not be described in detail.

[0076] Stage 1: The input shape is (64, 56, 56) and the output shape is (256, 56, 56). It consists of two types of BottleNeck blocks, referred to as BTNK1 and BTNK2.

[0077] BTNK1: The number of input and output channels is different. It includes four variable parameters: C, W, C1 and S. C represents the number of input data channels, W represents the width of the input data, C1 represents the number of output data channels, and S represents the step size. BTNK1 has one more convolution layer on the right than BTNK2. The function of this convolution layer is to directly change the number of input data channels to make it the same as the number of input data channels after three layers of convolution. Then the two can be added together to get the final output through the RELU activation function.

[0078] BTNK2: The number of channels of input and output is the same. It includes two variable parameters: C and W. C represents the number of channels of input data, and W represents the width of input data. Since the number of channels of input data does not change after three layers of convolution, the final output can be obtained by directly adding it to the input data and using the RELU activation function.

[0079] The specific modification steps of the Res2Net model Block are as follows: 1. Introduce a hyperparameter scale, which means how many groups the feature map is divided into, abbreviated as s; 2. Assuming that the number of channels of the output feature of the first 1 × 1 convolutional layer is n, the Res2Net model divides it into s groups according to the number of channels. Assuming that the number of channels of each group of features is w, then n = s × w. Each group of features after equal division is represented as x(i), where i∈{1, 2, …, s}; 3. The first group of features after grouping does not undergo convolution operations, and the other groups correspond to convolution operations K(i), where i∈{2, 3, …, s}. The output of the convolution operation K(i) is represented by y(i). Starting from the second group, before each convolution operation K(i), the output y(i-1) of the previous group of convolution operations needs to be residually connected with the features x(i) of the group as the input of K(i), and so on, until the end of the last group of features. Its expression is as follows:

[0080] Through this inter-layer overlap, each convolution operation receives information from each set of features X(j) (j≤i) before, so that the output of each set of convolution contains richer information, including multi-scale features. This multi-scale feature not only improves the capacity of the system, but also helps the system perform better when generalizing to unknown types of attacks.

[0081] 4. Concatenate the outputs corresponding to each set of features and input them into the last 1 × 1 convolutional layer to fuse these multi-scale features to obtain the final output.

[0082] After the model training of the RawNet2 model and the Res2Net model, the two models are fused: the two models in this embodiment are output in the format of triples, and their contents are expressed as follows: α i =<input voice, label of the voice, score>.

[0083] Assume that the score of each speech in the RawNet2 model is X i , the score of each speech in the Res2Net model is Y i , respectively assign weights of β1 and β2 to the two models (β1+β2=1), then the score of each speech after fusion can be expressed as: Z i =β1X i +β2Y i .

[0084] Secondly, we use the equal error rate (EER) as the performance evaluation indicator of the fusion model (the fusion model of the RawNet2 model and the Res2Net model is referred to as the fusion model).

[0085] Equal Error Rate (EER) is the error rate when the False Accept Rate (FAR) and the False Rejection Rate (FRR) are equal. When the fusion model mistakenly classifies a forged voice as real voice, it is a false acceptance, and when the fusion model mistakenly classifies a real voice as a forged voice, it is a false rejection. Given the score and threshold θ of each voice of the fusion model, θ is the decision threshold. When FAR (θ) is equal to FRR (θ), the error rate is called equal error rate EER. The smaller the EER value, the better the system performance. It represents the performance of the voiceprint fraud detection system and is an important parameter for measuring system performance.

[0086] The false acceptance rate and false rejection rate can be calculated according to the following formula (1) and formula (2) respectively:

[0087]

[0088] Assuming the threshold θ is x, the false rejection rate FRR represents the proportion of true speech scores lower than x, and the false acceptance rate FAR represents the proportion of deceptive speech scores higher than x. The process of solving EER is to find a score threshold x such that FRR=FAR. At this time, FAR (or FRR) is the equal error rate EER. The smaller the EER, the better the model performance.

[0089] Each speech score obtained by the fusion model is processed and stored in arrays, which are the real speech score (bona_cm) and the spoof speech score (spoof_cm). The EER is calculated using the compute_eer function. The function uses the compute_det_curve function to draw the DET curve, that is, the image of FRR and FAR changing with the threshold. Then, the EER is determined by finding the intersection of the FRR and FAR curves. This process is automated, ensuring that the classification boundary between the real speech and the spoof speech of the fusion model is optimal.

[0090] Embodiment 2, a pseudo-speech detection method based on the RawNet2 model and the Res2Net model, firstly prepares the original data, and then performs the following steps: Step 1: First, extract manual features from the preprocessed speech signal, then input the manual features of each speech into the Res2Net model, and finally the Res2Net model outputs the score of each speech signal. It is divided into the following steps: S11, data preprocessing; The S11 data preprocessing method is to divide the voice files into real voice and spoofed voice, and the metadata files into protocol files and CM protocol files. The spk2utt file, utt2index file, utt2spk file and utt2systemID are generated through the protocol files.

[0091] S12, feature extraction, Extract manual features; the types of manual features extracted from speech signals include cqt, spec, and mfcc. Manual features are extracted by inputting corresponding codes into the Python interpreter. Before extracting manual features, the input speech signal is pre-emphasized, framed, and windowed.

[0092] S13, model training; S14, model scoring, The model scores each speech.

[0093] S15, calculating the EER of the model. Models with different parameters have different EERs. The model with the lowest EER is selected and then proceed to step three.

[0094] Step 2: First, input the speech signal into the RawNet2 model, and then the RawNet2 model processes the input speech signal end-to-end and outputs the score of each speech signal. It is divided into the following steps: S21, data preprocessing; S21, the data preprocessing method is to organize each voice data into a file containing speaker_id, file_name, path, sys_id and key, and save the preprocessed voice data into a .npy file.

[0095] S22, feature extraction and scoring, The RawNet2 model processes the input speech signal end-to-end and scores each speech.

[0096] S23, calculating the EER of the model. Models with different parameters have different EERs. The model with the lowest EER is selected and then proceed to step three.

[0097] Step 3: Data normalization: Normalize the scores of the RawNet2 model and the Res2Net model.

[0098] The RawNet2 model and Res2Net model scores are mapped to the (0,1) interval, and the data normalization method is maximum and minimum normalization, sigmoid change, or inverse tangent function normalization.

[0099] The three methods of data normalization were used to calculate the normalized data scores respectively, and then the EER of the original data and the normalized data were calculated respectively. The data normalization method with the closest results between the original data EER and the normalized data EER was taken as the final data normalization algorithm.

[0100] Step 4: score fusion, output the score of each speech signal; By Z i =β1X i +β2Y i The scores of the Res2Net model and the RawNet2 model are combined, where the score of each voice in the RawNet2 model is X i , the score of each speech in the Res2Net model is Y i , assign weights to the two models β1 and β2 respectively (β1+β2=1).

[0101] Step 5: Calculate the EER of the fusion model.

[0102] In Example 3, different manual features (mfcc, cqt, and spec) are extracted to train the Res2Net model. This example compares the scores of the Res2Net model under the three manual features and selects the two groups of results with the best performance (judged based on EER).

[0103] First, the manual features are extracted from the original data set, and then the Res2Net model itself is used for classification or confirmation. In this embodiment, the manual features need to be confirmed by the Res2Net model.

[0104] The RawNet2 model of the end-to-end recognition algorithm is trained, and a deep neural network is used to autonomously learn speech features for scoring.

[0105] Different weights are assigned to the two models respectively, and the scores of the Res2Net model and the RawNet2 model are fused. After obtaining the fusion score of the two models, the evaluation index EER of the fusion model is calculated.

[0106] Example 4: In order to obtain a large number of real voice and forged voice data sets and ensure the diversity and representativeness of the data sets. This example uses the data set provided by the ASVspoof 2019 challenge, involving data such as recording playback, speech synthesis, and speech conversion. Compared with traditional methods, the fusion model greatly improves the security of voiceprint authentication.

[0107] Different parameter settings during the Res2Net model training process will have a great impact on the final model accuracy. This embodiment selects three sets of parameters to train the model, namely Res2Net50Model_14w_8s, Res2Net50Model_26w_4s and Res2Net50Model_26w_8s. 50 represents the depth of the model, w represents the number of channels for each set of features, and s represents the feature scale, that is, the number of feature groups.

[0108] Batch_size: The number of samples used for gradient descent. In this example, the batch_size is set to 16 and 32 respectively to train the model.

[0109] The best performing parameters during training, including warm-up learning rate, number of iterations, number of samples used for gradient descent, etc., are used to test on the validation set. The EER of the model is further calculated based on the score of each speech to evaluate the model performance.

[0110] When solving the minimum value of the model loss function, it is a common method to use the small batch gradient descent method to iterate step by step. Different batches, that is, different batch-sizes, may also result in different final convergence results of the model. A large batch-size converges faster if resources allow, but may fall into a local optimum. A small batch-size converges slower, has greater randomness, and may achieve better performance.

[0111] In the process of training the RawNet2 model, this paper selected three batch-sizes for testing, and the test results are shown in the following table:

[0112] In the process of training the Res2Net model, considering that the number of channels w and the number of feature groups s of each set of features will affect the final result of the model, this embodiment uses three sets of parameters for training (Res2Net50 model_14w_8s, Res2Net50 model_26w_4s and Res2Net50 model_26w_8s). In the process of training these three models, considering that when the RawNet2 model was trained in the early stage, it was found that the model converged too slowly when batch-size=8, and there was no obvious performance improvement, so batch-sizes of 16 and 32 were selected for testing respectively. In the subsequent experiment on the first set of parameters of the Res2Net model, it was found that the model results were not ideal when batch-size=16, and the convergence speed was slow, so it is planned to test only the case of batch-size=32 for the subsequent two sets of parameter experiments. However, in the final experiment, due to limited computer hardware resources and too many parameters in the Res2Net50 model_26w_8s model, we could only train the model by reducing batch-size to 16. The final results of the experiment are shown in the following table:

[0113] Since the two models have different scoring standards: the RawNet2 model has a higher real speech score, close to 1, while the Res2Net model has a lower real speech score, close to zero, the data needs to be processed first. In this paper, the Res2Net model data is processed so that the real speech score is close to 1, and the deceptive speech score is close to 0, which is consistent with the RawNet2 model. The expression used is as follows: score i =max_score- score i max_score represents the highest score value among all speech data.

[0114] The fusion model is then normalized. Considering that the maximum and minimum normalization is a linear transformation, it is more suitable for normalizing data with smaller differences. Therefore, this embodiment finally selects the normalization method of the maximum and minimum normalization algorithm to map the score of the Res2Net model to the (0, 1) interval. This embodiment selects the Res2Net50 model_14w_8s model with batch-size=32 and feature=Lowpowcqt for testing. The test results show that the EER is 8.44268%.

[0115] In the final score fusion stage, this paper selects the best performance of the RawNet2 model when batch-size=16 and the three groups with good performance of the Res2Net model, namely, Res2Net50 model_14w_8s with batch-size=32, feature=lowpowcqt, Res2Net50 model_26w_4s with batch-size=32, feature=lowpowcqt, and Res2Net50 model_26w_8s with batch-size=16, feature=lowpowcqt, and gives the two models different weights for experiments. The experimental results are shown in the following table. The data marked in bold are the best performance results of this group.

[0116]

[0117] The following conclusions can be drawn from the data in the table: (1) When the Res2Net50 model_14w_4s is fused with the RawNet2 model, the RawNet2 model and the Res2Net model are given weights of 0.95 and 0.05 respectively. The final result EER is 4.51425%, which is better than the 4.62421% of the RawNet2 model and the 8.44268% of the Res2Net model alone. (2) When fusing the Res2Net50 model_26w_4s model with the RawNet2 model, this experiment used six sets of different weight parameters, and the results were all better than those of a single model. The best EER result was 50.4% and 25.7% higher than that of the Res2Net50 model and the RawNet2 model, respectively.

[0118] (3) When fusing the Res2Net50 model_26w_8s model with the RawNet2 model, this experiment used five sets of different weight parameters, and the results were all better than those of a single model. The best EER result was 46.5% and 19.7% higher than that of the Res2Net50 model and the RawNet2 model, respectively.

[0119] This proves that the score fusion of the two models in this paper can improve the accuracy of fraudulent speech detection to a certain extent.

[0120] Through this embodiment, it can be found that compared with the results of two types of single voiceprint fraud detection models, the error rate EER of the model after score fusion is reduced. The set of test results with the best fusion performance has an EER reduced by 50.4% and 25.7% respectively compared with the single Res2Net model and RawNet2 model.

[0121] Example 5: Res2Net model data preprocessing method The voice files are divided into: Bonafide Speech: Real, unmodified speech files, usually used for training and testing the normal behavior of the model.

[0122] Spoofed Speech: A tampered voice file, including different types of attacks such as recording playback, speech synthesis, and speech conversion.

[0123] File format: Usually WAV format, including sampling rate and sampling point information.

[0124] The metadata files are divided into: Protocol Files: Contains detailed information about the dataset, such as speaker ID, file name, generation algorithm, etc.

[0125] CM Protocol Files: Protocol files used for detection tasks, containing labels for real and spoofed speech.

[0126] File format: Usually a text file (.txt or .csv), each line contains metadata information for a sample.

[0127] Generates some auxiliary files that help the model process data more efficiently during training and testing.

[0128] Generate spk2utt file: The spk2utt file records each speaker ID and its corresponding voice file name.

[0129] Format: Each line contains a speaker ID and multiple voice file names, separated by spaces.

[0130] Generate utt2index file: The utt2index file records the generation algorithm and label corresponding to each speech.

[0131] Format: Each line contains a voice file name, generation algorithm ID and label (0 for real voice, 1 for spoof voice).

[0132] Generate utt2spk file: The utt2spk file records each voice file and its corresponding speaker ID.

[0133] Format: Each line contains a voice file name and speaker ID.

[0134] The utt2systemID file records each voice file and its corresponding generation algorithm ID.

[0135] Format: Each line contains a voice file name and generation algorithm ID.

[0136] Preprocess the data set and prepare the data packets required for the feature extraction stage, including four files: spk2utt, utt2index, utt2spk, and utt2systemID.

[0137] spk2utt: speaker ID and the voice file name corresponding to the speaker; utt2index: the generation algorithm corresponding to each voice (A01-A19; the real voice is represented by "-"), and its corresponding label (1 for spoofed voice and 0 for real voice); utt2spk: each voice file and the speaker ID corresponding to the voice; utt2systemID: Each voice file and the generation algorithm corresponding to the voice file.

[0138] Example 6: RawNet2 model data preprocessing Process the data set and change it to: speaker_id: speaker ID; file_name: the voice file name corresponding to the speaker; path: the storage location of the voice file; sys_id: the generation algorithm ID corresponding to the voice; key: the ID of the voice (real voice is 1, spoofed voice is 0) Store various parameters of the preprocessed speech data in the specified npy file.

[0139] Compared with the prior art, the present invention has the following beneficial effects: Multi-scale feature extraction: The multi-scale feature extraction capability of the Res2Net model enables the network to capture speech features at different levels simultaneously, thereby improving the ability to distinguish forged speech.

[0140] Deeper network structure: Compared with the traditional ResNet, the Res2Net model can build a deeper network structure without increasing too much computational effort, thereby extracting richer feature information.

[0141] The system uses multiple front-end feature extraction methods such as MFCC, CQT and spectrogram (SPEC). Compared with the traditional single feature extraction method, this diversified feature extraction method can more comprehensively capture the characteristics of the speech signal and improve the system's ability to detect forged speech.

[0142] Deep learning models are able to learn more advanced feature representations from data. Combining the two models takes full advantage of their respective strengths and improves the detection accuracy of the system.

[0143] The two deep learning models, Res2Net and RawNet2, are combined to fuse their feature representations into the final detection results. Through this fusion, the system can better understand the speech signal and improve the accuracy and robustness of pseudo-speech detection. The end-to-end learning of the RawNet2 model and its ability to extract features directly from the original audio signal are combined with the multi-scale feature representation and deep residual learning of the Res2Net model to build a more powerful and efficient pseudo-speech detection model.

[0144] The equal error rate (EER) is used as the evaluation indicator of the pseudo-speech detection model. It has the advantages of balancing false positive and false negative errors, providing a single value for easy comparison between models, intuitive and easy to understand concepts, threshold independence, and standardization. It can effectively evaluate the overall performance of the system.

[0145] The fake voice detection security system can be deployed in multiple fields, such as voice recognition systems, voice social platforms, mobile payment authentication, security systems, etc., to protect users from fake voice interference. This comprehensive application design makes the system more practical and has a wider application prospect.

[0146] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model, characterized in that: First, prepare the original data, and then proceed as follows: Step 1: First, extract manual features from the preprocessed speech signal, then input the manual features of each speech into the Res2Net model, and finally the Res2Net model outputs the score of each speech signal. It is specifically divided into the following steps: S11, data preprocessing; S12, feature extraction, Extracting manual features; S13, model training; S14, model scoring, The model scores each speech; Step 2: First, input the speech signal into the RawNet2 model. Then the RawNet2 model processes the input speech signal end-to-end and outputs the score of each speech signal. The specific steps are as follows: S21, data preprocessing; S22, feature extraction and scoring, The RawNet2 model processes the input speech signal end-to-end and scores each speech. Step 3: Data normalization. Normalize the scores of the RawNet2 model and the Res2Net model; Step 4: score fusion, output the score of each speech in the fusion model; Step 5: EER calculation: calculate the EER of the fusion model.

2. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model according to claim 1, characterized in that: The S11 data preprocessing method is to divide the voice files into real voice and spoofed voice, and the metadata files into protocol files and CM protocol files. The spk2utt file, utt2index file, utt2spk file and utt2systemID are generated through the protocol files.

3. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model according to claim 1, characterized in that: S21, the data preprocessing method is to organize each voice data into a file containing speaker_id, file_name, path, sys_id and key, and save the preprocessed voice data into a .npy file.

4. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model according to claim 1, characterized in that: In step 1, the manual feature types extracted from the speech signal include cqt, spec, and mfcc. The manual features are extracted by inputting the corresponding code into the Python interpreter. Before extracting the manual features, the input speech signal is pre-emphasized, framed, and windowed.

5. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model according to claim 1, characterized in that: In step 4, pass Z i =β1X i +β2Y i Merge the Res2Net model and the RawNet2 model scores, The score of each voice in the RawNet2 model is X i , the score of each speech in the Res2Net model is Y i , assign weights of β1 and β2 to the two models respectively, and β1+β2=1.

6. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model according to claim 1, characterized in that: Step 1 also includes S15, calculating the EER of the model. Models with different parameters have different EERs. The model with the lowest EER is selected and then step 3 is performed.

7. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model according to claim 8, characterized in that: Step 2 also includes S23, calculating the EER of the model. Models with different parameters have different EERs. The model with the lowest EER is selected and then step 3 is performed.

8. A pseudo speech detection method based on a RawNet2 model and a Res2Net model according to any one of claims 1 to 7, characterized in that: In step 3, the scores of the RawNet2 model and the Res2Net model are mapped to the interval (0, 1), and the data normalization method is maximum and minimum normalization, sigmoid change, or inverse tangent function normalization.

9. A pseudo-speech detection method based on the RawNet2 model and the Res2Net model according to claim 8, characterized in that: The three methods of data normalization were used to calculate the normalized data scores respectively, and then the EER of the original data and the normalized data were calculated respectively. The data normalization method with the closest results between the original data EER and the normalized data EER was taken as the final data normalization algorithm.

Images