Voice forgery detection method based on multi-scale GMM-ResNet model

Abstract

The invention belongs to the technical field of voice forgery detection, and particularly relates to a voice forgery detection method based on a multi-scale GMM-ResNet model, real voice data and forgery voice data of a speaker are acquired as original voice data, and the multi-scale GMM-ResNet model mainly comprises two parts: a multi-scale logarithmic Gaussian probability feature fusion model and a multi-scale feature aggregation ResNet model. According to the multi-scale LGP feature fusion, weighted fusion is carried out on three different-scale LGP features obtained based on GMM of different orders, so that information exchange among the different-scale LGP features is promoted, the multi-scale feature aggregation ResNet model aggregates features output by each ResNet block, feature information of different levels in the network is fully fused, and the multi-scale LGP feature fusion is realized. Therefore, the feature extraction capability of the network is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of voice forgery detection, and in particular to a voice forgery detection method based on a multi-scale GMM-ResNet model. Background Art

[0002] The Automatic Speaker Verification (ASV) system is a system that authenticates the speaker's identity based on the speaker's voice information. It is currently widely used in various life scenarios such as mobile phone unlocking, smart access control, and bank identity authentication. With the application of deep learning models in recent years, the ASV system has also made significant progress and demonstrated its good performance. However, it is also vulnerable to forgery attacks by synthesized or converted speech. The synthetic speech forgery detection system is committed to solving this problem.

[0003] In practical applications, although voice forgery detection technology has become increasingly mature, it still faces the problem of insufficient model generalization performance. This is because actual application scenarios are often more vulnerable to voice attacks of unknown attack types, so it is necessary to adopt effective voice forgery detection methods to reduce the impact of various codecs and transmission channels on the voice forgery detection system and improve the generalization performance of the model. Summary of the invention

[0004] The purpose of the present invention is to provide a voice forgery detection method based on a multi-scale GMM-ResNet model to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] A method for detecting speech forgery based on a multi-scale GMM-ResNet model. The specific steps of the method for detecting speech forgery are as follows:

[0007] S1. Acquire the speaker's real voice data and forged voice data as original voice data;

[0008] S2, calculating the linear frequency cepstral coefficient characteristics of the original speech data;

[0009] S3, using 128-order, 256-order and 512-order Gaussian mixture models to model the linear frequency cepstral coefficient characteristics of the original speech data, and training them on the original speech data to obtain Gaussian mixture models of different orders, including a 128-order Gaussian mixture model, a 256-order Gaussian mixture model and a 512-order Gaussian mixture model;

[0010] S4, using the linear frequency cepstral coefficient features of the original speech data as inputs of a 128-order Gaussian mixture model, a 256-order Gaussian mixture model, and a 512-order Gaussian mixture model, respectively calculating and obtaining 128-order, 256-order, and 512-order logarithmic Gaussian probability features;

[0011] S5. Construct a multi-scale GMM-ResNet model, use the logarithmic Gaussian probability features based on the above three different orders as the input of the multi-scale GMM-ResNet model, train the multi-scale GMM-ResNet model, and use the trained multi-scale GMM-ResNet model to detect forgery of new voice data.

[0012] Preferably, the speech Gaussian mixture model is trained using a maximum expectation algorithm.

[0013] Preferably, S4 includes:

[0014] S4.1, calculate the logarithmic Gaussian probability feature y of the linear frequency cepstral coefficient feature x of the original speech data, whose size is equal to the order of the Gaussian mixture model, where each component y i The calculation formula is:

[0015] y i =log(w i ·p i (x))

[0016] Where: w i is the weight of the i-th Gaussian component of the Gaussian mixture model, p i (x) is a single Gaussian probability density function, μ i is the mean vector, Σ i is the covariance matrix, as shown in the formula:

[0017]

[0018] S4.2, for y i Perform mean variance normalization to obtain the final logarithmic Gaussian probability feature f i :

[0019]

[0020] in: and It is the mean and variance of all the logarithmic Gaussian probability features based on the original speech Gaussian mixture model.

[0021] Preferably, S5 includes:

[0022] S5.1. Construct a multi-scale logarithmic Gaussian probability feature fusion network, and use the probability features obtained based on the 128-order Gaussian mixture model, the 256-order Gaussian mixture model, and the 512-order Gaussian mixture model as the input of the multi-scale logarithmic Gaussian probability feature fusion network;

[0023] S5.2, Multi-scale logarithmic Gaussian probability feature fusion network performs weighted fusion on the logarithmic Gaussian probability features of three different scales, and the weighted sum of the coefficients is 1. After fusion, features of three scales of 128 dimensions, 256 dimensions and 512 dimensions are obtained;

[0024] S5.3. Construct three multi-scale feature aggregation ResNet models, and use the above fused 128-dimensional features, 256-dimensional features, and 512-dimensional features as the input of their respective multi-scale feature aggregation ResNet models;

[0025] S5.4, three multi-scale feature aggregation ResNet models map the three sets of features into three sets of two-dimensional vectors respectively;

[0026] S5.5, performing an adaptive maximum pooling operation on the two-dimensional vectors obtained above to obtain three groups of one-dimensional vectors of fixed sizes;

[0027] S5.6. Concatenate the three groups of one-dimensional vectors of fixed size to obtain a group of one-dimensional vectors, and input them into a fully connected linear layer, and output the scores of the real speech and the forged speech labels through the Softmax function;

[0028] S5.7. Determine whether the voice is real or forged based on the classification score.

[0029] Preferably, S5.1 includes:

[0030] S5.1.1, the multi-scale logarithmic Gaussian probability feature fusion network adopts a modular structure, with three modules in total;

[0031] S5.1.2. Each multi-scale log-Gaussian probability feature fusion module includes two convolutional layers and a batch normalization layer, where the convolutional layer uses one-dimensional convolution;

[0032] S5.1.3. The input features in each multi-scale log-Gaussian probability feature fusion module are log-Gaussian probability features of three scales: 128-order, 256-order, and 512-order.

[0033] Preferably, S5.3 includes:

[0034] S5.3.1. Each multi-scale feature aggregation ResNet network path adopts a modular structure, with a total of six modules;

[0035] S5.3.2. Each module in the multi-scale feature aggregation ResNet network path includes a convolutional layer, a batch normalization layer, and a nonlinear activation layer, where the convolutional layer uses one-dimensional convolution;

[0036] S5.3.3. The number of input feature channels in each multi-scale feature aggregation ResNet network path is 128, 256, and 512 respectively.

[0037] Preferably, S5.7 includes:

[0038] S5.7.1. Obtain the scores of the real speech and the forged speech according to the classification results;

[0039] S5.7.2. If the score of the real voice is greater than the score of the forged voice, the speaker's voice is judged to be real voice; if the score of the real voice is less than the score of the forged voice, the speaker's voice is judged to be forged voice.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present application discloses a ResNet residual block based on one-dimensional convolution; the speech data is modeled using Gaussian mixture models of order 128, order 256 and order 512 respectively, and logarithmic Gaussian probability features of different scales are extracted. Combining the multi-scale GMM-ResNet model can effectively improve the performance of the speech forgery detection system. The multi-scale GMM-ResNet model mainly includes two parts: multi-scale logarithmic Gaussian probability feature fusion and multi-scale feature aggregation ResNet model. Multi-scale LGP feature fusion performs weighted fusion of three different scale LGP features obtained based on GMMs of different orders, thereby promoting information exchange between LGP features of different scales. The multi-scale feature aggregation ResNet model aggregates the features output by each ResNet block, fully integrates the feature information of different levels in the network, and thus improves the feature extraction capability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0045] Example:

[0046] like Figure 1 As shown, the present invention provides a technical solution: a voice forgery detection method based on a multi-scale GMM-ResNet model, and the specific steps of the voice forgery detection method are as follows:

[0047] S1. Acquire the speaker's real voice data and forged voice data as original voice data;

[0048] S2, calculating the linear frequency cepstral coefficient characteristics of the original speech data;

[0049] S2.1. In the feature extraction process, the speech signal is first divided into frames with a frame length of 20ms and a frame shift of 10ms. Each frame of the framed signal is then windowed using a Hamming window, and after a 1024-point Fourier transform, it is input into a linear triangular filter bank for processing, where the number of filters is 20, and finally a logarithmic operation and discrete cosine transform are performed to obtain the cepstral coefficients. By extracting the first-order difference and second-order difference coefficients of the linear frequency cepstral coefficients and splicing them with the original coefficients, a 60-dimensional linear frequency cepstral coefficient feature vector is finally obtained. The linear frequency cepstral coefficient features of speech retain a fixed length of 400 frames along the time axis. If the length is greater than 400 frames, it is directly truncated, and if it is less than 400 frames, it is repeated and padded.

[0050] S3, using 128-order, 256-order and 512-order Gaussian mixture models to model the linear frequency cepstral coefficient characteristics of the original speech data, and training them on the original speech data to obtain Gaussian mixture models of different orders, including a 128-order Gaussian mixture model, a 256-order Gaussian mixture model and a 512-order Gaussian mixture model;

[0051] The Gaussian mixture model (GMM) of this embodiment is composed of a linear weighted combination of K single Gaussian probability density functions with different weights and parameters, and can fit data distribution of any shape in the sample space. Its probability density function is shown in the formula:

[0052]

[0053] Among them, K is the number of Gaussian distributions, x is the sample vector, and w i is the weight of the i-th Gaussian component and satisfies p i (x) is a single Gaussian probability density function, μ iis the mean vector, Σ i is the covariance matrix, as shown in the formula:

[0054]

[0055] S4, using the linear frequency cepstral coefficient features of the original speech data as inputs of a 128-order Gaussian mixture model, a 256-order Gaussian mixture model, and a 512-order Gaussian mixture model, respectively calculating and obtaining 128-order, 256-order, and 512-order logarithmic Gaussian probability features;

[0056] S4.1, calculate the logarithmic Gaussian probability feature y of the linear frequency cepstral coefficient feature x of the original speech data, whose size is equal to the order of the Gaussian mixture model, where each component y i The calculation formula is:

[0057] y i =log(w i ·p i (x))

[0058] Where: w i is the weight of the i-th Gaussian component of the Gaussian mixture model, p i (x) is a single Gaussian probability density function.

[0059] S4.2, for y i Perform mean variance normalization to obtain the final logarithmic Gaussian probability feature f i :

[0060]

[0061] in: and is the mean and variance of all the logarithmic Gaussian probability features based on the original post-speech Gaussian mixture model.

[0062] S5. Construct a multi-scale GMM-ResNet model, use the logarithmic Gaussian probability features based on the above three different orders as the input of the multi-scale GMM-ResNet model, train the multi-scale GMM-ResNet model, and use the trained multi-scale GMM-ResNet model to detect forgery of new voice data.

[0063] S5.1. Construct a multi-scale logarithmic Gaussian probability feature fusion network, and use the Gaussian probability features obtained based on the 128-order Gaussian mixture model, the 256-order Gaussian mixture model, and the 512-order Gaussian mixture model as the input of the multi-scale logarithmic Gaussian probability feature fusion network;

[0064] Assume that the number of speech frames is T, and the three Gaussian mixture models are composed of 128, 256 and 512 weighted Gaussian components respectively. S4 converts the speech data into 128*T, 256*T and 512*T Gaussian probability features. The multi-scale Gaussian probability feature fusion network includes three basic modules. Each basic module includes two one-dimensional convolutional layers and a batch normalization layer.

[0065] S5.2, Multi-scale logarithmic Gaussian probability feature fusion network performs weighted fusion on the logarithmic Gaussian probability features of three different scales, and the weighted sum of the coefficients is 1. After fusion, features of three scales of 128 dimensions, 256 dimensions and 512 dimensions are obtained;

[0066] The feature fusion calculation formula is:

[0067] X'1=w 11 X1+w 12 f 12 (X2)+w 13 f 13 (X3)

[0068] X'2=w 21 f 21 (X1)+w 22 X2+w 23 f 23 (X3)

[0069] X'3=w 31 f 31 (X1)+w 32 f 32 (X2)+w 33 X3

[0070] w i1 +w i2 +w i3 =1

[0071] Among them, X1, X2, and X3 represent Gaussian probability features of order 128, 256, and 512 respectively, X'1, X'2, and X'3 represent the results of multi-scale Gaussian probability feature fusion, f represents the convolution operation, and w represents the trainable weight parameter. Since the features of the three scales have different numbers of channels, we use the convolution layer f for expansion or compression, where the convolution kernel size is 1 and the step size is 1.

[0072] S5.3. Construct three multi-scale feature aggregation ResNet models, and use the above fused 128-dimensional features, 256-dimensional features, and 512-dimensional features as the input of their respective multi-scale feature aggregation ResNet models;

[0073] For the Resnet network path with 128-dimensional input features, there are 7 modules in total: one convolution module and six ResNet Block modules with the same structure. The first convolution module in the ResNet network path includes a one-dimensional convolution layer, a batch normalization layer, and a ReLU nonlinear activation layer. The number of input channels is 128 and the number of output channels is 128. The ResNet Block module in the ResNet network path includes a one-dimensional convolution layer, a batch normalization layer, a ReLU nonlinear activation layer, a one-dimensional convolution layer, and a ReLU nonlinear activation layer. The number of input channels is 128 and the number of output channels is 128. The output of each ResNet Block module is spliced ​​and input to the batch normalization layer to achieve the purpose of multi-scale feature aggregation. For the ResNet network path with 256 and 512-dimensional input features, the structure is the same as above, with 256 and 512 input channels and 256 and 512 output channels respectively.

[0074] S5.4, three multi-scale feature aggregation ResNet models map the three sets of features into three sets of two-dimensional vectors respectively;

[0075] After the Gaussian probability features of 128*T, 256*T and 512*T pass through three ResNet network paths, two-dimensional vectors of sizes 128*T, 256*T and 512*T are obtained respectively;

[0076] S5.5, performing an adaptive maximum pooling operation on the two-dimensional vectors obtained above to obtain three groups of one-dimensional vectors of fixed sizes;

[0077] The three network paths respectively use adaptive maximum pooling operations to compress the obtained two-dimensional vectors in the time dimension to obtain the most representative feature vectors of the corresponding dimensions, and obtain one-dimensional vectors of sizes 128, 256, and 512 respectively;

[0078] S5.6. Concatenate the three groups of one-dimensional vectors of fixed size to obtain a group of one-dimensional vectors, and input them into a fully connected linear layer, and output the scores of the real speech and the forged speech labels through the Softmax function;

[0079] The three groups of one-dimensional vectors with sizes of 128, 256 and 512 are concatenated to obtain a one-dimensional vector with a size of 896. The one-dimensional vector is classified using a fully connected linear layer, and a one-dimensional vector with a size of 2 is output as the classification score;

[0080] S5.7. Determine whether the voice is real or forged based on the classification score.

[0081] If the score of the real voice is greater than the score of the forged voice, the speaker's voice is judged to be real voice; if the score of the real voice is less than the score of the forged voice, the speaker's voice is judged to be forged voice.

[0082] In order to verify the effectiveness of the present invention, this embodiment uses the ASVspoof2019 and ASVspoof2021 data sets to verify the scheme of the present invention, which consists of real and fake voices. The ASVspoof2019 challenge is divided into two evaluation scenarios: logical access (LA) scenario and physical access (PA) scenario, and these two scenarios respectively include three parts: training set, development set and evaluation set. The ASVspoof2021 challenge includes three evaluation scenarios: logical access (LA), physical access (PA) and deep fake voice (Deepfake, DF). The ASVspoof2021 data set only contains an evaluation set. This embodiment uses the LA training set of ASVspoof2019 for training, and verifies the scheme of the present invention on the LA evaluation set of ASVspoof2019 and the LA evaluation set of ASVspoof2021. The experiment uses a 512-order single-scale GMM-ResNet model as the baseline model, uses the equal error rate (EER) as the performance indicator, the training set is used to train the model parameters, and the evaluation set is used to compare the performance of the model. The experimental results are shown in Table 1:

[0083] Table 1

[0084]

[0085] It can be seen from the experimental results that the performance of the multi-scale GMM-ResNet model of the present invention is significantly better than that of the single-scale GMM-ResNet model.

[0086] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is limited by the attached claims rather than the above description. Therefore, it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the present invention, and any figure marks in the claims should not be regarded as limiting the claims involved.

[0087] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting voice forgery based on a multi-scale GMM-ResNet model, characterized in that: The specific steps of the voice forgery detection method are as follows: S1. Acquire the speaker's real voice data and forged voice data as original voice data; S2, calculating the linear frequency cepstral coefficient characteristics of the original speech data; S3, using 128-order, 256-order and 512-order Gaussian mixture models to model the linear frequency cepstral coefficient characteristics of the original speech data, and training them on the original speech data to obtain Gaussian mixture models of different orders, including a 128-order Gaussian mixture model, a 256-order Gaussian mixture model and a 512-order Gaussian mixture model; S4, using the linear frequency cepstral coefficient features of the original speech data as inputs of a 128-order Gaussian mixture model, a 256-order Gaussian mixture model, and a 512-order Gaussian mixture model, respectively calculating and obtaining 128-order, 256-order, and 512-order logarithmic Gaussian probability features; S5. Construct a multi-scale GMM-ResNet model, use the logarithmic Gaussian probability features based on the above three different orders as the input of the multi-scale GMM-ResNet model, train the multi-scale GMM-ResNet model, and use the trained multi-scale GMM-ResNet model to detect forgery of new voice data.

2. According to claim 1, a method for detecting voice forgery based on a multi-scale GMM-ResNet model is characterized in that: The speech Gaussian mixture model is trained using the maximum expectation algorithm.

3. According to claim 1, a method for detecting voice forgery based on a multi-scale GMM-ResNet model, characterized in that: The S4 includes: S4.1, calculate the logarithmic Gaussian probability feature y of the linear frequency cepstral coefficient feature x of the original speech data, whose size is equal to the order of the Gaussian mixture model, where each component y i The calculation formula is: y i =log(w i ·p i (x)) Where: w i is the weight of the i-th Gaussian component of the Gaussian mixture model, p i (x) is a single Gaussian probability density function, μ i is the mean vector, Σ i is the covariance matrix, as shown in the formula: S4.2, for y i Perform mean variance normalization to obtain the final logarithmic Gaussian probability feature f i : in: and It is the mean and variance of all the logarithmic Gaussian probability features based on the original speech Gaussian mixture model.

4. The method for detecting voice forgery based on a multi-scale GMM-ResNet model according to claim 1, characterized in that: The S5 includes: S5.

1. Construct a multi-scale logarithmic Gaussian probability feature fusion network, and use the probability features obtained based on the 128-order Gaussian mixture model, the 256-order Gaussian mixture model, and the 512-order Gaussian mixture model as the input of the multi-scale logarithmic Gaussian probability feature fusion network; S5.2, Multi-scale logarithmic Gaussian probability feature fusion network performs weighted fusion on the logarithmic Gaussian probability features of three different scales, and the weighted sum of the coefficients is 1. After fusion, features of three scales of 128 dimensions, 256 dimensions and 512 dimensions are obtained; S5.

3. Construct three multi-scale feature aggregation ResNet models, and use the above fused 128-dimensional features, 256-dimensional features, and 512-dimensional features as the input of their respective multi-scale feature aggregation ResNet models; S5.4, three multi-scale feature aggregation ResNet models map the three sets of features into three sets of two-dimensional vectors respectively; S5.5, performing an adaptive maximum pooling operation on the two-dimensional vectors obtained above to obtain three groups of one-dimensional vectors of fixed sizes; S5.

6. Concatenate the three groups of one-dimensional vectors of fixed size to obtain a group of one-dimensional vectors, and input them into a fully connected linear layer, and output the scores of the real speech and the forged speech labels through the Softmax function; S5.

7. Determine whether the voice is real or forged based on the classification score.

5. A method for detecting voice forgery based on a multi-scale GMM-ResNet model according to claim 4, characterized in that: The S5.1 includes: S5.1.1, the multi-scale logarithmic Gaussian probability feature fusion network adopts a modular structure, with three modules in total; S5.1.

2. Each multi-scale log-Gaussian probability feature fusion module includes two convolutional layers and a batch normalization layer, where the convolutional layer uses one-dimensional convolution; S5.1.

3. The input features in each multi-scale log-Gaussian probability feature fusion module are log-Gaussian probability features of three scales: 128-order, 256-order, and 512-order.

6. A method for detecting voice forgery based on a multi-scale GMM-ResNet model according to claim 4, characterized in that: The S5.3 includes: S5.3.

1. Each multi-scale feature aggregation ResNet network path adopts a modular structure, with a total of six modules; S5.3.

2. Each module in the multi-scale feature aggregation ResNet network path includes a convolutional layer, a batch normalization layer, and a nonlinear activation layer, where the convolutional layer uses one-dimensional convolution; S5.3.

3. The number of input feature channels in each multi-scale feature aggregation ResNet network path is 128, 256, and 512 respectively.

7. A method for detecting voice forgery based on a multi-scale GMM-ResNet model according to claim 4, characterized in that: The S5.7 includes: S5.7.

1. Obtain the scores of the real speech and the forged speech according to the classification results; S5.7.

2. If the score of the real voice is greater than the score of the forged voice, the speaker's voice is judged to be real voice; if the score of the real voice is less than the score of the forged voice, the speaker's voice is judged to be forged voice.

Images