A forged speech detection method and system based on correlation feature discriminant fusion network

Through the correlation feature discriminant fusion network, the bidirectional attention network and SCG-Res2Net50 extract and fuse advanced acoustic features, solving the problem of low accuracy in identifying fake speech and real speech, significantly improving fake speech detection performance.

CN118486330BActive Publication Date: 2024-11-05HEFEI XUAN TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410582116.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-11
Publication Date
2024-11-05
Estimated Expiration
2044-05-11

AI Technical Summary

Technical Problem

When the prior art recognizes fake speech and real speech, the accuracy of a single deep neural network extracting speech features is low, making it difficult to effectively distinguish synthesized speech from real speech.

Method used

Using a method based on the discriminant fusion network of the associated feature, advanced acoustic features are extracted through the bidirectional attention network and SCG-Res2Net50, and network parameters are initialized through the discriminant fusion method of the associated feature to fusion features to improve recognition performance.

Benefits of technology

It significantly improves the recognition performance of the forged voice detection system, and compared with ordinary feature splicing methods, it significantly improves the accuracy and performance of fraud detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118486330B_ABST
    Figure CN118486330B_ABST
Patent Text Reader

Abstract

A forged speech detection method and system based on a discriminative fusion network of associated features belongs to the field of speech processing and biometrics. The present invention first pre-processes the speech and CQT features, and uses them as the input of two parallel networks, namely the bidirectional attention network and SCG-Res2Net50. In the training stage, the network parameters are initialized by the discriminative fusion method of associated features, and the initialization projection matrix parameters are multiplied with the original features to obtain the converted new features, and then the two features are spliced, and then the spliced ​​features are input into the fully connected layer for classification, and the trained network model is obtained as a classifier for real speech and fraudulent speech, and the real speech and forged speech are classified according to the result of subtracting the two numerical values ​​in the output of the fully connected layer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of speech processing and biometric identification, and in particular relates to a forged speech detection method and system based on a correlation feature discriminant fusion network. Background Art

[0002] Speech processing technology makes voices "readable" and enables computers to "understand" human language and respond. It is one of the key technologies for artificial intelligence to achieve human-computer interaction. Among them, machine-synthesized voice fraud detection has gradually become an important scientific issue in the field of speech processing and has broad application prospects. Voice fraud refers to the "modification and impersonation" of an illegal voice that has not been authenticated by the automatic speaker verification (ASV) system through recording, speech synthesis, speech conversion, etc., in order to pass the detection of the ASV system. Among various attack methods, speech synthesis and conversion attacks pose a greater threat to the ASV system. It can use speech processing technology to convert any text or other people's voice into the target speaker's voice. And due to the emergence of deep learning, speech synthesis technology has been brought to a new level, making artificially synthesized speech almost as natural as real human speech.

[0003] Different from the acoustic features used in conventional speaker verification and speech processing, when detecting synthesized speech, special acoustic features are usually required to better distinguish real speech from synthesized speech. In order to better identify the "fraud features" in synthesized speech, it is necessary to select a suitable front-end feature extraction method. Among them, the constant Q transform (CQT) is an important time-frequency analysis method, which can represent the relationship between time and frequency on a logarithmic scale and has a higher resolution in the high-frequency region, so that it can better distinguish forged speech from real speech. In recent years, with the rapid development of deep learning, image processing, speech recognition and other fields have shined under the framework based on deep neural networks. Because it can distinguish complex nonlinear features, it greatly improves the classification accuracy of complex samples and has also achieved good results in the field of speech fraud detection. Common deep neural networks include convolutional neural networks, recurrent neural networks, generative adversarial networks, etc. However, when using a single deep neural network to extract speech features for recognition, it cannot have high accuracy.

[0004] To address this problem, the present invention proposes a Correlation Feature Discriminative Fusion Network (CFDFN) based on the front-end features of CQT in the back-end model. This network first uses a bidirectional attention network and SCG-Res2Net50 to extract high-level representations of two front-end acoustic features, and then initializes the network parameters through the Correlation Feature Discriminative Fusion Method (CFDFN). The initialization projection matrix parameters are multiplied by the original features to obtain the converted new features, and then the two features are spliced ​​to integrate the importance of different dimensions in the two features to obtain a more discriminative feature representation, and then the relationship features between them can be established. Then, the fused features are input into the fully connected layer classification, and finally the network parameters are updated using the gradient descent algorithm. This network can effectively improve the recognition performance of the forged speech detection system, and has important research significance and application value. Summary of the invention

[0005] The purpose of the present invention is to improve the accuracy of identifying forged speech and real speech, thereby proposing a forged speech detection method and system based on a discriminative fusion network of associated features.

[0006] A forged speech detection method based on a correlation feature discriminative fusion network, comprising:

[0007] Extract the CQT feature matrix from the speech data and input the CQT matrix into the recognition model. The processing of the recognition model includes:

[0008] First, the CQT matrix is ​​input into two dual parallel networks, the bidirectional attention network and SCG-Res2Net50, and two high-level representation feature matrices are obtained through the dual parallel networks, denoted as X and Y;

[0009] The bidirectional attention network processing process is as follows:

[0010] First, feature extraction is performed through the first feature extraction network to obtain the embedded features of the speech; then the embedded features of the speech are converted into a one-dimensional sequence and input into the forward GRU and reverse GRU networks to extract the temporal features, where GRU is a gate-controlled recurrent unit; the temporal features obtained by the forward GRU network and the reverse GRU network are respectively passed through the self-attention network, and then the outputs are concatenated and passed through a fully connected layer to obtain the features output by the bidirectional attention network;

[0011] For the feature matrices X and Y, the inter-class divergence matrix S of the two feature matrices is calculated respectively. bx , S by and two intra-class scatter matrices S wx , S wy; Then multiply the inverse of the intra-class scatter matrix and the inter-class scatter matrix S wx -1 S bx , which is called the joint intra-class scatter matrix, performs eigendecomposition on the joint intra-class scatter matrix to obtain the eigenvector P and eigenvalue Λ; use Project the feature matrix X to obtain the corresponding projection matrix X′=W bx X; get And project the projection matrix of the feature matrix Y to obtain the corresponding projection matrix Y'=W by Y;

[0012] Perform correlation transformation on X′ and Y′ to determine the inter-set covariance matrix S′ xy =X′Y′ T , and then use singular value decomposition to xy Diagonalize and get S′ xy =U∑V T , where U and V are orthogonal matrices after singular value decomposition, ∑ is a diagonal matrix with singular values ​​as diagonal elements; let W cx =U∑ -1 / 2 , W cy =V∑ -1 / 2 , W cx and W cy As the projection matrix of X' and Y', project X * =W cx X' and Y * =W cy Y';

[0013] Then X * and Y * The feature fusion is obtained by splicing them together, and then passes through a fully connected layer. The output dimension of the fully connected layer is 2, which are the node values ​​indicating that the sample is a fake speech and the node values ​​indicating that the sample is a real speech. That is, the final output of the recognition model is out = {neg_score, pos_score}, where neg_score is the node value of the fake speech node, and pos_score is the node value of the real speech node.

[0014] Finally, the real speech and the forged speech are classified according to the result of subtracting the two numerical values ​​in the final output out of the recognition model.

[0015] Furthermore, in the process of classifying the real speech and the forged speech according to the result of subtracting the two numerical values ​​in the final output out of the recognition model, neg_score-pos_score>0 is judged as forged speech, and pos_score-neg_score≥0 is judged as real speech.

[0016] Furthermore, before extracting the CQT feature matrix from the speech data, the speech data needs to be preprocessed. During the speech preprocessing, the input speech signal is first sampled and quantized, and then the length of the quantized speech is trimmed and padded.

[0017] Furthermore, the first feature extraction network is a ResNet18 network.

[0018] Furthermore, the SCG-Res2Net50 includes 1 convolutional layer, 4 residual layers, 1 mean pooling layer and a flatten layer, and the 4 residual layers include 3, 4, 6, and 3 Res2Net blocks respectively.

[0019] Furthermore, the recognition model is pre-trained, and the training process includes the following steps:

[0020] A training set is constructed based on the real speech and forged speech used for training, and then the training samples of the training set are input into the recognition model for processing. Each sample in the fully connected layer with an output dimension of 2 will obtain corresponding values ​​at these two nodes, and the real speech and the forged speech are classified according to the result of subtracting the two values ​​in the output out; the result of subtracting the node values ​​of the two dimensions of the network by the cross entropy loss function to determine whether the sample is real or forged speech is compared with the exact label of the sample as real or forged speech, and finally the projection matrix network parameters are updated using the gradient descent algorithm;

[0021] After training with the training set, a trained network model is obtained.

[0022] A forged speech detection system based on a correlation feature discriminative fusion network, comprising:

[0023] CQT feature matrix extraction unit: extracts CQT feature matrix from speech data;

[0024] Speech recognition unit: input the CQT matrix into the recognition model, and finally classify the real speech and the forged speech according to the result of subtracting the two values ​​in the final output out of the recognition model;

[0025] The processing process of the identification model includes:

[0026] First, the CQT matrix is ​​input into two dual parallel networks, the bidirectional attention network and SCG-Res2Net50, and two high-level representation feature matrices are obtained through the dual parallel networks, denoted as X and Y;

[0027] The bidirectional attention network processing process is as follows:

[0028] First, feature extraction is performed through the first feature extraction network to obtain the embedded features of the speech; then the embedded features of the speech are converted into a one-dimensional sequence and input into the forward GRU and reverse GRU networks to extract the temporal features, where GRU is a gate-controlled recurrent unit; the temporal features obtained by the forward GRU network and the reverse GRU network are respectively passed through the self-attention network, and then the outputs are concatenated and passed through a fully connected layer to obtain the features output by the bidirectional attention network;

[0029] For the feature matrices X and Y, the inter-class divergence matrix S of the two feature matrices is calculated respectively. bx , S by and two intra-class scatter matrices S wx , S wy ; Then multiply the inverse of the intra-class scatter matrix and the inter-class scatter matrix S wx -1 S bx , which is called the joint intra-class scatter matrix, performs eigendecomposition on the joint intra-class scatter matrix to obtain the eigenvector P and eigenvalue Λ; use Project the feature matrix X to obtain the corresponding projection matrix X′=W bx X; get And project the projection matrix of the feature matrix Y to obtain the corresponding projection matrix Y'=W by Y;

[0030] Perform correlation transformation on X′ and Y′ to determine the inter-set covariance matrix S′ xy =X′Y′ T , and then use singular value decomposition to xy Diagonalize and get S′ xy =U∑V T , where U and V are orthogonal matrices after singular value decomposition, ∑ is a diagonal matrix with singular values ​​as diagonal elements; let W cx =U∑ -1 / 2 , W cy =V∑ -1 / 2 , W cx and W cy As the projection matrix of X' and Y', project X * =W cx X' and Y * =W cy Y';

[0031] Then X * and Y *The features are concatenated to obtain the fused features, and then passed through a fully connected layer. The output dimension of the fully connected layer is 2, which respectively represent the node values ​​of the sample being judged to be fake speech and the node values ​​of the sample being judged to be real speech. That is, the final output of the recognition model is out = {neg_score, pos_score}, where neg_score is the node value of the judged fake speech, and pos_score is the node value of the judged real speech.

[0032] Furthermore, in the process of classifying the real speech and the forged speech according to the result of subtracting the two numerical values ​​in the final output out of the recognition model, neg_score-pos_score>0 is judged as forged speech, and pos_score-neg_score≥0 is judged as real speech.

[0033] Furthermore, the forged speech detection system based on the associated feature discriminant fusion network also includes a speech preprocessing module;

[0034] Speech preprocessing module: Before the CQT feature matrix extraction unit extracts the CQT feature matrix from the speech data, the speech data is preprocessed. During the speech preprocessing process, the input speech signal is first sampled and quantized, and then the quantized speech length is trimmed and padded.

[0035] Furthermore, the first feature extraction network is a ResNet18 network; the SCG-Res2Net50 includes 1 convolutional layer, 4 residual layers, 1 mean pooling layer and a flatten layer, and the 4 residual layers include 3, 4, 6, and 3 Res2Net blocks respectively.

[0036] Beneficial effects of the present invention:

[0037] The beneficial effects of the present invention are as follows: the present invention proposes a forged speech detection method and system based on a discriminative fusion network of associated features, which can effectively utilize the characteristics of two parallel networks, the bidirectional attention network and SCG-Res2Net50, to obtain more discriminative features, initialize network parameters through the discriminative fusion method of associated features, multiply the initialized projection matrix parameters with the original features to obtain the converted new features, and then connect these two high-level representation features to improve the system recognition performance. The comparative experiment was verified on the dataset ASVSpoof2019. Compared with the ordinary feature splicing feature method, the method of the present invention significantly improved the performance of the fraud detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A forged speech detection method and system flow chart based on a correlation feature discriminant fusion network;

[0039] Figure 2 A network structure diagram of the method of the present invention;

[0040] Figure 3 It is a curve diagram showing the accuracy change of the method of the present invention and the CAT direct splicing feature method at different training times;

[0041] Figure 4 This is a loss change curve diagram of the method of the present invention and the CAT direct splicing feature method at different training times. DETAILED DESCRIPTION

[0042] Specific implementation method 1: Combination Figure 1 and Figure 2 To explain this embodiment,

[0043] This embodiment is a method for detecting forged speech based on a correlation feature discriminative fusion network, comprising the following steps:

[0044] Step 1: Preprocess the speech data and extract the CQT feature matrix;

[0045] In the process of preprocessing speech, the input speech signal is first sampled and quantized, and then the length of the quantized speech is trimmed and padded.

[0046] CQT stands for Constant Q Transform.

[0047] Step 2: In the network model training stage, the CQT feature matrix is ​​used as the input of the dual parallel network, that is, the CQT feature matrix is ​​used as the input of the bidirectional attention network and SCG-Res2Net50. The network parameters are initialized by the correlation feature discriminant fusion method (CFDFN). The initialization projection matrix parameters are multiplied by the original features to obtain the converted new features. The two features are then concatenated, and the concatenated features are input into the fully connected layer for classification. Finally, the gradient descent algorithm is used to update the network parameters. The specific process includes:

[0048] Step 201, input the CQT matrix into two dual parallel networks, namely the bidirectional attention network and SCG-Res2Net50, and obtain two high-level representation feature matrices through the dual parallel networks, denoted as X and Y;

[0049] The bidirectional attention network processing process is as follows:

[0050] First, the shallow feature representation is extracted through the ResNet18 network; the ResNet18 network includes 1 convolutional layer, 8 residual blocks, 1 average pooling layer and 1 fully connected layer.

[0051] Each residual block consists of two convolutional layers and one pooling layer. It first passes through the convolutional layer and then the pooling layer. The pooling layer uses maximum pooling, and then passes through the convolutional layer again. After convolution, the data will be normalized, and the activation function uses the ReLU function.

[0052] After the feature representation of 8 residual blocks, the feature representation is passed through the average pooling layer to obtain the middle-level feature representation, and finally the middle-level feature representation is reduced in dimension through the fully connected layer to obtain the embedded features of the speech;

[0053] Then the embedded features of the speech are converted into a one-dimensional sequence and input into the forward GRU and reverse GRU networks to extract the temporal features. GRU is a gate-controlled recurrent unit.

[0054] The time series features obtained by the forward GRU network and the reverse GRU network are respectively summarized and learned with different weights through the self-attention network. The time series sequence is passed through the self-attention layer, and the time series features are given different weights for inductive learning, which can obtain a more discriminative feature representation; the outputs of the forward and reverse attention layers are spliced, and then the output features are obtained through a fully connected layer;

[0055] The SCG-Res2Net50 processing process is as follows:

[0056] SCG-Res2Net50 consists of 1 convolutional layer, 4 residual layers, 1 mean pooling layer and a flatten layer. The 4 residual layers include 3, 4, 6, and 3 Res2Net blocks respectively.

[0057] Step 202: For the feature matrices X and Y, respectively calculate the inter-class scatter matrix S of the two feature matrices bx , S by and two intra-class scatter matrices S wx , S wy .

[0058] Then multiply the inverse of the intra-class scatter matrix and the inter-class scatter matrix S wx -1 S bx , which is called the joint intra-class scatter matrix, performs eigendecomposition on the joint intra-class scatter matrix to obtain the eigenvector P and eigenvalue Λ; use Project the feature matrix X to obtain the corresponding projection matrix X′=W bx X;

[0059] In the same way, we get And project the projection matrix of the feature matrix Y to obtain the corresponding projection matrix Y'=W by Y.

[0060] Step 203: Perform correlation transformation on X′ and Y′, so that the features in one set have non-zero correlation only with the corresponding features in another set, and determine the inter-set covariance matrix S′ xy =X′Y′ T , and then use singular value decomposition (SVD) to xy Diagonalize and get S′ xy =U∑V T , where U and V are orthogonal matrices after singular value decomposition, ∑ is a diagonal matrix with singular values ​​as diagonal elements. Let W cx =U∑ -1 / 2 , W cy =V∑ -1 / 2 , W cx and W cy Project as the projection matrix of X′ and Y′ to obtain the new feature matrix X * =W cx X' and Y * =W cy Y'.

[0061] Step 204: Obtain the two transformed features X * and Y * Finally, the two features are concatenated by concatenation (CAT) to obtain the feature after feature fusion, which contains the relevant information of the two parallel network features, and can obtain a more discriminative feature representation; then pass through a fully connected layer, the input dimension of the fully connected layer is 512, and the output dimension is 2. Each sample will have corresponding values ​​at these two nodes. The first dimension is the node value indicating that the sample is a forged voice, and the second dimension is the node value indicating that the sample is a real voice. The final judgment is made according to the size of the two nodes. For forged voice, the node value of the first dimension is greater than the node value of the second dimension, and for real voice, the node value of the second dimension is greater than the node value of the first dimension; that is, the result obtained by subtracting the node value of the first dimension from the node value of the second dimension, if it is greater than zero, it is judged as real voice, and if it is less than zero, it is judged as forged voice, and binary classification is performed.

[0062] The result of determining whether the sample is real or fake speech is compared with the exact label of whether the sample is real or fake speech by subtracting the node values ​​of the two dimensions of the network through the cross entropy loss function, and finally the gradient descent algorithm is used to update the projection matrix network parameters. After the update, the joint intra-class divergence matrix will tend to the unit matrix, the diagonal elements indicate that the relationship between classes is the largest, and the non-diagonal elements indicate that the relationship between classes is not the same class is the smallest, so that classes are separated; and after the update, the inter-set covariance matrix is ​​diagonalized, and the corresponding feature correlation is strong.

[0063] Finally, a trained network model is obtained.

[0064] Step 3: In the testing phase, the trained network model is used as a classifier for real speech and fake speech to classify the speech;

[0065] During the testing phase, the last fully connected layer in the network structure is used to process the speech. The output result is an N×2-dimensional matrix of N samples. Each sample contains two-dimensional values. The first dimension is the node value indicating that the sample is a fake speech, and the second dimension is the node value indicating that the sample is a real speech.

[0066] out={neg_score,pos_score}

[0067] Among them, out is the final output of the model, neg_score is the value of the forged speech node, and pos_score is the value of the real speech node.

[0068] Step 4: classify the real speech and the forged speech according to the result of subtracting the node values ​​of the speech;

[0069] After passing through the network classifier, the two node values ​​are subtracted. If the subtraction result is greater than zero, it is judged as real speech, and if it is less than zero, it is judged as fake speech;

[0070] The following judgment rules are used to determine whether the voice is forged voice. The judgment rules are as follows:

[0071] Specific implementation method one:

[0073] This embodiment is a forged speech detection system based on a correlation feature discriminative fusion network, comprising:

[0074] Speech preprocessing module: preprocess the speech data. In the process of speech preprocessing, the input speech signal is first sampled and quantized, and then the quantized speech length is trimmed and padded.

[0075] CQT feature matrix extraction unit: extracts CQT feature matrix from speech data.

[0076] Speech recognition unit: The CQT matrix is ​​input into the recognition model, and finally the real speech and the forged speech are classified according to the result of subtracting the two numerical values ​​in the final output out of the recognition model.

[0077] The processing process of the identification model includes:

[0078] First, the CQT matrix is ​​input into two dual parallel networks, the bidirectional attention network and SCG-Res2Net50, and two high-level representation feature matrices are obtained through the dual parallel networks, denoted as X and Y;

[0079] The bidirectional attention network processing process is as follows:

[0080] First, feature extraction is performed through the first feature extraction network to obtain the embedded features of the speech; then the embedded features of the speech are converted into a one-dimensional sequence and input into the forward GRU and reverse GRU networks to extract the time series features, where GRU is a gate-controlled recurrent unit; the time series features obtained by the forward GRU network and the reverse GRU network are respectively passed through the self-attention network, and then the outputs are spliced, and then the features of the bidirectional attention network output are obtained through a fully connected layer; the first feature extraction network is a ResNet18 network;

[0081] The SCG-Res2Net50 includes 1 convolutional layer, 4 residual layers, 1 mean pooling layer and a flatten layer, and the 4 residual layers include 3, 4, 6, and 3 Res2Net blocks respectively.

[0082] For the feature matrices X and Y, the inter-class divergence matrix S of the two feature matrices is calculated respectively. bx , S by and two intra-class scatter matrices S wx , S wy ; Then multiply the inverse of the intra-class scatter matrix and the inter-class scatter matrix S wx -1 S bx , which is called the joint intra-class scatter matrix, performs eigendecomposition on the joint intra-class scatter matrix to obtain the eigenvector P and eigenvalue Λ; use Project the feature matrix X to obtain the corresponding projection matrix X′=W bx X; get And project the projection matrix of the feature matrix Y to obtain the corresponding projection matrix Y'=W by Y;

[0083] Perform correlation transformation on X′ and Y′ to determine the inter-set covariance matrix S′ xy =X′Y′ T , and then use singular value decomposition to xy Diagonalize and get S′ xy =U∑V T , where U and V are orthogonal matrices after singular value decomposition, ∑ is a diagonal matrix with singular values ​​as diagonal elements; let W cx =U∑ -1 / 2 , W cy =V∑ -1 / 2 , W cx and W cy As the projection matrix of X' and Y', project X * =Wcx X' and Y * =W cy Y';

[0084] Then X * and Y * The features are concatenated to obtain the fused features, and then passed through a fully connected layer. The output dimension of the fully connected layer is 2, which respectively represent the node values ​​of the sample being judged to be fake speech and the node values ​​of the sample being judged to be real speech. That is, the final output of the recognition model is out = {neg_score, pos_score}, where neg_score is the node value of the judged fake speech, and pos_score is the node value of the judged real speech.

[0085] In the process of classifying real speech and forged speech according to the result of subtracting two numerical values ​​in the final output out of the recognition model, neg_score-pos_score>0 is judged as forged speech, and pos_score-neg_score≥0 is judged as real speech.

[0086] Example:

[0087] The technical solution adopted by the present invention is a forged speech detection method based on a correlation feature discriminant fusion network. The method comprises the following steps:

[0088] Step 1: Preprocess the speech data and extract the CQT feature matrix;

[0089] Step 2: In the training network model stage, the CQT feature matrix is ​​used as the input of the dual parallel network, that is, the CQT feature matrix is ​​used as the input of the bidirectional attention network and SCG-Res2Net50. The network parameters are initialized by the associated feature discriminant fusion method (CFDFN). The initialized projection matrix parameters are multiplied by the original features to obtain the converted new features. The two features are then concatenated, and the concatenated features are input into the fully connected layer for classification. Finally, the gradient descent algorithm is used to update the network parameters.

[0090] Step 3: In the testing phase, the trained network model is used as a classifier for real speech and fake speech to classify the speech;

[0091] Step 4: Classify the real speech and the forged speech according to the result of subtracting the node values ​​of the speech.

[0092] In this example, the specific operation process of step 1 is:

[0093] Step 101, pre-processing the speech: sampling and quantizing the input speech signal, and trimming and padding the length of the quantized speech;

[0094] Step 102: extract CQT features from the preprocessed speech.

[0095] In this example, the specific operation process of step 2 is:

[0096] Step 201, using the CQT feature matrix as the input of the deep neural network, first input the CQT matrix into the two parallel networks of the bidirectional attention network and SCG-Res2Net50, where the bidirectional attention network extracts shallow feature representation by the ResNet18 network. The ResNet18 network consists of 1 convolutional layer, 8 residual blocks, and 1 fully connected layer. Each residual block consists of two convolutional layers, and the pooling layer uses the maximum pooling. After the convolution, the data will be normalized, and the activation function uses the ReLU function; after the feature representation of multiple residual blocks, the shallow feature representation is obtained by the average pooling layer to obtain the middle feature representation, and finally the middle feature representation is reduced in dimension through the fully connected layer to obtain the embedded features of the speech; the embedded features of the speech are converted into a one-dimensional sequence, and are respectively input into the forward gated recurrent unit (GRU) and the reverse GRU network to extract the temporal features; the temporal features obtained by the forward GRU network and the reverse GRU network are respectively summarized and learned with different weights through the self-attention network. By passing the time series through the self-attention layer, the time series features are given different weights for inductive learning, which can obtain more discriminative feature representations; the outputs of the forward and reverse attention layers are concatenated and then passed through a fully connected layer to obtain the output features; SCG-Res2Net50 consists of 1 convolutional layer, 4 residual layers, 1 average pooling layer and a flatten layer. The 4 residual layers include 3, 4, 6, and 3 Res2Net blocks respectively.

[0097] Step 202: Through the dual parallel networks, two high-level representation feature matrices are obtained, represented as X and Y, and the inter-class divergence matrix S of the two feature matrices is calculated respectively. bx , S by and two intra-class scatter matrices S wx , S wy .

[0098] Then the inverse of the intra-class scatter matrix is ​​multiplied by the inter-class scatter matrix, which is called the joint intra-class scatter matrix S wx -1 S bx , which can be expressed as:

[0099]

[0100] Among them, S bx is the inter-class scatter matrix of X, p represents the dimension, n is the number of columns of the matrix, n i represents the number of samples in the i-th category, and Respectively represent the mean of the vector of the i-th category and the mean of the vector in the entire feature set, that is, c is the number of categories in the feature, S wx is the intra-class scatter matrix of X.

[0101] Then, we perform eigendecomposition on the joint intra-class scatter matrix, find the transformation that diagonalizes it, find the eigenvector P and eigenvalue Λ, and find the most significant eigenvector of the inter-class scatter matrix. We only need to find the covariance matrix Φ wbx If the classes are well separated, Φ wbx will be a diagonal matrix, since the initial Φ wbx is a symmetric positive semidefinite matrix, and we can find a transformation that diagonalizes it:

[0102]

[0103] Where P is Φ wbx The characteristic vector of is Φ wbx The characteristic value of .

[0104] Therefore, It is expressed as the projection matrix of the feature matrix x, the purpose is to transform S wx -1 S bx Unify, and finally the new feature after projection processing is X′. The transformation formula is:

[0105] W bx T S wx -1 S bx W bx =I (3)

[0106] X'=W bx T X (4)

[0107] Similarly, the joint intra-class divergence matrix S wy -1 S by The same transformation is performed to find the projection matrix, and finally we get:

[0108] W by T S wy -1 S by W by =I (5)

[0109] Y'=W by Y (6)

[0110] Step 203: Perform correlation transformation on the newly transformed features X′ and Y′. It is necessary to make the features in one set have non-zero correlation only with the corresponding features in the other set. To achieve this, it is necessary to diagonalize the inter-set covariance matrix S′ of the transformed feature set. xy , where the inter-set covariance matrix S′ xy It can be expressed as:

[0111] S′ xy =X′Y′ T (7)

[0112] Singular value decomposition (SVD) is used to xy Diagonalize and we get:

[0113] S′ xy =U∑V T (8)

[0114] Where U and V are orthogonal matrices after singular value decomposition, ∑ is a diagonal matrix with singular values ​​as diagonal elements. cx =U∑ -1 / 2 , W cy =V∑ -1 / 2 , get the transformation matrix of X and Y:

[0115] X * =W cx T X'=A x ×X (9)

[0116] Y * =W cy T Y'=A y ×Y (10)

[0117] Among them A x is the projection matrix of the original feature X, A y is the projection matrix of the original feature Y. During back propagation, the two projection matrices are updated by the gradient descent algorithm.

[0118] Step 204: After obtaining the two transformed features, the two features are concatenated by concatenation (CAT) to obtain a feature fusion feature, wherein the CAT concatenation formula is:

[0119]

[0120] Among them, Z is the new feature after feature fusion, which contains the relevant information of the two parallel network features, and can obtain a more discriminative feature representation, and then passes through a fully connected layer.

[0121] In this example, the specific operation process of step 3 is:

[0122] During the testing phase, the trained network model is used as a classifier for real speech and fake speech, and two node values ​​are obtained: neg_score is the fake speech node value, and pos_score is the real speech node value.

[0123] In this example, the specific operation process of step 4 is:

[0124] After passing through the network classifier, the two node values ​​are subtracted. If the subtraction result is greater than zero, it is judged as real speech, and if it is less than or equal to zero, it is judged as fake speech. The judgment rules are as follows:

[0125]

[0126] Experimental results:

[0127] The present invention uses the logical attack (LA) dataset of the 2019 Automatic Speaker Recognition Fraud Attack and Defense Countermeasures Challenge (ASVspoof2019) database for performance verification, and its task is to identify the synthesized converted speech and the real speech. The performance evaluation indicators are the tandem-minimum cost detection function (t-DCF) and the equal error rate (EER), both of which are smaller values, indicating better performance. The performance comparison of different methods is shown in Table 1, where methods 1 and 2 use LFCC and CQCC as front-end features, respectively, and use Gaussian mixture mixture model (GMM) for classification; methods 3 and 4 use SincNet+VGG model and RawNet2 model to process and classify speech respectively; method 5 first extracts CQT as the front-end feature, and then extracts high-level features through a parallel network, and directly performs splicing fusion (CAT) on the features; method 6 is a forged speech detection method based on the discriminative fusion network of associated features proposed by the present invention.

[0128] Table 1 Performance comparison of different methods

[0129]

[0130]

[0131] The experimental results show that compared with the classic method, the proposed method significantly reduces the t-DCF and EER of synthetic speech fraud detection. The t-DCF and EER of the proposed method are 0.1225 and 3.07% respectively. Compared with the common feature concatenation feature CAT method, as the number of training times increases, the changes in the accuracy and loss of the development set are as follows: Figure 3 , Figure 4 As shown, the accuracy of the present invention increases faster and the loss decreases faster. It can be seen from the experimental results that the forged speech detection method and system based on the associated feature discriminant fusion network proposed by the present invention can effectively improve the performance of the fraud detection system.

[0132] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A forged speech detection method based on a correlation feature discriminant fusion network, characterized in that: include: Extract the CQT feature matrix from the speech data and input the CQT matrix into the recognition model. The processing of the recognition model includes: First, the CQT matrix is ​​input into two dual parallel networks, the bidirectional attention network and SCG-Res2Net50, and two high-level representation feature matrices are obtained through the dual parallel networks, denoted as X and Y; The bidirectional attention network processing process is as follows: First, feature extraction is performed through the first feature extraction network to obtain the embedded features of the speech; then the embedded features of the speech are converted into a one-dimensional sequence and input into the forward GRU and reverse GRU networks to extract the temporal features, where GRU is a gate-controlled recurrent unit; the temporal features obtained by the forward GRU network and the reverse GRU network are respectively passed through the self-attention network, and then the outputs are concatenated and passed through a fully connected layer to obtain the features output by the bidirectional attention network; For the feature matrices X and Y, the inter-class divergence matrix S of the two feature matrices is calculated respectively. bx , S by and two intra-class scatter matrices S wx , S wy ; Then multiply the inverse of the intra-class scatter matrix and the inter-class scatter matrix S wx -1 S bx , which is called the joint intra-class scatter matrix, performs eigendecomposition on the joint intra-class scatter matrix to obtain the eigenvector P and eigenvalue Λ; use Project the feature matrix X to obtain the corresponding projection matrix X′=W bx X; get And project the projection matrix of the feature matrix Y to obtain the corresponding projection matrix Y'=W by Y; Perform correlation transformation on X′ and Y′ to determine the inter-set covariance matrix S′ xy =X′Y′ T , and then use singular value decomposition to xy Diagonalize and get S′ xy =U∑V T , where U and V are orthogonal matrices after singular value decomposition, ∑ is a diagonal matrix with singular values ​​as diagonal elements; let W cx =U∑ -1 / 2 , W cy =V∑ -1 / 2 , W cx and W cy As X ′ and Y ′ Projection matrix is ​​used to obtain X * =W cx X' and Y * =W cy Y'; Then X * and Y * The feature fusion is obtained by splicing them together, and then passes through a fully connected layer. The output dimension of the fully connected layer is 2, which are the node values ​​indicating that the sample is a fake speech and the node values ​​indicating that the sample is a real speech. That is, the final output of the recognition model is out = {neg_score, pos_score}, where neg_score is the node value of the fake speech node, and pos_score is the node value of the real speech node. Finally, the real speech and the forged speech are classified according to the result of subtracting the two numerical values ​​in the final output out of the recognition model.

2. According to claim 1, a method for detecting forged speech based on a correlation feature discriminative fusion network is characterized in that: In the process of classifying real speech and forged speech according to the result of subtracting two numerical values ​​in the final output out of the recognition model, neg_score-pos_score>0 is judged as forged speech, and pos_score-neg_score≥0 is judged as real speech.

3. The method for detecting forged speech based on a correlation feature discriminative fusion network according to claim 1, characterized in that: Before extracting the CQT feature matrix from speech data, the speech data needs to be preprocessed. During the speech preprocessing process, the input speech signal is first sampled and quantized, and then the quantized speech length is trimmed and padded.

4. The method for detecting forged speech based on a correlation feature discriminative fusion network according to claim 1, characterized in that: The first feature extraction network is a ResNet18 network.

5. The method for detecting forged speech based on a correlation feature discriminative fusion network according to claim 1, characterized in that: The SCG-Res2Net50 includes 1 convolutional layer, 4 residual layers, 1 mean pooling layer and a flatten layer, and the 4 residual layers include 3, 4, 6, and 3 Res2Net blocks respectively.

6. A forged speech detection method based on a correlation feature discriminative fusion network according to any one of claims 1 to 5, characterized in that: The recognition model is pre-trained, and the training process includes the following steps: A training set is constructed based on the real speech and forged speech used for training, and then the training samples of the training set are input into the recognition model for processing. Each sample in the fully connected layer with an output dimension of 2 will obtain corresponding values ​​at these two nodes, and the real speech and the forged speech are classified according to the result of subtracting the two values ​​in the output out; the result of subtracting the node values ​​of the two dimensions of the network by the cross entropy loss function to determine whether the sample is real or forged speech is compared with the exact label of the sample as real or forged speech, and finally the projection matrix network parameters are updated using the gradient descent algorithm; After training with the training set, a trained network model is obtained.

7. A forged speech detection system based on a correlation feature discriminative fusion network, characterized in that: include: CQT feature matrix extraction unit: extracts CQT feature matrix from speech data; Speech recognition unit: input the CQT matrix into the recognition model, and finally classify the real speech and the forged speech according to the result of subtracting the two values ​​in the final output out of the recognition model; The processing process of the identification model includes: First, the CQT matrix is ​​input into two dual parallel networks, the bidirectional attention network and SCG-Res2Net50, and two high-level representation feature matrices are obtained through the dual parallel networks, denoted as X and Y; The bidirectional attention network processing process is as follows: First, feature extraction is performed through the first feature extraction network to obtain the embedded features of the speech; then the embedded features of the speech are converted into a one-dimensional sequence and input into the forward GRU and reverse GRU networks to extract the temporal features, where GRU is a gate-controlled recurrent unit; the temporal features obtained by the forward GRU network and the reverse GRU network are respectively passed through the self-attention network, and then the outputs are concatenated and passed through a fully connected layer to obtain the features output by the bidirectional attention network; For the feature matrices X and Y, the inter-class divergence matrix S of the two feature matrices is calculated respectively. bx , S by and two intra-class scatter matrices S wx , S wy ; Then multiply the inverse of the intra-class scatter matrix and the inter-class scatter matrix S wx -1 S bx , which is called the joint intra-class scatter matrix, performs eigendecomposition on the joint intra-class scatter matrix to obtain the eigenvector P and eigenvalue Λ; use Project the feature matrix X to obtain the corresponding projection matrix X′=W bx X; get And project the projection matrix of the feature matrix Y to obtain the corresponding projection matrix Y'=W by Y; Perform correlation transformation on X′ and Y′ to determine the inter-set covariance matrix S′ xy =X′Y′ T , and then use singular value decomposition to xy Diagonalize and get S′ xy =U∑V T , where U and V are orthogonal matrices after singular value decomposition, ∑ is a diagonal matrix with singular values ​​as diagonal elements; let W cx =U∑ -1 / 2 , W cy =V∑ -1 / 2 , W cx and W cy As the projection matrix of X' and Y', project X * =W cx X' and Y * =W cy Y'; Then X * and Y * The features are concatenated to obtain the fused features, and then passed through a fully connected layer. The output dimension of the fully connected layer is 2, which respectively represent the node values ​​of the sample being judged to be fake speech and the node values ​​of the sample being judged to be real speech. That is, the final output of the recognition model is out = {neg_score, pos_score}, where neg_score is the node value of the judged fake speech, and pos_score is the node value of the judged real speech.

8. A forged speech detection system based on a correlation feature discriminative fusion network according to claim 7, characterized in that: In the process of classifying real speech and forged speech according to the result of subtracting two numerical values ​​in the final output out of the recognition model, neg_score-pos_score>0 is judged as forged speech, and pos_score-neg_score≥0 is judged as real speech.

9. A forged speech detection system based on a correlation feature discriminative fusion network according to claim 7 or 8, characterized in that: The system also includes a speech preprocessing module; Speech preprocessing module: Before the CQT feature matrix extraction unit extracts the CQT feature matrix from the speech data, the speech data is preprocessed. During the speech preprocessing process, the input speech signal is first sampled and quantized, and then the quantized speech length is trimmed and padded.

10. A forged speech detection system based on a correlation feature discriminative fusion network according to claim 9, characterized in that: The first feature extraction network is a ResNet18 network; the SCG-Res2Net50 includes 1 convolutional layer, 4 residual layers, 1 mean pooling layer and a flatten layer, and the 4 residual layers include 3, 4, 6, and 3 Res2Net blocks respectively.