A voice fraud detection method based on feature fusion and single classification

Abstract

The present invention discloses a voice fraud detection method based on feature fusion and single classification, and the method comprises the following steps: S1: constructing an enhanced voice training set; S2: extracting the Log-Mel spectrum of the voice to be detected, using TRawNet to extract the time domain features of the voice to be detected, and fusing the two features; S3: sending the fused features to a graph attention network, modeling the information in different time domains and frequency domains, and training the entire network with a single classification loss function. The method uses a feature fusion method based on a graph attention network to combine the time domain features and frequency domain features of the voice, and models the relationship between different segments to extract more discriminative features; the proposed single classification loss function not only retains the acoustic diversity of the real voice, but also solves the problem of decreased detection capability caused by the channel effect, so that the generalization ability of the model is further enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of voiceprint recognition and deep learning, and relates to a voice fraud detection method, and specifically to a voice fraud detection method based on feature fusion and single classification. Background Art

[0002] In today's information age, how to scientifically and accurately identify a person's identity and protect information security and privacy has received special attention. The existing technologies for identity recognition using biometrics include face recognition, iris recognition, fingerprint recognition, speaker recognition, etc. The Automatic Speaker Verification (ASV) system has also emerged as a result. The system uses a specific algorithm to perform pattern recognition and matching on the input voice to determine whether the verified speaker's voice is the voice of a legitimate user.

[0003] However, the ASV system is vulnerable to fraud attacks. An attacker can record the target speaker's voice through a recording device and replay it to the ASV system to achieve a replay attack on the ASV system. With the development of artificial intelligence, attackers can use text-to-speech (TTS) and voice conversion (VC) algorithms to synthesize the target speaker's voice. These attacks have brought many threats to people's production and life, such as using synthesized voice to control smart IoT devices, send text messages or visit websites, steal personal data, and extort, which endangers social security.

[0004] There are a lot of studies on voice anti-fraud, which are mainly divided into two aspects: artificially designed acoustic features and neural network-based methods. Artificially designed acoustic features are carefully designed to ensure that the features can contain as comprehensive information as possible, such as constant Q cepstral coefficients (CQCC), linear frequency cepstral coefficients (LFCC), Mel frequency cepstral coefficients (MFCC), etc. The neural network-based method uses convolutional neural networks (CNN), residual networks (ResNet), etc. to extract voice features, and then sends the extracted voice features to the classifier to determine whether the voice is fraudulent.

[0005] Although current voice anti-fraud methods perform well on certain datasets, their generalization ability is generally poor. When tested using other datasets, their detection performance will drop significantly. One of the reasons is that the performance of voice anti-fraud methods depends largely on the information captured in the features. The more information the features capture, the easier it is for the system to detect fraud attacks, while artificially designed acoustic features may lose some information. In addition, the relationship between information in different time periods and frequency bands of speech can be used to distinguish real speech from fraudulent speech, but feature networks based on CNN and ResNet find it difficult to model these relationships.

[0006] Another reason for the poor generalization ability of speech anti-fraud methods is that most methods formulate the speech anti-fraud problem as a binary classification problem, that is, judging whether the speech to be detected is real speech or fraudulent speech. However, binary classification requires that the distribution of test data is the same or similar to the distribution of training data. Although this assumption is reasonable for real speech, it cannot be used for fraudulent speech. The rapid development of speech synthesis technology has caused existing anti-fraud methods to face the problem that the distribution of fraudulent speech is inconsistent with the distribution in the training set. At present, some studies have proposed single-classification speech anti-fraud methods, such as OC-Softmax and SAMO. These methods set the real speech as the target class, specifically learn the distribution of the target class, and set classification boundaries around it to avoid the distribution problem of fraudulent speech. These single-classification-based methods have achieved good performance on the ASVspoof2019 LA dataset, but they still have room for improvement: First, the real speech of different speakers will naturally form multiple clusters in the feature space. Some fraudulent speech may be located between these real speech clusters. Learning only one classification boundary will lose the acoustic diversity of the real speech and may also misjudge some fraudulent speech as real speech, thus posing a huge threat to the ASV system. Secondly, although the distribution of real speech in the training set and the test set is similar, the channel effect caused by the telecommunications network audio codec or device impulse response will still have a certain impact on the data distribution, thereby reducing the detection ability of the model. Summary of the invention

[0007] In view of the above-mentioned defects in the prior art, the present invention provides a voice fraud detection method based on feature fusion and single classification. This method uses a feature fusion method based on a graph attention network to combine the time domain features and frequency domain features of speech, and models the relationship between different segments to extract more discriminative features; the proposed single classification loss function not only retains the acoustic diversity of real speech, but also solves the problem of reduced detection ability caused by channel effects, so that the generalization ability of the model is further enhanced.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] A voice fraud detection method based on feature fusion and single classification includes the following steps:

[0010] Step S1: generating enhanced data of real speech according to an open source data set and a channel simulator, and constructing an enhanced speech training set, wherein the enhanced speech training set includes original real speech and enhanced real speech;

[0011] Step S2: Extract the Log-Mel spectrum of the speech to be detected s , and use the time domain feature extraction network TRawNet to extract the time domain features of the speech to be detected t , f s and f t After fusion, the features are obtained and

[0012] Step S3: The fused features and It is sent to the Graph Attention Network (GAT) to model the information in different time domains and frequency domains to obtain the deep feature g of speech. * , and train the entire network with a single classification loss function so that the model learns the distribution of real speech.

[0013] Compared with the prior art, the present invention has the following advantages:

[0014] (1) The feature fusion method based on graph attention network designed by the present invention can solve the problem of information loss of artificially designed acoustic features, and at the same time, it can also model the relationship between information in different time periods and frequency bands to obtain features with more discriminative ability;

[0015] (2) The present invention designs a new single-classification loss function, which can maintain the acoustic diversity of real speech under the single-classification learning objective, while solving the problem of decreased model detection ability caused by channel effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a voice fraud detection method based on feature fusion and single classification;

[0017] Figure 2 This is the overall framework diagram of the network;

[0018] Figure 3 This is the structural diagram of the time domain feature extraction network TRawNet. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0020] The present invention provides a voice fraud detection method based on feature fusion and single classification, such as Figure 1 and Figure 2 As shown, the method comprises the following steps:

[0021] Step S1: Generate enhanced data of real speech according to the open source data set and the channel simulator, and construct an enhanced speech training set, wherein the enhanced speech training set includes original real speech and enhanced real speech.

[0022] In this step, enhanced data of real speech is generated to simulate the influence of channel effect on real speech, and the generated enhanced data is used to train a model with stronger generalization ability.

[0023] Step S2: Extract the Log-Mel spectrum of the speech to be detected s , and use the time domain feature extraction network TRawNet to extract the time domain features of the speech to be detected t , f s and f t After fusion, the features are obtained and The specific steps are as follows:

[0024] Step S201: Use short-time Fourier transform and Mel filter bank to extract the Log-Mel spectrum of the speech to be detected. s ;

[0025] Step S202: Use the time domain feature extraction network TRawNet to extract the time domain features f of the speech to be detected t , the TRawNet consists of a convolutional layer and 5 residual blocks with similar structures;

[0026] Step S203: transform f from the time domain dimension and the frequency domain dimension respectively. s and f t Fusion, get the fused features and

[0027] Step S3: The fused features and Send it to GAT to model the information in different time domains and frequency domains to obtain the speech depth feature g * , and use a single classification loss function to train the entire network so that the model learns the distribution of real speech. The specific steps are as follows:

[0028] Step S301: The fused features and They are respectively sent to the graph attention network G t and G s In the above example, the relationship between different time periods and frequency bands is modeled to obtain the feature g s and g t , these two features are fused and then sent to the graph attention network G st In the above example, we get the speech depth feature g * , the formula is as follows:

[0029] Step S302: The speech depth feature g obtained in step S301 is * The entire network is trained using a single classification loss function. The formula is as follows:

[0030]

[0031] Where n is the number of training data, x i is the i-th training data, is x i The mean of the k neighbors of , λ is the weight parameter; and They represent the original speech deep feature set and the enhanced speech deep feature set respectively; MMD is the maximum mean discrepancy (MMD).

[0032] Step S303: For the test data, the following indicators are calculated to obtain the detection score:

[0033]

[0034] Among them, N k (F(x)) is the set of k neighbors of speech x, and F(·) is the entire feature extraction network. When the score is less than or equal to the threshold, the test speech is judged to be real speech; when the score is greater than the threshold, the test speech is judged to be fraudulent speech.

[0035] In this step, the graph attention network consists of three parts: processing fusion features Graph Attention Network G for Modeling Relationships in Different Time Periods t ; Processing fusion features Graph Attention Network G for Modeling Relationships between Different Frequency Bands s ; After fusing the outputs of the two graph attention networks, we get the fused graph g and feed it into the graph attention network G st Get the speech depth feature g * .

[0036] In this step, the first term of the single-class loss function groups the speech belonging to the same speaker together to solve the problem of speaker diversity in real speech; the second term of the single-class loss function measures the similarity of the distribution of the original real speech and the enhanced real speech, so that the features of the enhanced real speech extracted by the model are as similar as possible to the features of the original real speech, so as to improve the generalization ability of the model.

[0037] Example:

[0038] Step S1: Generate enhanced data of real speech from three aspects based on open source data sets and channel simulators: background noise, audio codec, and impulse response of the device. The original real speech and enhanced real speech form a training set. During training, L speech pieces are randomly selected and sent to the network. The L speech pieces include at least one original real speech and one enhanced real speech.

[0039] Step S2: Preprocess the speech to be detected, intercept the speech of about 4 seconds in length, and perform padding operation on the speech of less than 4 seconds in length. Then use short-time Fourier transform and Mel filter bank to extract Log-Mel spectrum. The specific steps are as follows:

[0040] First, calculate the frequency domain features of the speech, the Log-Mel spectrum:

[0041] f s =log 10 [Mel(STFT(x))]

[0042] in, is the extracted Log-Mel spectrum, M represents the frequency domain dimension, N represents the time domain dimension; x is the input speech signal, Mel(·) is the Mel filter bank, expressed as:

[0043]

[0044] Where f is the frequency in Hz.

[0045] STFT is short-time Fourier transform, expressed as:

[0046]

[0047] Wherein, x(n) is the input signal, w(n) is the window function, and X(t,w) is the spectrum of the speech. In this embodiment, the frame size is set to 1024, the frame jump size is set to 512, and the number of Mel filter banks is 128.

[0048] Then, the speech is fed into the TRawNet convolutional network to extract the time domain features of the speech. t :

[0049] f t=TRawNet(x)

[0050] Among them, x is the input speech, With f s The shape of TRawNet is the same as that of the original image. TRawNet consists of 5 residual blocks with similar structures. The residual blocks are used to process input features to extract frame-level embedding. The residual blocks include convolutional layers with feature maps, regularization layers, LeakyReLU activation layers, and maximum pooling layers. The parameters of the convolutional layers are (3,3,128), (3,1,128), and (3,1,256), and the parameters of the maximum pooling layers are 3 and 12. The specific structure is as follows: Figure 3 shown.

[0051] Since the Log-Mel spectrum is based on the characteristics of human hearing and is insensitive to the high-frequency part, this part of information will be lost. Therefore, the feature f is calculated in the frequency domain dimension. s and f t Fusion is performed to make up for the information lost by Log-Mel at high frequencies:

[0052]

[0053] At the same time, f s and f t Fusion is performed to obtain more informative time-frequency acoustic features:

[0054]

[0055] For the fused features,

[0056] Step S3: Send the fused features into the graph attention network to obtain the deep features of the speech, that is, for the fused features The frequency domain dimension is regarded as a node, and the time domain dimension is regarded as a node feature and sent to the graph attention network G. s In the above example, the information relationship in different frequency bands is modeled to obtain the output g s :

[0057]

[0058] Similarly, for the fusion feature The time domain dimension is regarded as a node, and the frequency domain dimension is regarded as a node feature and sent to the graph attention network G. t In the above example, the information relationship at different time periods is modeled to obtain the output g t :

[0059]

[0060] in, and Ns and N t represents the number of nodes, and D is the feature dimension. s and g t Send it to the graph pooling layer to select a subset of nodes containing more information to generate a more discriminative graph and The two images are fused into one image g by element-by-element multiplication, and then sent to the graph attention network again to extract the speech deep feature g * :

[0061]

[0062] According to the obtained speech depth feature g * , calculate the single classification loss function:

[0063]

[0064] Where n is the number of voices in the training set, x i represents the i-th speech in the training set, Represents voice x i The first term of the loss function causes the network to group the speech belonging to the same speaker together and separate the speech belonging to different speakers, thus forming multiple hyperspheres in the feature space. In this embodiment, the value of k is 5.

[0065] The second term of the loss function, whose S and T are and That is, the deep features of the original real speech and the enhanced real speech; MMD is the maximum mean difference, which is used to measure the distribution similarity between the original real speech and the enhanced real speech, expressed as:

[0066]

[0067] Among them, x and y are the speech depth features in S and T, k(·,·) is the Gaussian kernel function, and m and n correspond to the number of speech contained in S and T, that is, m = s n , n = t n .

[0068] For the test data, the following indicators are calculated to obtain the detection score:

[0069]

[0070] Among them, N k (F(x)) is the set of k neighbors of speech x, and F(·) is the entire feature extraction network. When the score is less than or equal to the threshold, the test speech is judged to be a real speech; when the score is greater than the threshold, the test speech is judged to be a fraudulent speech.

Claims

1. A voice fraud detection method based on feature fusion and single classification, characterized in that The method comprises the following steps: Step S1: Generate enhanced data of real speech based on the open source data set and the channel simulator, and construct an enhanced speech training set; Step S2: Extract the Log-Mel spectrum of the speech to be detected s , and use the time domain feature extraction network TRawNet to extract the time domain features of the speech to be detected t , f s and f t After fusion, the features are obtained and M represents the frequency domain dimension, and N represents the time domain dimension; Step S3: The fused features and It is sent to the graph attention network to model the information in different time domains and frequency domains to obtain the deep feature g of speech. * , and use a single classification loss function to train the entire network so that the model learns the distribution of real speech. The specific steps are as follows: Step S301: The fused features and They are respectively sent to the graph attention network G t and G s In the above example, the relationship between different time periods and frequency bands is modeled to obtain feature g s and g t , these two features are fused and then sent to the graph attention network G st In the above example, we get the speech depth feature g * , the formula is as follows: g s , g t , g * The calculation formula is as follows: g * =G st (g s ||g t ) in, and N s and N t represents the number of nodes, and D is the feature dimension; Step S302: The speech depth feature g obtained in step S301 is * The entire network is trained using a single classification loss function. The formula is as follows: Where n is the number of training data, x i is the i-th training data, is x i The mean of the k neighbors of , λ is the weight parameter; and They represent the original speech deep feature set and the enhanced speech deep feature set respectively; MMD is the maximum mean difference loss function; The calculation formula of the MMD is as follows: Among them, x and y are the speech depth features in S and T, k(·,·) is the Gaussian kernel function, and m and n correspond to the number of speech contained in S and T, that is, m = s n , n = t n ; Step S303: For the test data, the following indicators are calculated to obtain the detection score: Among them, N k (F(x)) is the set of k neighbors of speech x, and F(·) is the entire feature extraction network. When the score is less than or equal to the threshold, the test speech is judged to be real speech; when the score is greater than the threshold, the test speech is judged to be fraudulent speech.

2. The voice fraud detection method based on feature fusion and single classification according to claim 1 is characterized in that The enhanced speech training set includes original real speech and enhanced real speech.

3. The voice fraud detection method based on feature fusion and single classification according to claim 1 is characterized in that The specific steps of step S2 are as follows: Step S201: Use short-time Fourier transform and Mel filter bank to extract the Log-Mel spectrum of the speech to be detected. s ; Step S202: Use the time domain feature extraction network TRawNet to extract the time domain features f of the speech to be detected t , the TRawNet consists of a convolutional layer and 5 residual blocks with similar structures; Step S203: transform f from the time domain dimension and the frequency domain dimension respectively. s and f t Fusion, get the fused features and 4. The voice fraud detection method based on feature fusion and single classification according to claim 1 or 3, characterized in that The f s and f t The calculation formula is as follows: f s =log 10 [Mel(STFT(x))] f t =TRawNet(x) in, x is the input speech signal, Mel(·) is the Mel filter bank, and STFT is the short-time Fourier transform.

5. The voice fraud detection method based on feature fusion and single classification according to claim 4 is characterized in that The Mel(·) is expressed as: Where f is the frequency in Hz.

6. The voice fraud detection method based on feature fusion and single classification according to claim 4 is characterized in that The STFT is expressed as: Among them, x(n) is the input signal, w(n) is the window function, and X(t,w) is the spectrum of the speech.

Images