Noise robust fake speech detection system based on joint training and method thereof

Abstract

The application discloses a noise robust fake speech detection system based on joint training, comprising: a speech enhancement module for enhancing the original input noisy speech by using a deep neural network to remove background noise; a speech fusion module for linearly weighting and fusing the enhanced speech and the original input noisy speech to obtain fused speech, realize spectrum compensation, and save fake traces; a fake detection module for extracting log power spectrum features from the fused speech, modeling based on low-frequency features, and obtaining a true or false prediction result of the speech; and a joint training module for jointly training and optimizing the speech enhancement module and the fake detection module. The application also discloses a noise robust fake speech detection method based on joint training. The application introduces speech enhancement technology in the field of fake speech detection, and adopts a speech fusion module and a joint training module to improve the effect of speech enhancement, so that the fake speech detection system can maintain robustness in a noisy environment.

Description

Technical Field

[0001] This invention relates to the field of signal processing technology, and in particular to a noise-robust forged speech detection system and method based on joint training. Background Technology

[0002] Over the past decade, with the development of deep learning, advanced text-to-speech synthesis and speech conversion technologies have enabled the generation of fake audio that is virtually indistinguishable from real speech. Fake speech detection systems can effectively prevent this phenomenon, and many organizations and competitions have conducted comprehensive research on fake speech detection technology, primarily focusing on improving system performance in noise-free scenarios. The powerful modeling capabilities of convolutional neural networks have greatly enhanced system performance. However, even state-of-the-art fake speech detection systems trained under noise-free conditions experience a significant performance drop when faced with noisy backgrounds, and this performance deteriorates further as the signal-to-noise ratio (SNR) decreases.

[0003] To improve the noise robustness of fake audio detection systems, researchers have proposed various methods, such as data augmentation or trying new training strategies. Current research confirms that fake speech exhibits different characteristics in the low-frequency range (0–4kHz) and the high-frequency range (4kHz–8kHz). Low-frequency features can significantly reduce the error rate of fake speech detection and improve system performance.

[0004] Speech enhancement refers to extracting the purest possible target speech from noisy speech. It is widely used in speech recognition to preprocess speech in noisy environments, but it has not yet been applied to speech spoofing detection. Deep learning-based speech enhancement leverages the powerful modeling capabilities of deep learning to effectively learn the mapping relationship between noisy speech and the target speech signal. This allows any input noisy speech signal to be processed by the established enhancement model to output a denoised speech signal, thus achieving speech enhancement. However, the biggest problem with speech enhancement is the distortion of the enhanced speech. Speech distortion results in the loss of much important information. If directly applied to speech spoofing detection, this distortion can also lead to the loss of traces of forgery, impacting system performance.

[0005] Therefore, there is an urgent need to provide a new type of fake voice detection system and method to solve the above problems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a noise-robust speech spoofing detection system and method based on joint training, which introduces speech enhancement technology to maintain the robustness of the speech spoofing detection system in noisy background environments.

[0007] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: to provide a noise-robust forged speech detection system based on joint training, comprising: The speech enhancement module is used to enhance the original noisy input speech using a deep neural network to remove most of the background noise. The speech fusion module is used to linearly weight and fuse the enhanced speech with the original input noisy speech to obtain fused speech, achieve spectrogram compensation, and save forgery traces; The forgery detection module is used to extract log power spectrum features from the fused speech, model based on low-frequency features, and obtain the prediction results of the authenticity of the speech. A joint training module is used to jointly train and optimize the speech enhancement module and the forgery detection module.

[0008] In a preferred embodiment of the present invention, the step of the voice enhancement module to enhance the input voice signal includes: First, a short-time Fourier transform is performed on the original input noisy speech signal to transform the time-domain signal into a time-frequency domain signal, thus obtaining the amplitude spectrum and phase spectrum of the original input speech. Then, the amplitude spectrum of the original input speech is modeled and trained by a deep neural network. Its output is the enhanced amplitude spectrum. The mean square error between the enhanced amplitude spectrum and the true amplitude spectrum is calculated as the training objective function. Finally, an inverse Fourier transform is performed on the phase spectrum of the original input speech and the amplitude spectrum of the enhanced speech to obtain the enhanced reconstructed speech.

[0009] In a preferred embodiment of the present invention, the specific steps of the forgery detection module in determining the authenticity of the fused speech include: First, regarding fused speech X fusion The complex spectrum is obtained by performing a Fourier transform. : (5) Secondly, perform complex spectrum operations to extract logarithmic power spectrum features. : (6) Then, the low-frequency band (0-4kHz) logarithmic power spectrum features are extracted as input to the forgery detection module: (7) Finally, the forgery detection module, based on a residual convolutional neural network, obtains the final output score. : (8) in, This represents a mapping function based on a residual convolutional neural network.

[0010] In a preferred embodiment of the present invention, the joint training module linearly combines the objective functions of the speech enhancement module and the forgery detection module with a weight as the final objective function, simultaneously optimizing the two modules and promoting each other, so that the speech enhancement module is optimized in a direction that is beneficial to forgery detection.

[0011] Furthermore, the objective function of the forgery detection module is an A-Softmax loss based on angle margin: (9) (10) in, For the sample size, It is the first The feature vector of each sample Corresponding tags θ j,i (0≤) θ j,i ≤ π ) is the weight vector and The angle between them, of which Weight The List, It is an integer that controls the magnitude of the angular distance.

[0012] To address the aforementioned technical problems, another technical solution adopted by this invention is: providing a noise-robust forged speech detection method based on joint training, comprising the following steps: First, a deep neural network is used to enhance the original noisy input speech to remove most of the background noise; Secondly, the enhanced speech is linearly weighted and fused with the original noisy input speech to obtain fused speech, achieve spectrogram compensation, and preserve forgery traces; Then, logarithmic power spectrum features are extracted from the fused speech, and modeling is performed based on low-frequency features to obtain the prediction results of the authenticity of the speech. Finally, the speech enhancement and forgery detection technologies are jointly trained and optimized.

[0013] In a preferred embodiment of the present invention, the step of enhancing the original input noisy speech using a deep neural network includes: First, a short-time Fourier transform is performed on the original input noisy speech signal to transform the time-domain signal into a time-frequency domain signal, thus obtaining the amplitude spectrum and phase spectrum of the original input speech. Then, the amplitude spectrum of the original input speech is modeled and trained by a deep neural network. Its output is the enhanced amplitude spectrum. The mean square error between the enhanced amplitude spectrum and the true amplitude spectrum is calculated as the training objective function. Finally, an inverse Fourier transform is performed on the phase spectrum of the original input speech and the amplitude spectrum of the enhanced speech to obtain the enhanced reconstructed speech.

[0014] In a preferred embodiment of the present invention, the specific steps for determining the authenticity of the fused speech include: First, regarding fused speech X fusion The complex spectrum is obtained by performing a Fourier transform. : (5) Secondly, perform complex spectrum operations to extract logarithmic power spectrum features. : (6) Then, the low-frequency band (0-4kHz) logarithmic power spectrum features are extracted as input to the forgery detection module: (7) Finally, the forgery detection module, based on a residual convolutional neural network, obtains the final output score. : (8) in, This represents a mapping function based on a residual convolutional neural network.

[0015] In a preferred embodiment of the present invention, the specific steps for jointly training and optimizing speech enhancement and forgery detection include: The objective functions obtained from speech enhancement and forgery detection are linearly combined with a weight to form the final objective function. Both modules are optimized simultaneously and mutually promote each other, causing the speech enhancement module to be optimized in a direction that is beneficial to forgery detection.

[0016] The beneficial effects of this invention are: (1) In this invention, the speech enhancement module uses a deep neural network to denoise the original noisy speech in order to remove most of the background noise, thereby achieving the purpose of enhancing the input speech signal; (2) In this invention, since the speech enhancement module will produce speech distortion and thus lose important speech information, in order to solve the problem of speech distortion and recover the lost information, in the speech fusion module, the time-domain waveform of the original input speech and the time-domain waveform of the enhanced speech are linearly weighted to obtain fused speech, thereby realizing spectrogram compensation, recovering the speech information lost due to speech distortion, preserving forgery traces, and further enhancing the speech. (3) In this invention, the joint training module is used to simultaneously optimize the speech enhancement module and the forgery detection module, which can optimize the two modules at the same time and promote each other, so that the enhancement module is optimized in a direction that is beneficial to forgery detection, thereby enhancing the robustness of the system to noisy environments. Attached Figure Description

[0017] Figure 1 This is a block diagram of the noise-robust forged speech detection system based on joint training according to the present invention; Figure 2 This is a schematic diagram illustrating the principle of the aforementioned voice enhancement module; Figure 3 This is a schematic diagram of the principle of the speech fusion module; Figure 4 This is a schematic diagram illustrating the principle of the counterfeit detection module. Figure 5 This is a schematic diagram illustrating the principle of the joint training module; Figure 6 This is a flowchart of the noise-robust forged speech detection method based on joint training. Detailed Implementation

[0018] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0019] Please see Figure 1 The embodiments of the present invention include: A noise-robust forged speech detection system based on joint training includes a speech enhancement module 1, a speech fusion module 2, a forgery detection module 3, and a joint training module 4. The speech enhancement module 1 uses a deep neural network to enhance the original noisy input speech, removing most of the background noise. The speech fusion module 2 compensates for the distortion introduced by the speech enhancement module 1 by linearly weighting and fusing the enhanced speech with the original input speech, recovering lost information and preserving forgery traces. The forgery detection module 3 extracts logarithmic power spectrum features from the fused speech, models it based on low-frequency features, and obtains a prediction of the speech's authenticity. The joint training module 4 jointly trains and optimizes the speech enhancement module 1 and the forgery detection module 3, allowing the two modules to mutually benefit each other and promote the optimization of the speech enhancement module 1 towards the final goal of forgery detection.

[0020] The following is a detailed description of each module: In this embodiment, as Figure 2As shown, speech enhancement module 1 first performs a short-time Fourier transform on the input noisy speech signal, transforming the time-domain signal to the time-frequency domain signal, obtaining the amplitude spectrum and phase spectrum of the original input speech. Then, it models the amplitude spectrum of the original input speech. The speech enhancement module removes most of the interference signal, thus enhancing the speech. It is trained using a deep neural network, and its output is the enhanced amplitude spectrum. .

[0021] (1) in, This represents the amplitude spectrum of the input noisy speech. These represent the number of frames and the number of frequency points in the input speech, respectively. This represents the mapping function based on a convolutional recurrent deep neural network. For ease of explanation, it will be referred to as... Omitted.

[0022] For the speech enhancement module, its training objective function To estimate the mean square error between the amplitude spectra of the enhanced speech and the target clean speech: (2) in, Indicates the number of time-frequency units. This represents the squared Frobenius norm.

[0023] The enhanced amplitude spectrum is obtained It can be compared with the phase spectrum of the original input speech. By combining these methods, the time-domain waveform of the enhanced speech is obtained through inverse Fourier transform: (3) in, That is, the inverse Fourier transform operation. This indicates the enhanced speech obtained after inverse Fourier transform.

[0024] Combination Figure 3 The speech fusion module 2 is connected to the speech enhancement module 1 and is used to compensate for the information loss caused by speech distortion. It obtains the enhanced speech waveform from the speech enhancement module 1 as one input. Since the original input speech time-domain waveform has no information loss, it is used as another input. A linear weighted sum of this sum and the enhanced speech time-domain waveform is then applied, requiring the use of hyperparameters. Adjustment, among which The original noisy speech. For the fusion result: (4) Combination Figure 4The forgery detection module 3 is connected to the voice fusion module 2, and firstly processes the fused voice... X fusion The complex spectrum is obtained by performing a Fourier transform. : (5) Secondly, perform complex spectrum operations to extract logarithmic power spectrum features. : (6) Then, the low-frequency band (0-4kHz) logarithmic power spectrum features are extracted as input to the forgery detection module: (7) Finally, the forgery detection module, based on a residual convolutional neural network, obtains the final output score. : (8) in, This represents a mapping function based on a residual convolutional neural network.

[0025] The objective function of the forgery detection module 3 is the A-Softmax loss based on angle margin. (9) (10) in, For the sample size, It is the first The feature vector of each sample Corresponding tags θ j,i (0≤) θ j,i ≤ π ) is the weight vector and The angle between them, of which Weight The List, It is an integer that controls the magnitude of the angular distance.

[0026] Combination Figure 5 The joint training module is connected to the speech enhancement module 1 and the forgery detection module 3, and is used to jointly optimize the various modules. The overall training objective function is... for: (11) in, These are hyperparameters used to adjust the weights of the loss functions for the speech enhancement and forgery detection modules. Ultimately, the entire forgery detection system is optimized through joint training.

[0027] Using formula (8) This is the final output of the entire fake voice detection system.

[0028] This invention, based on a jointly trained, noise-robust spoofing detection system, constructs a deep learning-based speech enhancement module to enhance the input noisy speech, thereby removing most of the noise signal. To address the speech distortion problem, a speech fusion module is used to fuse the enhanced speech with the original input speech, recovering the speech information lost due to distortion. Spoofing detection is then performed on the fused speech. Finally, a joint optimization method is used to further improve the system's performance. By introducing speech enhancement technology into the field of spoofing detection and employing a speech fusion module and a jointly trained module, the effectiveness of speech enhancement is improved, enabling the spoofing detection system to maintain robustness in noisy environments.

[0029] In this embodiment, combined with Figure 6 Furthermore, a noise-robust speech forgery detection method based on joint training is provided, including the following steps: First, a deep neural network is used to enhance the original noisy input speech to remove most of the background noise; Secondly, the enhanced speech is linearly weighted and fused with the original noisy input speech to obtain fused speech, achieve spectrogram compensation, and preserve forgery traces; Then, logarithmic power spectrum features are extracted from the fused speech, and modeling is performed based on low-frequency features to obtain the prediction results of the authenticity of the speech. Finally, the speech enhancement and forgery detection technologies are jointly trained and optimized.

[0030] Furthermore, the steps for enhancing the original noisy input speech using deep neural networks include: First, a short-time Fourier transform is performed on the original input noisy speech signal to transform the time-domain signal into a time-frequency domain signal, thus obtaining the amplitude spectrum and phase spectrum of the original input speech. Then, the amplitude spectrum of the original input speech is modeled and trained by a deep neural network. Its output is the enhanced amplitude spectrum. The mean square error between the enhanced amplitude spectrum and the true amplitude spectrum is calculated as the training objective function. Finally, an inverse Fourier transform is performed on the phase spectrum of the original input speech and the amplitude spectrum of the enhanced speech to obtain the enhanced reconstructed speech.

[0031] Furthermore, the specific steps for determining the authenticity of the fused speech include: First, regarding fused speechX fusion The complex spectrum is obtained by performing a Fourier transform. : (5) Secondly, perform complex spectrum operations to extract logarithmic power spectrum features. : (6) Then, the low-frequency band (0-4kHz) logarithmic power spectrum features are extracted as input to the forgery detection module: (7) Finally, the forgery detection module, based on a residual convolutional neural network, obtains the final output score. : (8) in, This represents a mapping function based on a residual convolutional neural network.

[0032] Furthermore, joint training is performed on speech enhancement and forgery detection, and the specific optimization steps include: The objective functions obtained from speech enhancement and forgery detection are linearly combined with a weight to form the final objective function. Both modules are optimized simultaneously and mutually promote each other, causing the speech enhancement module to be optimized in a direction that is beneficial to forgery detection.

[0033] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A noise-robust forged speech detection system based on joint training, characterized in that, include: The speech enhancement module is used to enhance the original noisy input speech using a deep neural network to remove most of the background noise. The speech fusion module is used to linearly weight and fuse the enhanced speech with the original input noisy speech to obtain fused speech, achieve spectrogram compensation, and save forgery traces; The forgery detection module is used to extract log power spectrum features from the fused speech, model based on low-frequency features, and obtain the prediction results of the authenticity of the speech. A joint training module is used to jointly train and optimize the speech enhancement module and the forgery detection module; The joint training module will use the objective function of the speech enhancement module. L ENH and the objective function of the forgery detection module L ASV The final objective function is a linear combination of weights λ. L The two modules are optimized simultaneously and mutually reinforcing, causing the speech enhancement module to be optimized in a direction that is beneficial to forgery detection; The objective function of the forgery detection module is an A-Softmax loss based on angle margin. ； in, N For the sample size, x i It is the first i The feature vector of each sample y i Indicates the first i The label corresponding to each sample θ j,i It is a weight vector W j and x i The angle between them, of which W j Weight W The j Column, 0≤ θ j,i ≤ π ; k ∈[0, m -1], m ≥1 is an integer that controls the magnitude of the angular distance.

2. The noise-robust forged speech detection system based on joint training according to claim 1, characterized in that, The steps by which the voice enhancement module enhances the input voice signal include: First, a short-time Fourier transform is performed on the original input noisy speech signal to transform the time-domain signal into a time-frequency domain signal, thus obtaining the amplitude spectrum and phase spectrum of the original input speech. Then, the amplitude spectrum of the original input speech is modeled and trained by a deep neural network. Its output is the enhanced amplitude spectrum. The mean square error between the enhanced amplitude spectrum and the true amplitude spectrum is calculated as the training objective function. Finally, an inverse Fourier transform is performed on the phase spectrum of the original input speech and the amplitude spectrum of the enhanced speech to obtain the enhanced reconstructed speech.

3. The noise-robust forged speech detection system based on joint training according to claim 1, characterized in that, The specific steps by which the forgery detection module determines the authenticity of the fused speech include: First, regarding fused speech X fusion The complex spectrum is obtained by performing a Fourier transform. : ； Secondly, perform complex spectrum operations to extract logarithmic power spectrum features. : ； Then, the logarithmic power spectrum features in the low-frequency range of 0-4kHz are extracted as input to the forgery detection module: ； Finally, the forgery detection module, based on a residual convolutional neural network, obtains the final output score. : ； in, This represents a mapping function based on a residual convolutional neural network.

4. A method for detecting forged speech using the noise-robust forged speech detection system based on joint training as described in claim 1, characterized in that, Includes the following steps: First, a deep neural network is used to enhance the original noisy input speech to remove most of the background noise; Secondly, the enhanced speech is linearly weighted and fused with the original noisy input speech to obtain fused speech, achieve spectrogram compensation, and preserve forgery traces; Then, log power spectrum features are extracted from the fused speech, and modeling is performed based on low-frequency features to obtain the prediction results of the authenticity of the speech. Finally, speech enhancement and forgery detection are jointly trained and optimized.

5. The method for detecting forged speech in a noise-robust forged speech detection system based on joint training according to claim 4, characterized in that, The steps for enhancing noisy input speech using deep neural networks include: First, a short-time Fourier transform is performed on the original input noisy speech signal to transform the time-domain signal into a time-frequency domain signal, thus obtaining the amplitude spectrum and phase spectrum of the original input speech. Then, the amplitude spectrum of the original input speech is modeled and trained by a deep neural network. Its output is the enhanced amplitude spectrum. The mean square error between the enhanced amplitude spectrum and the true amplitude spectrum is calculated as the training objective function. Finally, an inverse Fourier transform is performed on the phase spectrum of the original input speech and the amplitude spectrum of the enhanced speech to obtain the enhanced reconstructed speech.

6. The method for detecting forged speech in a noise-robust forged speech detection system based on joint training according to claim 4, characterized in that, The specific steps for determining the authenticity of the fused speech include: First, regarding fused speech X fusion The complex spectrum is obtained by performing a Fourier transform. : ； Secondly, perform complex spectrum operations to extract logarithmic power spectrum features. : ； Then, the logarithmic power spectrum features in the low-frequency range of 0-4kHz are extracted as input to the forgery detection module: ； Finally, the forgery detection module, based on a residual convolutional neural network, obtains the final output score. : ； in, This represents a mapping function based on a residual convolutional neural network.

7. The method for detecting forged speech in a noise-robust forged speech detection system based on joint training according to claim 4, characterized in that, The specific steps for jointly training and optimizing speech enhancement and forgery detection include: The objective function of the speech enhancement module L ENH and the objective function of the forgery detection module L ASV The final objective function is a linear combination of weights λ. L The two modules are optimized simultaneously and mutually reinforcing, causing the voice enhancement module to be optimized in a direction that is beneficial to forgery detection.

Images