Voice cloning anti-fraud recognition method, system and device and storage medium

Abstract

The invention discloses an anti-fraud recognition method, system and device for voice clone and a storage medium, belongs to the technical field of voice clone anti-counterfeiting, and solves the technical problem that the existing voice clone anti-counterfeiting technology is poor in limitation. The method comprises the following steps: applying non-uniform band-pass filtering in a Mel spectrogram generation stage by adopting a spectrum confusion technology, and artificially manufacturing frequency domain random fluctuation consistent with real voice; a dual-channel adversarial training mechanism is designed by adopting an adversarial noise embedding technology, a generator implants irreversible noise in the synthesized speech, noise concealment optimization is realized, and the MOS value of the finally synthesized speech reaches 3.792 in an experiment. According to the invention, a dynamic adaptive frame window convolution algorithm is provided, and the difficulty and complexity of synthesizing speech clone can be improved by injecting structured noise into a feature extraction process of an active interference speech clone model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of anti-fraud technology for voice cloning, and more particularly, to a voice cloning fraud identification method, system, device and storage medium. BACKGROUND

[0002] Some experts have conducted artificial analysis on related AI voice cloning from their own practice. Chen et al. found from the auditory perspective that there is some similarity in "medium tone, tone change and resonance mode" between the two, and there is a difference in the related "child-like sound" feature. Zhang et al. found from the perspective of spectrogram comparison that there is a difference in high-frequency spectrum between the synthesized voice and the real voice, and there is also a difference in the transition of some syllable consonants and vowels.

[0003] The existing anti-fraud technology for voice cloning is generally a passive detection technology, which has poor limitations. The current mainstream anti-fraud scheme focuses on voiceprint feature comparison (such as MFCC (Mel Frequency Cepstrum Coefficient) coefficient analysis) and frequency spectrum anomaly detection (high-frequency harmonic loss identification), and there are three major problems:

[0004] Response lag: it needs to be detected after the counterfeit voice is generated, and it cannot achieve pre-defense;

[0005] Weak generalization ability: it relies on a pre-defined voiceprint database (such as the ASVspoof 2019 benchmark library), and the new synthesis algorithm (VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech), Tacotron2) has a false detection rate of >23%;

[0006] Vulnerability to adversarial attacks: attackers can bypass the detection system by injecting specific noise (SNR (Signal-to-Noise Ratio) ≥15dB) through adversarial training.

[0007] Causes of technical defects:

[0008] Path dependence: existing research focuses too much on "improving detection accuracy", and lacks active intervention in the generation process;

[0009] Data closure: the voiceprint library update speed lags behind the progress of voice synthesis technology (the average lag period is ≥18 months);

[0010] Physical uncontrollability: passive detection relies on the sound pickup quality of terminal equipment, and the actual scene signal-to-noise ratio fluctuation leads to an increase in false positive rate. SUMMARY

[0011] The technical problem solved by the present application is to overcome the above-mentioned deficiencies of the prior art, and the first object of the present application is to provide a voice cloning anti-fraud identification method which can improve the anti-fake performance.

[0012] The second object of the present application is to provide a voice cloning anti-fraud identification system.

[0013] The third object of the present application is to provide a computer device.

[0014] The fourth object of the present application is to provide a computer storage medium.

[0015] In order to achieve the above-mentioned first object, the present application provides a voice cloning anti-fraud identification method, comprising:

[0016] The frequency domain random fluctuation is generated by applying non-uniform band-pass filtering in the mel-spectrogram generation stage using the spectrum confusion technology, and the frequency domain random fluctuation is artificially generated to match the real voice.

[0017] The double-channel adversarial training mechanism is designed by using the adversarial noise embedding technology, the generator implants irreversible noise in the synthesized voice, realizes the noise concealment optimization, and finally the MOS value of the synthesized voice reaches 3.792 in the experiment.

[0018] As a further improvement, the amount of frequency domain random fluctuation is a fluctuation amplitude of ±3dB.

[0019] Further, the method comprises the following steps:

[0020] Step 1. Frame division and windowing processing, comprising:

[0021] The frame division processing adopts a dynamic frame division strategy, i.e. dynamically adjusting the frame length according to the fundamental frequency (F0) of the voice signal;

[0022] The windowing processing adopts a composite windowing function, i.e. using the weighted superposition of Hamming window and rectangular window to suppress the frame edge effect;

[0023] Step 2. Adaptive convolution kernel generation, comprising:

[0024] Input features: MFCC coefficients and fundamental frequency F0 value of the current frame;

[0025] Generation network: based on the lightweight WaveGAN architecture, outputting a 3x3x2 convolution kernel matrix K=Gθ(MFCC,F0), K∈R^{3×3×2};

[0026] Physical constraints: max_ω|H(ω)|≤2.0,min_ω|H(ω)|≥0.5 (ω is the angular frequency, H(ω) is the frequency response); used to impose frequency domain constraints on the convolution kernel, to ensure that the frequency response is within the range of [0.5, 2.0];

[0027] Step 3. Multi-domain convolution operation, including:

[0028] Frequency domain processing uses Conv2D(Xmel, K[:,:,1]), and time domain processing uses Conv1D(Xwave, K[:,:,2]);

[0029] Time domain processing: one-dimensional convolution is performed on the subframe in the waveform domain, and the continuity of the interference time domain waveform is disturbed: Xwave' = Conv1D(Xwave, K);

[0030] Step 4. Anti-reverse attack design, a dynamic mixing strategy is adopted to mix the processed speech signal and the original speech signal according to a dynamic proportion, and the mixing coefficient β is determined by the speech complexity.

[0031] Further, in the frame processing,

[0032] If F0≤150Hz, the speech signal is in a low frequency band, a wide window is selected, the wide window is 25ms, and the formant structure is retained;

[0033] If F0>150Hz, the speech signal is in a high frequency band, a narrow window is selected, the narrow window is 15ms, and spectral aliasing is avoided.

[0034] Further, the composite windowing function is:

[0035]

[0036] Further, in the time domain processing, one-dimensional convolution is performed on the subframe within 5ms in the waveform domain.

[0037] Further,

[0038] Mixing coefficient: β=0.3+0.2·tanh(S MFCC -6.0);

[0039] Final output: Y final =β·X' wave +(1-β)·X raw .

[0040] In order to achieve the above-mentioned purpose two, the application provides an anti-fraud identification system for voice cloning, comprising:

[0041] Frame and windowing module, for frame and windowing processing, comprising:

[0042] Frame processing adopts a dynamic frame strategy, that is, the frame length is dynamically adjusted according to the fundamental frequency (F0) of the speech signal;

[0043] The windowing processing adopts a composite windowing function, that is, the weighted superposition of Hamming window and rectangular window is adopted to suppress the frame edge effect;

[0044] An adaptive convolution kernel module is used for adaptive convolution kernel generation, comprising:

[0045] Input features: input the MFCC coefficients and the fundamental frequency F0 value of the current frame;

[0046] A generation network: based on a lightweight WaveGAN architecture, output a 3*3*2 convolution kernel matrix K=Gθ(MFCC,F0), K∈R^{3×3×2};

[0047] Physical constraints: impose frequency domain constraints on the convolution kernel to ensure that the frequency response is within the range [0.5, 2.0]: maxω∣H(ω)∣≤2.0,minω∣H(ω)∣≥0.5;

[0048] A multi-domain convolution module is used for multi-domain convolution operation, comprising:

[0049] Frequency domain processing uses Conv2D(Xmel,K[:,:,1]), and time domain processing uses Conv1D(Xwave,K[:,:,2]);

[0050] Time domain processing: one-dimensional convolution is performed on the subframe in the waveform domain to interfere with the time domain waveform continuity: Xwave′=Conv1D(Xwave,K);

[0051] An anti-reverse attack module is used for anti-reverse attack design, and a dynamic mixing strategy is adopted to mix the processed speech signal and the original speech signal according to a dynamic proportion, and the mixing coefficient β is determined by the speech complexity.

[0052] In order to achieve the above-mentioned purposes three, the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor realizes the above-mentioned voice cloning fraud identification method when executing the computer program.

[0053] In order to achieve the above-mentioned purposes four, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned voice cloning fraud identification method.

[0054] Advantages

[0055] Compared with the prior art, the present application has the advantages of:

[0056] 1. Quality improvement

[0057] The frame window convolution noise proposed in the present application reaches 3.792 in MOS value after 400 rounds of training, which is significantly better than other noise types (such as Gaussian noise 4.579), proving its noise robustness and model adaptability.

[0058] 2. Anti-fake performance verification

[0059] Noise irreversibility:

[0060] By comparing the same noise applied multiple times, it is found that frame window convolution noise requires at least 23 rounds of iterative optimization to restore the original sound quality when reconstructing speech (other noises only require 5-8 rounds), indicating that it has anti-reverse stripping capability.

[0061] Feature confusion effect:

[0062] The mel spectrum after frame window convolution processing appears non-uniform fluctuations (amplitude ±2.5dB) in the high frequency region (>8kHz), which is consistent with the frequency domain randomness of real speech, and can effectively interfere with voiceprint feature extraction.

[0063] 3. Real-time processing performance

[0064] Preprocessing efficiency:

[0065] The frame window convolution module has a single-frame processing time of 4.3ms on an Intel i7-1185G7 platform, meeting the real-time speech processing requirements (delay <10ms).

[0066] 4. Training cost optimization

[0067] Under the same MOS target, using frame window convolution noise can reduce the VITS model training rounds by 34% (from 400 rounds to 264 rounds), and save about 217 yuan / model in GPU computing power cost (based on the unit price of NVIDIA A100).

[0068] 5. Deployment scalability

[0069] Support running on low-power devices such as Raspberry Pi 4B, with a hardware modification cost reduction of 92% compared to traditional FPGA solutions. BRIEF DESCRIPTION OF DRAWINGS

[0070] Fig. 1 The flowchart of the present application;

[0071] Fig. 2 The schematic diagram of the untreated audio;

[0072] Fig. 3 The schematic diagram of the audio processed by the present application. DETAILED DESCRIPTION

[0073] The present application will be further described below in conjunction with specific embodiments in the accompanying drawings.

[0074] Referring to Figs. 1-3 A voice cloning anti-fraud identification method, comprising:

[0075] The spectrum confusion technique is used to apply non-uniform band-pass filtering in the mel spectrum generation stage, artificially creating frequency domain random fluctuations consistent with real speech, and the amount of frequency domain random fluctuations is ± 3dB fluctuation amplitude;

[0076] The adversarial noise embedding technique is used to design a double-channel adversarial training mechanism. The generator implants irreversible noise in the synthesized speech to optimize noise concealment. The MOS value of the final synthesized speech reaches 3.792 in the experiment.

[0077] The method proposes a dynamic adaptive frame window convolution algorithm, which actively interferes with the feature extraction process of the speech cloning model by injecting structured noise, including the following steps 1-4:

[0078] Step 1. Frame and windowing processing, including:

[0079] The frame processing adopts a dynamic framing strategy, i.e., dynamically adjusting the frame length according to the fundamental frequency (F0) of the speech signal.

[0080] If F0≤150Hz, the speech signal is in the low frequency band, and a wide window of 25ms is selected to preserve the formant structure;

[0081] If F0>150Hz, the speech signal is in the high frequency band, and a narrow window of 15ms is selected to avoid spectral aliasing; thus, the frame window Tframe framing formula is:

[0082] Tframe={25ms,if F0≤150Hz; 15ms,if F0>150Hz}

[0083] The windowing processing adopts a composite windowing function, i.e., using the weighted superposition of Hamming window and rectangular window to suppress frame edge effects.

[0084] The composite windowing function is:

[0085]

[0086] Where n is the sample index, N is the window length, and rect(x) is the rectangular function (1 when x is in [0,1], otherwise 0).

[0087] Step 2. Adaptive convolution kernel generation, including:

[0088] Input features: input the MFCC coefficients and fundamental frequency F0 value of the current frame;

[0089] Generation network: based on the lightweight WaveGAN architecture, output a 3*3*2 convolution kernel matrix K=Gθ(MFCC,F0), K∈R^{3×3×2}; where K is the convolution kernel tensor, G is the generation network, θ is the network parameter, and R^{3×3×2} represents a 3*3*2 real matrix.

[0090] Physical constraint: impose a frequency domain constraint on the convolution kernel, guaranteeing the frequency response in the range [0.5, 2.0]:

[0091]

[0092] where ω is the angular frequency, and H(ω) is the frequency response function of the convolution kernel.

[0093] Step 3. Multi-domain convolution operation, including:

[0094] The frequency domain processing uses Conv2D(Xmel, K[:,:,1]), and the time domain processing uses Conv1D(Xwave, K[:,:,2]). Preferably, one-dimensional convolution is performed on subframes within 5 ms in the waveform domain. As follows:

[0095] Frequency domain processing: Xmel' = Conv2D(Xmel, K[:,:,1)), i.e., using the frequency domain channel, Xmel is the number of channels;

[0096] Time domain processing: Xwave' = Conv1D(Xwave, K[:,:,2)), i.e., using the time domain channel, Xwave is the number of channels;

[0097] Subframe length 5 ms.

[0098] Step 4. Anti-reverse attack design, using a dynamic mixing strategy to mix the processed speech signal with the original speech signal according to a dynamic proportion, and the mixing coefficient β is determined by the speech complexity.

[0099] Mixing coefficient: β = 0.3 + 0.2·tanh(S MFCC -6.0);

[0100] Final output (audio): Y final = β·X' wave + (1-β)·X raw ;

[0101] where S MFCC is the Shannon entropy of the MFCC coefficient (unit: bit), X' wave is the output waveform, and X raw is the original speech signal.

[0102] The following Table1-Table4 represent the comparison results of the frame window convolution noise proposed by the present application and other noise types.

[0103] Table1. MCD (Mel Cepstral Distortion) values under different slice lengths

[0104]

[0105] Table 2. MCD values under different training rounds

[0106]

[0107]

[0108] Table 3. MOS values under different slice lengths

[0109]

[0110] Table 4. MOS values under different training rounds

[0111]

[0112] A voice cloning anti-fraud recognition system, comprising:

[0113] A frame dividing and windowing module for frame dividing and windowing processing, comprising:

[0114] The frame dividing processing adopts a dynamic frame dividing strategy, i.e., dynamically adjusting the frame length according to the fundamental frequency (F0) of the voice signal;

[0115] The windowing processing adopts a composite windowing function, i.e., adopting the weighted superposition of the Hamming window and the rectangular window to suppress the frame edge effect;

[0116] An adaptive convolution kernel module for adaptive convolution kernel generation, comprising:

[0117] Input features: MFCC coefficients and fundamental frequency F0 value of the current frame;

[0118] Generation network: based on a lightweight WaveGAN architecture, outputting a 3x3x2 convolution kernel matrix K=Gθ(MFCC,F0), K∈R^{3×3×2};

[0119] Physical constraints: imposing frequency domain constraints on the convolution kernel to ensure that the frequency response is within the range of [0.5, 2.0]: maxω∣H(ω)∣≤2.0,minω∣H(ω)∣≥0.5;

[0120] A multi-domain convolution module for multi-domain convolution operation, comprising:

[0121] Frequency domain processing uses Conv2D(Xmel,K[:,:,1]), and time domain processing uses Conv1D(Xwave,K[:,:,2]);

[0122] Time domain processing: one-dimensional convolution of the sub-frame in the waveform domain to interfere with the time domain waveform continuity: Xwave′=Conv1D(Xwave,K).

[0123] The anti-reverse attack module is used for anti-reverse attack design, and a dynamic mixed strategy is adopted to mix the processed voice signal and the original voice signal according to a dynamic proportion, and a mixing coefficient beta is determined by voice complexity.

[0124] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the voice cloning anti-fraud identification method when executing the computer program.

[0125] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the voice cloning anti-fraud identification method.

[0126] The above is only the preferred embodiment of the present application, it should be pointed out that for those skilled in the art, without departing from the structure of the present application, can also make several variations and improvements, these will not affect the effect and the practicality of the patent of the present application.

Claims

1. A method for anti-fraud identification of voice cloning, characterized in that: include: Spectral obfuscation technology is used to apply non-uniform bandpass filtering during the Mel-spectrogram generation stage to artificially create random fluctuations in the frequency domain that are consistent with real speech; A dual-channel adversarial training mechanism is designed using adversarial noise embedding technology. The generator implants irreversible noise into the synthesized speech to optimize the noise concealment. The final MOS value of the synthesized speech reaches 3.792 in the experiment.

2. The anti-fraud identification method for voice cloning according to claim 1, characterized in that: The amount of the random fluctuation in the frequency domain is a fluctuation amplitude of ±3dB.

3. The anti-fraud identification method for voice cloning according to claim 1 or 2, characterized in that: The following steps are involved: Step 1: Framing and windowing, including: The framing process adopts a dynamic framing strategy, that is, the frame length is dynamically adjusted according to the fundamental frequency (F0) of the speech signal; The windowing process uses a composite windowing function, that is, a weighted superposition of a Hamming window and a rectangular window to suppress the frame edge effect; Step 2. Adaptive convolution kernel generation, including: Input features: MFCC coefficients and fundamental frequency F0 value of the current frame; Generator network: Based on the lightweight WaveGAN architecture, it outputs a 3×3×2 convolution kernel matrix: K=Gθ(MFCC,F0),K∈R^{3×3×2}; Physical constraints: max_ω|H(ω)|≤2.0, min_ω|H(ω)|≥0.5 (ω is the angular frequency, H(ω) is the frequency response); used to impose frequency domain constraints on the convolution kernel to ensure that the frequency response is in the range [0.5, 2.0]; Step 3. Multi-domain convolution operation, including: Frequency domain processing: Xmel′=Conv2D(Xmel,K[:,:,1]); Time domain processing: Xwave′=Conv1D(Xwave,K[:,:,2]); Subframe length 5ms; Step 4. Anti-reverse attack design uses a dynamic mixing strategy to mix the processed speech signal with the original speech signal in a dynamic ratio. The mixing coefficient β is determined by the speech complexity.

4. The anti-fraud identification method for voice cloning according to claim 3, characterized in that: During framing processing, If F0≤150Hz, the speech signal is in the low frequency band, and a wide window of 25ms is selected to preserve the formant structure; If F0>150Hz, the speech signal is in the high frequency band, and a narrow window of 15ms is selected to avoid spectrum aliasing.

5. The anti-fraud identification method for voice cloning according to claim 3, characterized in that: The composite windowing function is:

6. The anti-fraud identification method for voice cloning according to claim 3, characterized in that: In the time domain processing, one-dimensional convolution is performed on the subframes within 5ms in the waveform domain.

7. The anti-fraud identification method for voice cloning according to claim 3, characterized in that: Mixing coefficient: β=0.3+0.2·tanh(S MFCC -6.0); Final output: Y final =β·X' wave +(1-β)·X raw .

8. A voice cloning anti-fraud identification system, characterized by: include: Framing and windowing module, used for framing and windowing processing, including: The framing process adopts a dynamic framing strategy, that is, the frame length is dynamically adjusted according to the fundamental frequency (F0) of the speech signal; The windowing process uses a composite windowing function, that is, a weighted superposition of a Hamming window and a rectangular window to suppress the frame edge effect; Adaptive convolution kernel module, used for adaptive convolution kernel generation, including: Input features: Input the MFCC coefficients and fundamental frequency F0 value of the current frame; Generative network: Based on the lightweight WaveGAN architecture, it outputs a 3×3×2 convolution kernel matrix K=Gθ(MFCC,F0),K∈R^{3×3×2}; Physical constraints: max_ω|H(ω)|≤2.0, min_ω|H(ω)|≥0.5 (ω is the angular frequency, H(ω) is the frequency response); used to impose frequency domain constraints on the convolution kernel to ensure that the frequency response is in the range [0.5, 2.0]; Multi-domain convolution module, used for multi-domain convolution operations, including: Frequency domain processing: Xmel′=Conv2D(Xmel,K[:,:,1]); Time domain processing: Xwave′=Conv1D(Xwave,K[:,:,2]); Subframe length 5ms; The anti-reverse attack module is used for anti-reverse attack design. It adopts a dynamic mixing strategy to mix the processed speech signal with the original speech signal in a dynamic proportion. The mixing coefficient β is determined by the complexity of the speech.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the anti-fraud identification method for voice cloning according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the anti-fraud identification method for voice cloning according to any one of claims 1 to 7 is implemented.

Images