Counterfeit voice detection method and system based on glottis flow features and voiceprint features
Through a forged speech detection method based on glottal flow characteristics and vocalprint characteristics, the problem of performance degradation in the existing technology when generating forged speech in the face of unknown algorithms is solved, and higher detection performance and robustness are achieved, and good interpretability and generalization are achieved.
Patent Information
- Application Number
- CN202411692866.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-07
AI Technical Summary
The existing forged speech detection technology has deteriorated performance when facing forged speech generated by unknown algorithms, and lacks analysis and utilization of inherent defects of forged speech, resulting in poor interpretability and generalization, and poses safety risks.
A forged speech detection method based on glottal flow characteristics and vocal mark features is proposed. By preprocessing speech signals, vocal mark features and glottal flow characteristics are extracted and inputted to the back-end classification model based on a two-dimensional convolutional neural network for detection.
It improves the performance and robustness of the forged speech detection system, has good interpretability and generalization, can effectively detect forged speech generated by unknown algorithms, and shows good detection capabilities in cross-channel scenarios.
Smart Images
Figure CN119580770A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of speech detection technology, and specifically relates to a method and system for detecting forged speech based on glottal flow features and voiceprint features. Background Art
[0002] With the development of deep speech synthesis technology and generative artificial intelligence technology, the threshold for speech generation is lowering and the quality is improving. This also provides opportunities for criminals, who can use speech synthesis technology to forge other people's voices and commit fraud or other illegal acts. In response to this problem, researchers have begun to develop forged speech detection technology. Although the currently deployed forged speech detection systems have achieved initial results, the performance usually decreases significantly when faced with forged speech generated by unknown algorithms. At the same time, the current methods lack analysis and utilization of the inherent defects of forged speech, and have poor interpretability and generalization, which brings security risks to applications in actual scenarios. Summary of the invention
[0003] The purpose of this application is to overcome the shortcomings of the prior art in terms of poor interpretability and generalization.
[0004] In order to achieve the above objectives, the present application proposes a method for detecting forged speech based on glottal flow features and voiceprint features, comprising:
[0005] The pre-processed target voice information is input into the trained forged voice detection model, and the forged voice detection result is output;
[0006] The preprocessing process of the target speech includes:
[0007] Use the pre-trained ECAPA-TDNN model to extract the voiceprint features of the training speech;
[0008] The glottal flow features of the training speech are extracted using the inverse filtering method; the robust features of the glottal flow features are extracted using the pre-trained wav2vec model.
[0009] As an improvement of the above method, the forged speech detection model is a back-end classification model based on a two-dimensional convolutional neural network.
[0010] As an improvement of the above method, the forged speech detection model uses a ResNet34 structure, including an initial layer, four main layers and a classification layer;
[0011] The main layer includes multiple residual blocks; each of the residual blocks includes two 3x3 convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function; within the residual block, a jump connection is also included to directly add the input to the output of the convolutional layer.
[0012] As an improvement of the above method, the detection process of the forged speech detection model includes:
[0013] Inputting the robustness features of the glottal flow features into the initial layer and the main layer of the forged speech detection model, and outputting the features after convolution dimensionality reduction;
[0014] The features after convolution dimensionality reduction are concatenated with the voiceprint features, and input into the classification layer of the forged speech detection model to obtain a classification result.
[0015] The present application also provides a forged voice detection system based on glottal flow features and voiceprint features, which is implemented based on the above method, and the system includes:
[0016] The data preprocessing module is used to extract the voiceprint features of the training speech using the pre-trained ECAPA-TDNN model; extract the glottal flow features of the training speech using the inverse filtering method; and extract the robustness features of the glottal flow features using the pre-trained wav2vec model;
[0017] The forged voice detection module is used to input the target voice information processed by the data preprocessing module into the trained forged voice detection model and output the forged voice detection result.
[0018] Compared with the prior art, the advantages of this application are:
[0019] The present invention improves the performance and robustness of the forged speech detection system by proposing a method and system for forged speech detection based on glottal flow and voiceprint features. The present invention uses glottal flow to model rhythm artifacts, and uses a method based on voiceprint features to enhance the detection capability of speech conversion algorithms. The proposed scheme utilizes the artifacts introduced by the acoustic model of speech synthesis and speech conversion algorithms, and has a certain degree of interpretability. At the same time, the proposed scheme has good detection capabilities for unknown algorithms and is generalizable in cross-channel scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 Shown is a flow chart of a forged speech detection method based on glottal flow features and voiceprint features. DETAILED DESCRIPTION
[0021] The technical solution of the present application is described in detail below with reference to the accompanying drawings.
[0022] The purpose of the present invention is to propose a method and system for detecting forged speech based on glottal flow features and voiceprint features, and to enhance the performance of the current forged speech detection system based on deep neural networks. The model based on glottal flow features can effectively detect speech synthesis algorithms, and the model based on voiceprint features can supplement its detection capabilities for speech conversion algorithms. The fusion model effectively improves the detection capabilities for forged algorithms and has generalization in cross-channel scenarios.
[0023] Example 1
[0024] The forged speech detection method based on glottal flow features and voiceprint features provided in this application includes:
[0025] Step 1), collect the speech signal used for training the model;
[0026] Step 2), using the pre-trained ECAPA-TDNN model to extract the voiceprint features of the training speech in step 1);
[0027] ECAPA-TDNN is a commonly used feature extraction model in voiceprint recognition tasks. This model introduces channel attention mechanism, propagation mechanism and aggregation mechanism to the traditional time-delay neural network, which effectively enhances the voiceprint feature representation capability. The squeeze-excitation module in the model weights different channels through attention weights, and adaptively strengthens the response to channels with stronger identification capabilities. The voiceprint features extracted by the shallow model of ECAPA-TDNN can improve the detection performance of the voice conversion algorithm, so as to avoid information loss caused by time domain pooling while extracting voiceprint information.
[0028] Step 3), using the inverse filtering method to extract the glottal flow features of the training speech in step 1);
[0029] For a speech signal, its glottal flow can be obtained by estimating the vocal tract filter and lip radiation filter for each frame of the signal and inverse filtering it. Among them, the source speech can be expressed as S(z)=E(z)G(z)V(z)L(z), S(z) represents the speech signal, E(z) represents the glottal flow, G(z) represents the glottal filter, V(z) represents the vocal tract filter, and L(z) represents the lip radiation filter. In the above formula, the modulation process of the vocal tract can be modeled as an all-pole model, and its transfer function is expressed as: Where N represents the number of poles, a k is the parameter of the vocal tract model, which changes continuously with the tuning process of the vocal tract. The speech signal has short-term stability, so the vocal tract parameters at the frame level can be estimated by the linear prediction coding algorithm. After the speech passes through the vocal tract, it is radiated by the lips and finally output to the air. Generally, a high-pass filter is used to simulate the occurrence of this process, and its transfer function is expressed as: L(z) = 1-dz -1For a speech signal, its glottal flow can be obtained by estimating the vocal tract filter and lip radiation filter for each frame of the signal and then inverse filtering them.
[0030] Step 4), using the pre-trained wav2vec model to extract the robust features of the glottal flow in step 3);
[0031] First, load the pre-trained wav2vec model, which is a speech model based on self-supervised learning. It uses the wav2vec 2.0 architecture to train a feature extraction model that effectively models the contextual relationship of speech signals through encoding, quantization, and unsupervised training. The wav2vec model has achieved good performance in the field of speech and is widely used in speech-related downstream tasks. Send the speech into the wav2vec model and use the model output as a feature. Since the output feature dimension is high and it is difficult to train directly using large amounts of computation, the feature is reduced in dimension through a linear layer to facilitate input into subsequent models for processing.
[0032] Step 5), sending the voiceprint features extracted in step 2) and the glottal flow robustness features extracted in step 4) into a back-end classification model based on a two-dimensional convolutional neural network for modeling;
[0033] The voiceprint features extracted in step 2) and the glottal flow robustness features extracted in step 4) are sent to the back-end classification model based on a two-dimensional convolutional neural network for modeling. This application uses a two-dimensional convolutional structure based on a lightweight convolutional neural network to further model and reduce the features, and uses a layer based on a long short-term memory network to process the timing information and model the rhythm artifacts of the forged speech. The back-end classification model based on a two-dimensional convolutional neural network uses a ResNet34 structure, including an initial layer, four main layers and a classification layer. Each main layer includes multiple residual blocks. Each residual block includes two 3x3 convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function. In the residual block, a jump connection is also included to add the input directly to the output of the convolutional layer. The modeled rhythm artifact is input into the linear classification layer, and finally a binary classification result is obtained.
[0034] Rhythmic artifacts refer to defects or unnatural features introduced by the acoustic model when predicting rhythmic information during the text-to-speech (TTS) synthesis process. These artifacts are usually caused by the TTS system not processing rhythmic information accurately or naturally when converting text to speech, resulting in differences in the rhythm of the generated speech and real human speech. This difference may include unnatural changes in rhythm, inaccurate syllable duration, mismatched intonation patterns, etc.
[0035] Step 6), using the training voice to train the fake voice detection system;
[0036] The wav2vec system in step 4) is trained and its parameters are updated, and the two-dimensional convolutional network and the linear classification layer in step 5) are trained to obtain a forged speech detection system based on glottal flow and voiceprint features.
[0037] Step 7), passing the target speech through the detection system trained in step 6), to obtain the final forged speech detection result.
[0038] The voiceprint features and glottal flow features obtained in steps 2), 3) and 4) of the training in step 6) are sent to the back-end classification model based on the two-dimensional convolutional neural network described in step 5) to obtain the target voice forged voice detection result. The specific process includes:
[0039] The robust features of the glottal flow features are input into the initial layer and main layer of the back-end classification model, and the features after convolutional dimension reduction are output;
[0040] The features after convolutional dimensionality reduction are concatenated with the voiceprint features and input into the classification layer of the backend classification model to obtain the classification results.
[0041] refer to Figure 1 , the method of the present invention comprises the following steps:
[0042] Step 101), collect the target speech signal used for training the model.
[0043] Step 102), using the pre-trained ECAPA-TDNN model to extract the voiceprint features of the training speech in step 101);
[0044] Step 103), using an inverse filtering method to extract the glottal flow features of the training speech in step 101);
[0045] Step 104), using the pre-trained wav2vec model to extract the robustness features of the glottal flow in step 103);
[0046] Step 105), sending the voiceprint features extracted in step 102) and the glottal flow robustness features extracted in step 104) into a ResNet-based backend classification model for modeling;
[0047] Step 106), using the training voice to train the forged voice detection system;
[0048] Step 107), the target speech is passed through the detection system trained in step 106), to obtain the final forged speech detection result.
[0049] Example 2
[0050] The present application also provides a forged voice detection system based on glottal flow features and voiceprint features, which is implemented based on the above method, and the system includes:
[0051] The data preprocessing module is used to extract the voiceprint features of the training speech using the pre-trained ECAPA-TDNN model; extract the glottal flow features of the training speech using the inverse filtering method; and extract the robustness features of the glottal flow features using the pre-trained wav2vec model;
[0052] The forged voice detection module is used to input the target voice information processed by the data preprocessing module into the trained forged voice detection model and output the forged voice detection result.
[0053] The present application may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.
[0054] The user interface may include a display, a keyboard or a pointing device, such as a mouse, a trackball, a touch pad or a touch screen.
[0055] It is understood that the memory in the embodiments disclosed in the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.
[0056] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.
[0057] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.
[0058] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:
[0059] Execute the steps of the above method.
[0060] The above method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The above-disclosed methods, steps and logic block diagrams can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the above-disclosed method can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0061] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.
[0062] For software implementation, the technology of the present application can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0063] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.
[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.
Claims
1. A method for detecting forged speech based on glottal flow features and voiceprint features, comprising: The pre-processed target voice information is input into the trained forged voice detection model, and the forged voice detection result is output; The preprocessing process of the target speech includes: Use the pre-trained ECAPA-TDNN model to extract the voiceprint features of the training speech; The glottal flow features of the training speech are extracted using the inverse filtering method; the robust features of the glottal flow features are extracted using the pre-trained wav2vec model.
2. The forged speech detection method based on glottal flow features and voiceprint features according to claim 1 is characterized in that: The forged speech detection model is a back-end classification model based on a two-dimensional convolutional neural network.
3. The forged speech detection method based on glottal flow features and voiceprint features according to claim 2 is characterized in that: The fake speech detection model uses a ResNet34 structure, including an initial layer, four main layers and a classification layer; The main layer includes multiple residual blocks; each of the residual blocks includes two 3x3 convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function; within the residual block, a jump connection is also included to directly add the input to the output of the convolutional layer.
4. The forged speech detection method based on glottal flow features and voiceprint features according to claim 3 is characterized in that: The detection process of the forged speech detection model includes: Inputting the robustness features of the glottal flow features into the initial layer and the main layer of the forged speech detection model, and outputting the features after convolution dimensionality reduction; The features after convolution dimensionality reduction are concatenated with the voiceprint features, and input into the classification layer of the forged speech detection model to obtain a classification result.
5. A forged speech detection system based on glottal flow features and voiceprint features, implemented based on any of the methods described in claims 1-4, characterized in that: The system comprises: A data preprocessing module for extracting voiceprint features of training speech using a pre-trained ECAPA-TDNN model; extracting glottal flow features of training speech using an inverse filtering method; extracting robustness features of glottal flow features using a pre-trained wav2vec model; and The forged voice detection module is used to input the target voice information processed by the data preprocessing module into the trained forged voice detection model and output the forged voice detection result.