Single-classification forged voice detection method and system based on teacher-student framework
By adopting a single classification method of teacher-student framework in the forged speech detection system, using the combination of teacher model and student model, extracting hidden layer features of speech and calculating cosine similarity, the problem of poor detection performance in the existing technology is solved, and the realization of the realization of the realization of the problem of poor detection performance is achieved. Efficient detection of the generation of fake speech by unknown algorithms.
Patent Information
- Application Number
- CN202411703134.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-02-21
AI Technical Summary
The existing forged speech detection system has poor detection performance when training resources are limited, especially when facing forged speech generated by unknown algorithms, it is difficult to effectively detect.
A single-class fake speech detection method based on the teacher-student framework is used. By inputting the target speech to trained teacher model and student model, hidden layer features are extracted and cosine similarity is calculated, and the speech is judged as true or forged. The teacher model includes the Wav2Vec model of multi-layer Transformer layer, graph neural network and classification layer, while the student model inserts the LoRA adapter module into the Transformer layer to lighten the parameters.
It improves the performance and robustness of the forged speech detection system, can more effectively detect forged speech generated by unknown algorithms, and enhances the learning ability of real speech feature space and boundaries.
Smart Images
Figure CN119495321A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of speech detection technology, and specifically relates to a single-classification forged speech detection method and system based on a teacher-student framework. Background Art
[0002] Speech synthesis technology includes text-to-speech and voice-to-speech algorithms, which aim to produce natural and human-like speech. In recent years, the naturalness of synthesized speech has been further improved with the increase of available data and the advancement of advanced large-scale language models. Although these synthesized utterances bring convenience and fun to our daily lives, they also bring potential security risks. In order to deal with these threats, it is necessary to develop effective forged speech detection models.
[0003] Most of the current forged speech detection systems are binary classification systems, and their performance is poor for forged speech that is inconsistent with the distribution of the training set. In addition, the target that the forged speech detection model needs to identify is constantly iterative, and the training resources available are limited. Therefore, how to detect unknown algorithms under the premise of limited training data is a difficult problem that needs to be solved urgently. Summary of the invention
[0004] The purpose of this application is to overcome the shortcomings of the prior art, which are limited training resources and poor detection performance.
[0005] In order to achieve the above objectives, this application proposes a single-classification forged speech detection method based on a teacher-student framework, including:
[0006] Input the target speech into the trained teacher model and student model respectively to obtain the final forged speech detection result;
[0007] The method of obtaining the final forged speech detection result includes: extracting the hidden features of the target speech through the teacher model and the student model respectively, calculating the cosine similarity of the features output by the teacher model and the student model, and when the cosine similarity is greater than a set threshold, the target speech is a real speech, otherwise it is a forged speech.
[0008] As an improvement of the above method, the teacher model includes a Wav2Vec model containing several Transformer layers, a graph neural network and a classification layer;
[0009] The student model includes a Wav2Vec model containing several Transformer layers and a graph neural network; wherein a LoRA-based adapter module is inserted into the Transformer layer;
[0010] The number of Transformer layers in the teacher model is the same as the number of Transformer layers in the student model.
[0011] As an improvement of the above method, the operation process of the teacher model includes: the target speech first passes through the Wav2Vec model to obtain two-dimensional intermediate features, then passes through the graph neural network to model the time-frequency domain feature relationship, and finally the spliced time domain features and frequency domain features are input into the classification layer to obtain a binary classification result.
[0012] The operation process of the student model includes: the target speech first passes through the Wav2Vec model to obtain two-dimensional intermediate features, and then passes through the graph neural network to model the time-frequency domain feature relationship.
[0013] As an improvement of the above method, the process of modeling the time-frequency domain feature relationship of the graph neural network includes:
[0014] Step A1), respectively perform pooling based on the attention mechanism on the time domain and frequency domain features to obtain time domain features and frequency domain features;
[0015] Step A2), modeling the time domain features and the frequency domain features as points, and extracting features from the time domain features and the frequency domain features through a graph attention network and a graph maximum pooling network respectively;
[0016] Step A3), jointly modeling the time domain features and frequency domain features extracted in step A2) through a heterogeneous graph attention network;
[0017] Step A4), perform maximum pooling on the time domain features and frequency domain features of the joint modeling in the time domain and frequency domain respectively, and separate the time domain features and the frequency domain features.
[0018] As an improvement of the above method, the classification layer includes two linear layers; the linear layers perform binary classification on the time domain features and frequency domain features output by the graph neural network to obtain classification results.
[0019] As an improvement of the above method, the training process of the teacher model includes:
[0020] Input the real samples and forged samples into the Wav2Vec model for training and extract the robust features of the samples;
[0021] The robust features are input into the graph neural network and classification layer for training to obtain the trained teacher model.
[0022] As an improvement of the above method, the training process of the student model includes:
[0023] The real samples are input into the student model for training. The output of the corresponding Transformer layer of the student model learns from the embedding of the corresponding Transformer layer of the teacher model, and the loss function to be optimized is the cosine loss. The graph neural network of the student model learns from the embedding output of the graph neural network of the teacher model.
[0024] The present application also provides a single-classification forged speech detection system based on a teacher-student framework, which is implemented based on the above method. The system includes a teacher model and a student model; wherein,
[0025] The teacher model includes a Wav2Vec model containing several Transformer layers, a graph neural network and a classification layer;
[0026] The student model includes a Wav2Vec model containing several Transformer layers and a graph neural network; wherein a LoRA-based adapter module is inserted into the Transformer layer;
[0027] The number of Transformer layers in the teacher model is the same as the number of Transformer layers in the student model.
[0028] Compared with the prior art, the advantages of this application are:
[0029] The present invention proposes a single-classification forged speech detection method and system based on a teacher-student framework to improve the performance and robustness of the forged speech detection system. The present invention learns the feature space and boundary of real speech through a single-classification method, and introduces the structure of the teacher model to enhance the learning ability of the real speech feature distribution, effectively improving the performance and robustness of the forged speech detection system for forged speech generated by unknown algorithms. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 Shown is a flow chart of a single-classification forged speech detection method based on a teacher-student framework;
[0031] Figure 2 Shown is the architecture diagram of single-classification forged speech detection based on the teacher-student framework. DETAILED DESCRIPTION
[0032] The technical solution of the present application is described in detail below with reference to the accompanying drawings.
[0033] The purpose of the present invention is to propose a single-classification forged speech detection method and system based on a teacher-student framework to enhance the detection capability of unknown algorithms. The forged speech detection framework based on single-classification learning can learn the distribution of real samples and correctly identify various forged samples far from the distribution. The teacher-student framework can introduce a trained teacher model to avoid the problem that the single-classification model is difficult to train.
[0034] Example 1
[0035] The single-classification forged speech detection method based on the teacher-student framework provided in this application includes:
[0036] Step 1) Collect speech signals used for training the model, including real samples and forged samples;
[0037] Step 2), construct a teacher model and use the pre-trained Wav2Vec model to extract the robust features of the training speech in step 1);
[0038] Extract features through the pre-trained Wav2Vec model. First, load the pre-trained Wav2Vec model, which is a speech model based on self-supervised learning. It uses the Wav2Vec 2.0 architecture to train a feature extraction model that effectively models the contextual relationship of speech signals through encoding, quantization, and unsupervised training. The Wav2Vec model has achieved good performance in the field of speech and is widely used in speech-related downstream tasks. Send the speech into the Wav2Vec model and use the model output as a feature. Since the output feature dimension is high and it is difficult to train directly using large amounts of computation, the feature is reduced in dimension through a linear layer to facilitate input into subsequent models for processing.
[0039] Step 3), the robust features extracted by Wav2Vec in step 2) are sent to the classification model based on the graph neural network and the classification layer. The overall structure of the teacher model includes a Wav2Vec model with 24 layers of Transformer layers, a graph neural network and a classification layer. The input speech first passes through Wav2Vec to obtain two-dimensional intermediate features, then further modeled through the graph neural network, and finally obtains the binary classification result through the linear layer; in other embodiments, the Wav2Vec model may also include other numbers of Transformer layers.
[0040] The process of sending the extracted robust features after dimensionality reduction into the graph network for time-frequency relationship modeling includes:
[0041] Step 3-1), respectively perform pooling based on the attention mechanism on the feature time domain and frequency domain to obtain time domain features and frequency domain features;
[0042] Step 3-2), model the time domain features and frequency domain features in step 3-1) as points, and extract features of the time domain and frequency domain features through the graph attention network and the graph maximum pooling network respectively;
[0043] Step 3-3), jointly model the time domain features and frequency domain features extracted by the graph network in step 3-2) through the heterogeneous graph attention network;
[0044] Step 3-4), the time-frequency domain features jointly modeled in step 3-3) are pooled at the maximum value in the time domain and frequency domain respectively to separate the time domain features and the frequency domain features.
[0045] A graph network is defined by a set of points and a set of edges connecting different pairs of points. Through high-dimensional vectors such as points, graph neural networks can model non-Euclidean data manifolds between different nodes. The graph network used in this application is a fully connected graph network. Each point is connected by an edge. Through the self-attention mechanism of the graph attention layer, data is driven to assign weights to each edge to reflect the correlation between each pair of points. This application reduces the number of points through graph attention pooling, thereby reducing model complexity and improving robustness. Modeling time-frequency relationships through heterogeneous stacked graph attention layers. The input of the heterogeneous stacked attention layer is first projected into another latent space, giving two nodes a dimension of D. t and D s The projection is done using two fully connected layers, each of which projects a component subgraph onto D st The point stacking method is used to accumulate heterogeneous information, that is, the information or relationship between the time domain and the frequency domain. The stacked points are connected to the entire set of points. Using one-way edges from all other nodes to the stacked nodes helps to preserve the information in the time domain graph and the frequency domain graph. The maximum graph operation is used to combine and utilize different features for forged speech detection and different possible defects. Two parallel branches are utilized here, where the element maximum is applied to the two branch outputs. Specifically, each branch involves two heterogeneous stacked graph attention layers in sequence, where a graph pooling layer is applied to the output of each heterogeneous stacked graph attention layer. Therefore, the maximum graph operation contains four heterogeneous stacked graph attention layers and four graph pooling layers. The two heterogeneous stacked graph attention layers in each branch share the stack node; the stack node of the previous heterogeneous stacked graph attention layer is passed to the next heterogeneous stacked graph attention layer. The element maximum is also applied to the stack node of each branch.
[0046] The classification layer consists of two linear layers. The concatenated time domain features and frequency domain features extracted by the multi-scale graph network are reduced in dimension and classified into two categories to obtain the classification results. The loss function of the adaptive transformation boundary according to the input speech duration is calculated.
[0047] Step 4), use the training speech to train the teacher model;
[0048] Train the teacher model, train the Wav2Vec system in step 2) and update the parameters, train the graph network and linear classification layer in step 3), and obtain the teacher model.
[0049] Step 5) Construct a student model, whose structure is consistent with the teacher model described in step 2) and step 3). The overall structure of the student model includes a Wav2Vec model with 24 Transformer layers and a graph neural network. The input speech first passes through Wav2Vec to obtain a two-dimensional intermediate feature, and then passes through the graph neural network for further modeling;
[0050] The student model is constructed through the same steps as step 2) and step 3).
[0051] Step 6), insert the LoRA-based adapter into the Transformer layer of the Wav2Vec of the student model constructed in step 5 to achieve parameter lightweight;
[0052] The LoRA adapter constrains the weight matrix update through low-rank decomposition. For the pre-trained weight matrix W0∈R d×k , the parameter update process can be expressed as W = W0 + ΔW. ΔW can be expressed as matrix multiplication in LoRA, ΔW = BA, where B∈R d×r , A∈R r×k , r is the rank of the matrix, and r<<min(d,k). In the LoRA-based fine-tuning process, W0 is frozen and no longer updated with gradients, while A and B are learnable weights. The forward process of LoRA can be expressed as h=W0x+ΔWx=W0x+BAx.
[0053] Step 7) Use the real sample single classification in the training speech to train the student model;
[0054] The student model constructed in step 6) is trained. The training data is the real speech part of the training speech collected in step 1). The output of the corresponding Transformer layer of the student model learns from the embedding of the corresponding Transformer layer of the teacher model. The loss function to be optimized is the cosine loss. The graph neural network also learns from the embedding output of the graph neural network of the teacher model.
[0055] Step 8) The target speech is simultaneously passed through the teacher model trained in step 4) and the student model trained in step 7) to obtain the final forged speech detection result.
[0056] In step 8), the hidden features of the target speech are extracted respectively by the teacher model obtained in steps 2) and 3) trained in step 4) and the student model obtained in steps 5) and 6) trained in step 7). The cosine similarity of the features output by the teacher and student models is calculated. If the similarity is greater than the prior threshold, the detection result of the target speech is real speech, otherwise the detection result of the target speech is forged speech.
[0057] refer to Figure 1 The single-classification forged speech detection method based on the teacher-student framework provided in this application comprises the following steps:
[0058] Step 101), collecting speech signals used for training the model, including real samples and forged samples;
[0059] Step 102), construct a teacher model, and use the pre-trained Wav2Vec model to extract the robustness features of the training speech in step 101);
[0060] Step 103), sending the robust features extracted by Wav2Vec in step 102) to a classification model based on a graph neural network and a classification layer;
[0061] Step 104), using the training speech to train the teacher model;
[0062] Step 105), constructing a student model, whose structure is consistent with the teacher model described in step 102) and step 103);
[0063] Step 106) inserting a LoRA-based adapter into the Transformer layer of the Wav2Vec of the student model described in step 5 to achieve parameter lightweighting;
[0064] Step 107), using the real sample single classification training student model in the training speech;
[0065] Step 108), the target speech is simultaneously passed through the teacher model trained in step 104) and the student model trained in step 107), to obtain the final forged speech detection result.
[0066] Example 2
[0067] The present application also provides a single-classification forged speech detection system based on a teacher-student framework, which is implemented based on the above method. The system includes a teacher model and a student model; wherein,
[0068] The teacher model includes a Wav2Vec model containing several Transformer layers, a graph neural network and a classification layer;
[0069] The student model includes a Wav2Vec model containing several Transformer layers, a graph neural network and a classification layer; wherein a LoRA-based adapter module is inserted into the Transformer layer;
[0070] The number of Transformer layers in the teacher model is the same as the number of Transformer layers in the student model.
[0071] The present application may also provide a computer device, comprising: at least one processor, a memory, at least one network interface and a user interface. The various components in the device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.
[0072] The user interface may include a display, a keyboard or a pointing device, such as a mouse, a trackball, a touch pad or a touch screen.
[0073] It is understood that the memory in the embodiments disclosed in the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.
[0074] In some embodiments, the memory stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and applications.
[0075] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present disclosure can be included in the application.
[0076] In the above embodiment, the processor may also call a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and is used to:
[0077] Execute the steps of the above method.
[0078] The above method can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The above-disclosed methods, steps and logic block diagrams can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the above-disclosed method can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0079] It is understood that the embodiments described in the present application can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.
[0080] For software implementation, the technology of the present application can be implemented by executing the functional modules (such as procedures, functions, etc.) of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.
[0081] The present application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, each step in the above method embodiment can be implemented.
[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.
Claims
1. A single-classification forged speech detection method based on a teacher-student framework, comprising: Input the target speech into the trained teacher model and student model respectively to obtain the final forged speech detection result; The method of obtaining the final forged speech detection result includes: extracting the hidden features of the target speech through the teacher model and the student model respectively, calculating the cosine similarity of the features output by the teacher model and the student model, and when the cosine similarity is greater than a set threshold, the target speech is a real speech, otherwise it is a forged speech.
2. The single-classification forged speech detection method based on the teacher-student framework according to claim 1 is characterized in that: The teacher model includes a Wav2Vec model containing several Transformer layers, a graph neural network and a classification layer; The student model includes a Wav2Vec model containing several Transformer layers and a graph neural network; wherein a LoRA-based adapter module is inserted into the Transformer layer; The number of Transformer layers in the teacher model is the same as the number of Transformer layers in the student model.
3. The single-classification forged speech detection method based on the teacher-student framework according to claim 2 is characterized in that: The operation process of the teacher model includes: the target speech first passes through the Wav2Vec model to obtain a two-dimensional intermediate feature, then passes through the graph neural network to model the relationship between time-frequency domain features, and finally inputs the spliced time-domain features and frequency-domain features into the classification layer to obtain a binary classification result; The operation process of the student model includes: the target speech first passes through the Wav2Vec model to obtain two-dimensional intermediate features, and then passes through the graph neural network to model the time-frequency domain feature relationship.
4. The single-classification forged speech detection method based on the teacher-student framework according to claim 3 is characterized in that: The process of modeling the time-frequency domain feature relationship of the graph neural network includes: Step A1), respectively perform pooling based on the attention mechanism on the time domain and frequency domain features to obtain time domain features and frequency domain features; Step A2), modeling the time domain features and the frequency domain features as points, and extracting features from the time domain features and the frequency domain features through a graph attention network and a graph maximum pooling network respectively; Step A3), jointly modeling the time domain features and frequency domain features extracted in step A2) through a heterogeneous graph attention network; Step A4), perform maximum pooling on the time domain features and frequency domain features of the joint modeling in the time domain and frequency domain respectively, and separate the time domain features and the frequency domain features.
5. The single-classification forged speech detection method based on the teacher-student framework according to claim 4 is characterized in that: The classification layer includes two linear layers; the linear layer performs binary classification on the time domain features and frequency domain features output by the graph neural network to obtain classification results.
6. The single-classification forged speech detection method based on the teacher-student framework according to claim 2 is characterized in that: The training process of the teacher model includes: Input the real samples and forged samples into the Wav2Vec model for training and extract the robust features of the samples; The robust features are input into the graph neural network and classification layer for training to obtain the trained teacher model.
7. The single-classification forged speech detection method based on the teacher-student framework according to claim 2 is characterized in that: The training process of the student model includes: The real samples are input into the student model for training. The output of the corresponding Transformer layer of the student model learns from the embedding of the corresponding Transformer layer of the teacher model, and the loss function to be optimized is the cosine loss. The graph neural network of the student model learns from the embedding output of the graph neural network of the teacher model.
8. A single-classification forged speech detection system based on a teacher-student framework, implemented based on any of the methods described in claims 1-7, characterized in that: The system includes a teacher model and a student model; wherein, The teacher model includes a Wav2Vec model containing several Transformer layers, a graph neural network and a classification layer; The student model includes a Wav2Vec model containing several Transformer layers and a graph neural network; wherein a LoRA-based adapter module is inserted into the Transformer layer; The number of Transformer layers in the teacher model is the same as the number of Transformer layers in the student model.