False voice detection method based on self-supervised pre-training and supervised fine tuning

Through self-supervised pre-training and supervised fine-tuning methods, acoustic features are extracted and fake voice detection is performed, which solves the problem of low recognition rate and weak generalization ability of voiceprint recognition system in fake voice attacks, and improves the recognition of fake voice ability.

CN119091912APending Publication Date: 2024-12-06NANJING DINGSHAN INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411064084.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing voiceprint recognition system has low recognition rate and weak generalization ability when facing forged voice attacks, especially the lack of the ability to identify fake audio for unknown algorithms, making it difficult to distinguish between real voice and fake voice.

Method used

Using a forged speech detection method based on self-supervised pre-training and supervised fine-tuning, the Mel cepshot, non-periodic components and logarithmic fundamental frequency are extracted, neural network processing and global contrast filtering are performed to generate converted speech, and input them into self-directed. Supervise the pre-training detection network forged speech detection.

Benefits of technology

The recognition accuracy and generalization ability of forged speech detection are improved, the recognition ability of forged speech is enhanced, and the real voice and fake voice can be more effectively distinguished.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091912A_ABST
    Figure CN119091912A_ABST
Patent Text Reader

Abstract

The invention discloses a forged voice detection method based on self-supervised pre-training and supervised fine tuning. The method comprises the following steps: acquiring a target voice, and extracting a Mel-cepstrum, an aperiodic component and a logarithmic fundamental frequency of the target voice; inputting the Mel-cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum; performing global contrast one-time filtering on the target amplitude spectrum to obtain a to-be-converted voice Mel spectrum; synthesizing the to-be-converted voice Mel spectrum, the aperiodic component and the logarithmic fundamental frequency to obtain converted voice; and inputting the converted voice into a self-supervised pre-training detection network, and checking whether the converted voice is a forged voice or not. According to the forged voice detection method based on self-supervised pre-training provided by the invention, the deficiency of forged information extracted by traditional features is made up by introducing pre-training, characterization with stronger ability is extracted by utilizing self-supervised pre-training, and the forged voice data with labels are used for fine tuning, so that the recognition accuracy and the generalization detection ability are greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of forged voice detection, and in particular to a forged voice detection method based on self-supervised pre-training and supervised fine-tuning. Background Art

[0002] As a biometric authentication method, voiceprint recognition has been increasingly widely used in scenarios such as smart homes and e-commerce due to its convenience, reliability and low cost. However, existing studies have shown that voiceprint recognition systems are very vulnerable to forged voice attacks, and voiceprint systems cannot distinguish between real natural people and malicious forged attack voices. At present, deception attacks can be mainly divided into the following four categories: voice imitation, timbre conversion, voice synthesis and recording playback. Generally speaking, voice synthesis and timbre conversion of forged voices in the digital domain are also called logical attacks. With the development of deep learning technology, the naturalness and similarity of deep synthesized voices have made significant progress. The threat of logical attacks to voiceprint systems will also become greater, making voice forgery detection technology face greater challenges. Therefore, this article mainly focuses on logical attack anti-deception and conducts corresponding research work from synthetic voice detection. At present, the synthetic voice detection technology framework mainly adopts a combination of front-end feature engineering and back-end binary classifiers to extract acoustic features and train classification models for recognition. The forgery representation extracted by existing methods is not strong, which leads to low recognition rate and weak generalization ability of the model when facing forged audio using unknown algorithms. Therefore, studying how to construct forged representations with strong generalization and robustness is a key issue in improving the effect of forged speech detection. Summary of the invention

[0003] In order to solve the above-mentioned technical problems, the present invention provides a forged speech detection method based on self-supervised pre-training and supervised fine-tuning to solve the technical problems that the existing prediction models have low recognition rate and weak generalization ability when facing forged audio by unknown algorithms.

[0004] A forged speech detection method based on self-supervised pre-training and supervised fine-tuning, comprising:

[0005] Acquire a target speech, and extract a Mel-frequency cepstrum, aperiodic components, and a logarithmic fundamental frequency of the target speech;

[0006] Inputting the Mel-frequency cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum;

[0007] Performing a global contrast one-time filtering on the target amplitude spectrum to obtain a mel spectrum of the speech to be converted;

[0008] The mel spectrum, the non-periodic component and the logarithmic fundamental frequency of the speech to be converted are synthesized to obtain the converted speech;

[0009] The converted speech is input into a self-supervised pre-trained detection network to check whether the converted speech is a forged speech.

[0010] Preferably, the step of acquiring the target speech and extracting the Mel-frequency cepstrum, the non-periodic component and the logarithmic fundamental frequency of the target speech comprises:

[0011] Extract features from the source speaker's speech to obtain target speech feature parameters;

[0012] The target speech feature parameters include the Mel-frequency cepstrum, the non-periodic component and the logarithmic fundamental frequency.

[0013] Preferably, the step of inputting the Mel-frequency cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum comprises:

[0014] Inputting the Mel-cepstrum input into the first memory layer through a first nonlinear activation function;

[0015] The output of the first memory layer is input into the second memory layer through a second nonlinear activation function;

[0016] The output of the second memory layer is input into the third memory layer through a third nonlinear activation function;

[0017] The output of the second memory layer is input into the fully connected output layer through a fourth nonlinear activation function to obtain a converted Mel cepstrum;

[0018] The converted Mel-frequency cepstrum is used to obtain the target amplitude spectrum through a loss function.

[0019] Preferably, the neural network conversion model further includes a residual connection:

[0020] A first residual connection is performed before the first nonlinear activation function and after the first memory layer;

[0021] A second residual connection is performed before the second nonlinear activation function and after the second memory layer;

[0022] A third residual connection is performed before the third nonlinear activation function and after the third memory layer;

[0023] A fourth residual connection is performed before the second non-linear activation function and after the third memory layer.

[0024] Preferably, performing global contrast one-time filtering on the target amplitude spectrum to obtain the mel spectrum of the speech to be converted includes:

[0025] Calculating the mean variance of the target amplitude spectrum;

[0026] According to the mean variance of the target amplitude spectrum, a global mean square error consistent filter is constructed to obtain primary filtered data;

[0027] According to the primary filtered data, parameters are set and filtered to obtain the mel spectrum of the speech to be converted.

[0028] Preferably, the step of inputting the converted speech into a self-supervised pre-trained detection network to check whether the converted speech is a forged speech comprises:

[0029] Pre-training the self-supervised pre-trained detection network to obtain a primary self-supervised pre-trained detection network;

[0030] Performing supervised training of multi-layer linear transformation and non-linear activation on the primary self-supervised pre-trained detection network to extract forged representations;

[0031] By using the forgery representation, a category projection of speech to forgery detection is constructed to obtain a final self-supervised pre-trained detection network;

[0032] The converted speech is input into the final self-supervised pre-trained detection network to detect whether the converted speech is a forged speech.

[0033] Preferably, the pre-training the self-supervised pre-trained detection network to obtain a primary self-supervised pre-trained detection network includes:

[0034] The original speech is input into the feature encoding layer, and after multiple layers of convolution, the hidden speech representation is output;

[0035] The output generated by the feature encoding layer is input into the context encoding layer to obtain context representation;

[0036] The output generated by the feature encoding layer is input into the quantization module, and the output is quantized and represented;

[0037] The self-supervised pre-trained detection network is initialized using the quantized representation, the context representation and the hidden layer speech representation to obtain a primary self-supervised pre-trained detection network.

[0038] Preferably, the self-supervised pre-training detection network further includes a fine-tuning process:

[0039] The context representation is converted from the frame level to the sentence level through a pooling function;

[0040] Adding multiple fully connected layers to the quantized representation, the context representation and the hidden layer speech representation so that the outputs of the quantized representation, the context representation and the hidden layer speech representation are mapped into two dimensions;

[0041] The quantized representation, context representation and hidden layer speech representation are numerically regularized and outputted through a softmax function.

[0042] A forged speech detection system based on self-supervised pre-training and supervised fine-tuning, comprising:

[0043] A feature extraction module, used to obtain a target speech, and extract a Mel-frequency cepstrum, aperiodic components, and a logarithmic fundamental frequency of the target speech;

[0044] A neural network module, used for inputting the Mel-frequency cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum;

[0045] A filtering module is used to perform a global contrast one-time filtering on the target amplitude spectrum to obtain a mel spectrum of the speech to be converted;

[0046] A speech synthesis module, used for synthesizing the mel spectrum, non-periodic component and logarithmic fundamental frequency of the speech to be converted to obtain the converted speech;

[0047] The detection module is used to input the converted speech into a self-supervised pre-trained detection network to check whether the converted speech is a forged speech.

[0048] An electronic device comprises: a processor and a memory, wherein the memory is used to store computer program code, and the computer program code comprises computer instructions. When the processor executes the computer instructions, the electronic device executes a forged speech detection method based on self-supervised pre-training and supervised fine-tuning as described in any one of claims 1 to 8.

[0049] The present invention discloses a forged speech detection method based on self-supervised pre-training and supervised fine-tuning. The method comprises obtaining a target speech, extracting the Mel cepstrum, non-periodic components and logarithmic fundamental frequency of the target speech; inputting the Mel cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum; performing global contrast one-time filtering on the target amplitude spectrum to obtain the Mel spectrum of the speech to be converted; synthesizing the Mel spectrum, non-periodic components and logarithmic fundamental frequency of the speech to be converted to obtain a converted speech; inputting the converted speech into a self-supervised pre-training detection network to check whether the converted speech is a forged speech. The present invention proposes a forged speech detection method based on self-supervised pre-training, which makes up for the deficiency of forged information extracted by traditional acoustic features by introducing pre-training, utilizes self-supervised pre-training of a large amount of unlabeled data to extract stronger representations, adds a fully connected layer in the network after pre-training is completed, and uses a labeled forged speech data set to fine-tune the entire network parameters to realize forged speech detection, thereby greatly improving the recognition accuracy and generalization detection capability.

[0050] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0052] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions disclosed in the present invention.

[0053] Figure 1 A schematic flow chart of a method for detecting forged speech based on self-supervised pre-training and supervised fine-tuning provided in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of converted speech is obtained by a forged speech detection method based on self-supervised pre-training and supervised fine-tuning provided in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a neural network conversion model provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of a self-supervised pre-training detection network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0059] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] In addition, in order to better illustrate the present invention, numerous specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present invention can be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0062] As a biometric authentication method, voiceprint recognition has been increasingly widely used in scenarios such as smart homes and e-commerce due to its convenience, reliability and low cost. However, existing studies have shown that voiceprint recognition systems are very vulnerable to forged voice attacks, and voiceprint systems cannot distinguish between real natural people and malicious forged attack voices. At present, deception attacks can be mainly divided into the following four categories: voice imitation, timbre conversion, voice synthesis and recording playback. Generally speaking, voice synthesis and timbre conversion of forged voices in the digital domain are also called logical attacks. With the development of deep learning technology, the naturalness and similarity of deep synthesized voices have made significant progress. The threat of logical attacks to voiceprint systems will also become greater, making voice forgery detection technology face greater challenges. Therefore, this article mainly focuses on logical attack anti-deception and conducts corresponding research work from synthetic voice detection. At present, the synthetic voice detection technology framework mainly adopts a combination of front-end feature engineering and back-end binary classifiers to extract acoustic features and train classification models for recognition. The forgery representation extracted by existing methods is not strong, which leads to low recognition rate and weak generalization ability of the model when facing forged audio using unknown algorithms. Therefore, studying how to construct forged representations with strong generalization and robustness is a key issue in improving the effect of forged speech detection.

[0063] The present invention discloses a forged speech detection method based on self-supervised pre-training and supervised fine-tuning. The method comprises obtaining a target speech, extracting the Mel cepstrum, non-periodic components and logarithmic fundamental frequency of the target speech; inputting the Mel cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum; performing global contrast one-time filtering on the target amplitude spectrum to obtain the Mel spectrum of the speech to be converted; synthesizing the Mel spectrum, non-periodic components and logarithmic fundamental frequency of the speech to be converted to obtain a converted speech; inputting the converted speech into a self-supervised pre-training detection network to check whether the converted speech is a forged speech. The present invention proposes a forged speech detection method based on self-supervised pre-training, which makes up for the deficiency of forged information extracted by traditional acoustic features by introducing pre-training, utilizes self-supervised pre-training of a large amount of unlabeled data to extract stronger representations, adds a fully connected layer in the network after pre-training is completed, and uses a labeled forged speech data set to fine-tune the entire network parameters to realize forged speech detection, thereby greatly improving the recognition accuracy and generalization detection capability.

[0064] Example 1

[0065] A forged speech detection method based on self-supervised pre-training and supervised fine-tuning, reference Figure 1 ,include:

[0066] S100, obtaining the target speech, extracting the Mel-frequency cepstrum, non-periodic component and logarithmic fundamental frequency of the target speech;

[0067] S200, inputting the Mel-frequency cepstrum into the neural network conversion model for processing to obtain the target amplitude spectrum;

[0068] S300, performing a global contrast one-time filtering on the target amplitude spectrum to obtain the mel spectrum of the speech to be converted;

[0069] S400, synthesize the Mel spectrum, non-periodic component and logarithmic fundamental frequency of the speech to be converted, refer to Figure 2 , get the converted speech;

[0070] S500, input the converted speech into the self-supervised pre-trained detection network to check whether the converted speech is a forged speech.

[0071] Preferably, acquiring the target speech and extracting the Mel-frequency cepstrum, the non-periodic component and the logarithmic fundamental frequency of the target speech include:

[0072] Extract features from the source speaker's speech to obtain target speech feature parameters;

[0073] The target speech feature parameters include Mel-frequency cepstrum, non-periodic component and logarithmic fundamental frequency.

[0074] Preferably, the Mel cepstrum is input into the neural network conversion model for processing to obtain the target amplitude spectrum. Figure 3,include:

[0075] Input the Mel cepstrum input into the first memory layer through the first nonlinear activation function;

[0076] The output of the first memory layer is input into the second memory layer through a second nonlinear activation function;

[0077] The output of the second memory layer is input into the third memory layer through a third nonlinear activation function;

[0078] The output of the second memory layer is input to the fully connected output layer through the fourth nonlinear activation function to obtain the converted Mel cepstrum;

[0079] The converted Mel cepstrum is used to obtain the target amplitude spectrum through the loss function;

[0080] X represents the Mel cepstrum of the input source speech and Y represents the Mel cepstrum of the target speech. The conversion mapping function is f, then the converted speech Mel cepstrum It can be expressed as:

[0081]

[0082] but Z function between Y:

[0083]

[0084] Where M represents the total number of blocks in the local Mel-frequency cepstrum, and the image blocks are represented by xi and is the pixel center, i represents the index of the pixel block;

[0085] The evaluation basis of the Z function is that the higher its value, the closer the converted Mel cepstrum is to the target Mel cepstrum. Since the loss function is as small as possible, the loss function can be expressed as:

[0086]

[0087] Where N represents the total number of training samples. n represents the quantity index, and the network parameters are updated by minimizing J. θ represents the network parameters.

[0088] Preferably, for the neural network conversion model, refer to Figure 3 , also including residual connections:

[0089] A first residual connection is performed before the first nonlinear activation function and after the first memory layer;

[0090] A second residual connection is performed before the second nonlinear activation function and after the second memory layer;

[0091] A third residual connection is performed before the third nonlinear activation function and after the third memory layer;

[0092] A fourth residual connection is made before the second nonlinear activation function and after the third memory layer.

[0093] Preferably, the target amplitude spectrum is subjected to global contrast one-time filtering to obtain the mel spectrum of the speech to be converted, including:

[0094] Calculate the mean variance of the target magnitude spectrum:

[0095]

[0096] Where N represents the number of target sentences in the training phase, M represents the number of frames contained in each sentence, T represents the dimension of the Mel spectrum, i represents the i-th dimension Mel spectrum, and the mean value of each dimension of all frames of all training sentences is calculated. and variance tar represents the target statement;

[0097] According to the mean variance of the target amplitude spectrum, a global mean square error consistent filter is constructed to obtain the primary filtered data:

[0098]

[0099] where x tar represents the vector composed of the mean of the Mel spectrum of each dimension of the target sentence, x con represents the vector composed of the mean of the mel spectrum of each dimension of the converted sentence, and y represents the mel spectrum vector of the sentence to be converted;

[0100] According to the primary filtered data, set the parameters and filter to obtain the Mel spectrum of the speech to be converted:

[0101]

[0102] in, The mel spectrum of the speech to be converted.

[0103] Preferably, the converted speech is input into a self-supervised pre-trained detection network to check whether the converted speech is a forged speech, including:

[0104] Pre-training the self-supervised pre-trained detection network to obtain a primary self-supervised pre-trained detection network;

[0105] The primary self-supervised pre-trained detection network is supervisedly trained with multiple layers of linear transformation and nonlinear activation to extract the forged representation.

[0106] Through the forgery representation, the category projection from speech to forgery detection is constructed to obtain the final self-supervised pre-trained detection network;

[0107] The converted speech is input into the final self-supervised pre-trained detection network to detect whether the converted speech is a fake speech.

[0108] Preferably, the self-supervised pre-trained detection network is pre-trained to obtain a primary self-supervised pre-trained detection network, referring to Figure 4 ,include:

[0109] The original speech is input into the feature encoding layer, and after multiple layers of convolution, the hidden speech representation is output;

[0110] The output generated by the feature encoding layer is input into the context encoding layer to obtain context representation;

[0111] Feature encoding layer f(·): X→Z takes the original speech X as input and outputs the hidden speech representation Z = [z1, z2, …, zT] after multiple layers of convolution;

[0112] The feature encoding layer stacks multiple CNN modules, each of which contains one-dimensional convolution, layer normalization and GELU activation function. The convolution step size determines the number of final time steps T.

[0113] Context encoding layer g(·): Z→C uses Z generated by the feature encoding layer as input to obtain the context representation C. Before sending Z to the context network, all Z will be randomly sampled as the starting index with a certain probability, and then M consecutive time steps will be masked starting from each sampling position;

[0114] The context encoding layer is mainly composed of multiple Transformer layers. Each Transformer contains a multi-head attention part, a layer normalization part, and a fully connected layer. Compared with the standard Transformer structure, wav2vec2.0 replaces the absolute position encoding with a relative position encoding using convolution and activation functions.

[0115] The output generated by the feature encoding layer is input into the quantization module, and the output is quantized and represented;

[0116] Initialize the self-supervised pre-trained detection network using the quantized representation, the context representation, and the hidden layer speech representation to obtain a primary self-supervised pre-trained detection network;

[0117] The loss function of the self-supervised pre-trained detection network is:

[0118] L=L m +αL d

[0119] Among them, the contrast task loss L m It is required to identify the correct quantized hidden speech representation from a set of wrong counter-example interferences in the mask time step region, and the codebook difference loss L dis the codebook entry that encourages the model to average, and α is an adjustable parameter.

[0120] Preferably, the self-supervised pre-trained detection network, refer to Figure 4 , also includes fine-tuning processing:

[0121] The context representation is converted from the frame level to the sentence level through the pooling function;

[0122] Add multiple fully connected layers to the quantized representation, the context representation, and the hidden speech representation, so that the outputs of the quantized representation, the context representation, and the hidden speech representation are mapped into two dimensions;

[0123] The quantized representation, context representation and hidden speech representation are numerically regularized and output through the softmax function;

[0124] The average pooling function is used to convert the frame-level representation into the sentence-level representation;

[0125] After the pooling layer, multiple fully connected layers are added to map the final output into two dimensions, representing true or false sounds respectively. The network parameters of the fully connected layer are randomly initialized. The final output of the fully connected layer is numerically normalized by the softmax function, and the cross entropy loss function is used to guide the training of the entire network:

[0126]

[0127] Among them, P n is the probability that speech X belongs to category n. If n is the target category, then y n =1, otherwise y n =0, N is the total number of categories, and N=2 represents two categories: real natural sound and fake speech.

[0128] Example 2

[0129] A forged speech detection system based on self-supervised pre-training and supervised fine-tuning, comprising:

[0130] A feature extraction module is used to obtain the target speech and extract the Mel-frequency cepstrum, non-periodic component and logarithmic fundamental frequency of the target speech;

[0131] A neural network module is used to input the Mel-frequency cepstrum into the neural network conversion model for processing to obtain a target amplitude spectrum;

[0132] The filtering module is used to perform a one-time global contrast filtering on the target amplitude spectrum to obtain the mel spectrum of the speech to be converted;

[0133] The speech synthesis module is used to synthesize the mel spectrum, non-periodic component and logarithmic fundamental frequency of the speech to be converted to obtain the converted speech;

[0134] The detection module is used to input the converted speech into the self-supervised pre-trained detection network to check whether the converted speech is a forged speech.

[0135] Example 3

[0136] An electronic device comprises: a processor and a memory, the memory is used to store computer program code, the computer program code comprises computer instructions, when the processor executes the computer instructions, the electronic device executes a forged speech detection method based on self-supervised pre-training and supervised fine-tuning according to any one of claims 1 to 8.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present invention have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0139] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0143] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A forged speech detection method based on self-supervised pre-training and supervised fine-tuning, characterized in that: include: Acquire a target speech, and extract a Mel-frequency cepstrum, aperiodic components, and a logarithmic fundamental frequency of the target speech; Inputting the Mel-frequency cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum; Performing a global contrast one-time filtering on the target amplitude spectrum to obtain a mel spectrum of the speech to be converted; The mel spectrum, the non-periodic component and the logarithmic fundamental frequency of the speech to be converted are synthesized to obtain the converted speech; The converted speech is input into a self-supervised pre-trained detection network to check whether the converted speech is a forged speech.

2. A method for detecting fake speech based on self-supervised pre-training and supervised fine-tuning according to claim 1, characterized in that: The step of acquiring the target speech and extracting the Mel-frequency cepstrum, the non-periodic component and the logarithmic fundamental frequency of the target speech includes: Extract features from the source speaker's speech to obtain target speech feature parameters; The target speech feature parameters include the Mel-frequency cepstrum, the non-periodic component and the logarithmic fundamental frequency.

3. A method for detecting fake speech based on self-supervised pre-training and supervised fine-tuning according to claim 1, characterized in that: The step of inputting the Mel-frequency cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum includes: Inputting the Mel-cepstrum input into the first memory layer through a first nonlinear activation function; The output of the first memory layer is input into the second memory layer through a second nonlinear activation function; The output of the second memory layer is input into the third memory layer through a third nonlinear activation function; The output of the second memory layer is input into the fully connected output layer through a fourth nonlinear activation function to obtain a converted Mel cepstrum; The converted Mel-frequency cepstrum is used to obtain the target amplitude spectrum through a loss function.

4. A method for detecting fake speech based on self-supervised pre-training and supervised fine-tuning according to claim 3, characterized in that: The neural network conversion model further includes a residual connection: A first residual connection is performed before the first nonlinear activation function and after the first memory layer; A second residual connection is performed before the second nonlinear activation function and after the second memory layer; A third residual connection is performed before the third nonlinear activation function and after the third memory layer; A fourth residual connection is performed before the second non-linear activation function and after the third memory layer.

5. A method for detecting fake speech based on self-supervised pre-training and supervised fine-tuning according to claim 4, characterized in that: The step of performing global contrast one-time filtering on the target amplitude spectrum to obtain the mel spectrum of the speech to be converted includes: Calculating the mean variance of the target amplitude spectrum; According to the mean variance of the target amplitude spectrum, a global mean square error consistent filter is constructed to obtain primary filtered data; According to the primary filtered data, parameters are set and filtered to obtain the mel spectrum of the speech to be converted.

6. A method for detecting fake speech based on self-supervised pre-training and supervised fine-tuning according to claim 1, characterized in that: The step of inputting the converted speech into a self-supervised pre-trained detection network to check whether the converted speech is a forged speech comprises: Pre-training the self-supervised pre-trained detection network to obtain a primary self-supervised pre-trained detection network; Performing supervised training of multi-layer linear transformation and non-linear activation on the primary self-supervised pre-trained detection network to extract forged representations; By using the forgery representation, a category projection of speech to forgery detection is constructed to obtain a final self-supervised pre-trained detection network; The converted speech is input into the final self-supervised pre-trained detection network to detect whether the converted speech is a forged speech.

7. A method for detecting fake speech based on self-supervised pre-training and supervised fine-tuning according to claim 6, characterized in that: The pre-training of the self-supervised pre-trained detection network to obtain a primary self-supervised pre-trained detection network includes: The original speech is input into the feature encoding layer, and after multiple layers of convolution, the hidden speech representation is output; The output generated by the feature encoding layer is input into the context encoding layer to obtain context representation; The output generated by the feature encoding layer is input into the quantization module, and the output is quantized and represented; The self-supervised pre-trained detection network is initialized using the quantized representation, the context representation and the hidden layer speech representation to obtain a primary self-supervised pre-trained detection network.

8. A method for detecting fake speech based on self-supervised pre-training and supervised fine-tuning according to claim 7, characterized in that: The self-supervised pre-training detection network also includes a fine-tuning process: The context representation is converted from the frame level to the sentence level through a pooling function; Adding multiple fully connected layers to the quantized representation, the context representation and the hidden layer speech representation so that the outputs of the quantized representation, the context representation and the hidden layer speech representation are mapped into two dimensions; The quantized representation, context representation and hidden layer speech representation are numerically regularized and outputted through a softmax function.

9. A forged speech detection system based on self-supervised pre-training and supervised fine-tuning, characterized in that: include: A feature extraction module, used to obtain a target speech, and extract a Mel-frequency cepstrum, aperiodic components, and a logarithmic fundamental frequency of the target speech; A neural network module, used for inputting the Mel-frequency cepstrum into a neural network conversion model for processing to obtain a target amplitude spectrum; A filtering module is used to perform a global contrast one-time filtering on the target amplitude spectrum to obtain a mel spectrum of the speech to be converted; A speech synthesis module, used for synthesizing the mel spectrum, non-periodic component and logarithmic fundamental frequency of the speech to be converted to obtain the converted speech; The detection module is used to input the converted speech into a self-supervised pre-trained detection network to check whether the converted speech is a forged speech.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program code, wherein the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes a forged speech detection method based on self-supervised pre-training and supervised fine-tuning as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • BLSTM-based frequency division spectrum extension anti-noise voice conversion method

    CN112562704A

  • Cross-language timbre conversion system and method based on zero-order learning

    CN112767958A

  • Synthetic speech processing method and related device

    CN113066472A

  • Unsupervised pre-trained speech recognition model with enhanced local dependency relationship, and training method thereof

    CN113380237A

  • Forged voice detection method based on multi-feature fusion and device thereof

    CN113488073A

Cited By

  • Deep synthetic audio cross-domain detection method and device based on self-supervised auxiliary task

    CN119479611A

  • Audio depth forgery detection method and device, computer equipment and storage medium

    CN119694319A

  • Audio deepfake detection method and device, computer device, and storage medium

    CN119694319B

  • Deep forged audio detection protection method based on large-scale pre-training model Whisper

    CN120126481A

  • Multi-modal target identification method, system and equipment based on infrared radar composite signal

    CN121743842A