Speech authentic identification model training method and device, equipment and storage medium

By extracting and fusing multi-level features, the speech pseudo-detection model is trained, which solves the problems of weak generalization ability and low detection accuracy in the existing technology, and effectively detects advanced fake speech and improves resource efficiency.

CN119028322AActive Publication Date: 2024-11-26PING AN TECH (SHENZHEN) CO LTD

Patent Information

Application Number
CN202411129480.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The existing voice pseudo-recognition technology has weak generalization ability and low detection accuracy, making it difficult to effectively detect advanced forged voices, and the training process consumes a lot of resources and high data acquisition costs.

Method used

By obtaining false and real audio, use whisper, hubert and wav2vec2.0 pre-trained models to extract intermediate features, duration features and abstract features, fusion of multi-head attention mechanisms, input expert models and gated networks, generate fusion features and train voice To identify pseudo-false models, improve the model's adaptability to different types of fake pronunciations.

Benefits of technology

It enhances the generalization ability of the speech pseudo-identification model, improves the detection accuracy of complex forged speech, and reduces the consumption of training resources and data acquisition costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028322A_ABST
    Figure CN119028322A_ABST
Patent Text Reader

Abstract

The invention relates to a voice authentic identification model training method, and the method comprises the steps: obtaining a first preset number of false audios, and obtaining a second preset number of real audios; and respectively extracting intermediate features, duration features and abstract features of the false audio and the real audio through a whisper pre-training model, a hubert pre-training model and a wav2vec2.0 pre-training model. And fusing the intermediate feature, the duration feature and the abstract feature to generate a fused feature, and inputting the fused feature into the expert model and the gating network to obtain the expert model output and the weight of the expert model output. And training the expert model and the gating network according to the expert model output and the weight of the expert model output to obtain a voice authentic identification model. The multi-level features are extracted in combination with multiple pre-training models, and feature fusion is performed through an Attention mechanism, so that the adaptability of the model to different types of forged voices can be enhanced, and the generalization ability is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a training method, device, equipment and storage medium for a voice authentication model. Background Art

[0002] With the rapid development of artificial intelligence technology, text-to-speech (TTS) and voice conversion (VC) technologies have also made significant progress. These technologies have been widely used in many fields such as voice assistants, intelligent customer service, barrier-free communication, entertainment and education. Modern TTS and VC technologies can generate high-quality and highly natural speech, which is of great significance in improving user experience and application scenarios. However, the advancement of these technologies has also brought new challenges and risks, especially in the field of voice authentication.

[0003] Traditional speech authentication methods mainly rely on manually designed features, such as Mel-frequency cepstral coefficients (MFCC) and constant Q cepstral coefficients (CQCC). Although these features can capture certain speech information, their expressive power is limited and they are difficult to deal with complex forged speech. Manually designed features usually cannot fully reflect the deep information of speech, resulting in poor performance when facing advanced forgery techniques.

[0004] Neural network-based speech authentication methods use deep learning technology to automatically learn features from large amounts of data. This type of method shows high accuracy in specific scenarios, but its generalization ability is poor, and it is often difficult to effectively detect unseen forged speech. In addition, neural network models usually require a large amount of labeled data for training, which has high data acquisition costs, long training time, and high resource consumption.

[0005] With the development of generative adversarial networks (GANs) and other advanced generative models, the quality and naturalness of forged voices have been continuously improved, making the task of voice forgery detection more difficult. Modern forged voice technology can generate audio that is extremely similar to real voices, which not only poses a potential threat to users, but also brings huge challenges to financial security, identity authentication, and public safety. Summary of the invention

[0006] The main purpose of the present invention is to provide a training method, device, equipment and storage medium for a speech authentication model, aiming to solve the technical problems of weak generalization ability and low detection accuracy of existing speech authentication technology.

[0007] To achieve the above object, the present invention provides a method for training a speech authentication model, the method comprising the following steps:

[0008] Acquire a first preset number of false audios, and acquire a second preset number of real audios;

[0009] The intermediate features, duration features and abstract features of the fake audio and the real audio are respectively extracted through the whisper pre-training model, the hubert pre-training model and the wav2vec2.0 pre-training model;

[0010] The intermediate features, duration features and abstract features are fused to generate fused features, and the fused features are input into the expert model and the gating network to obtain the output of the expert model and the weight of the output of the expert model;

[0011] The expert model and the gating network are trained according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

[0012] Optionally, the step of fusing the intermediate features, duration features and abstract features to generate fused features includes:

[0013] The intermediate features, duration features and abstract features are fused using a multi-head attention mechanism. The intermediate features are used as Key (K) and Value (V), and the duration features and abstract features are used as Query (Q) for self-attention operation. The calculation formula is:

[0014]

[0015] Among them, Q stands for Query, K stands for Key, V stands for Value, and d k Indicates the dimension of the Key;

[0016] The fusion feature is generated according to the Attention output.

[0017] Optionally, the step of inputting the fusion feature into an expert model and a gating network to obtain an expert model output and a weight of the expert model output comprises:

[0018] The expert model includes three expert networks with different structures, and the fusion features are respectively input into the three expert networks to obtain the expert model output;

[0019] The gating network uses a softmax function to model the probability distribution, and the fusion feature is input into the gating network to obtain the weight output by the expert model.

[0020] Optionally, the step of training the expert model and the gating network according to the expert model output and the weight of the expert model output includes:

[0021] Input the expert model output and the weight of the expert model output into two fully connected layers to obtain a predicted label;

[0022] The expert model and the gating network are trained according to the predicted label using a binary cross entropy loss function, and the parameters of the expert model and the gating network are updated.

[0023] Optionally, the step of inputting the expert model output and the weight of the expert model output into two fully connected layers includes:

[0024] The expert model output is weightedly combined with the weight of the expert model output to generate a target feature output, and the target feature output is input into the two fully connected layers.

[0025] Optionally, after the steps of obtaining a first preset number of false audios and obtaining a second preset number of real audios, the method further includes:

[0026] Noise is randomly added to the false audio and the real audio according to the first preset number and the second preset number to perform a data enhancement operation.

[0027] Optionally, after the step of obtaining the voice authentication model, the method further includes:

[0028] The audio to be detected is obtained, and the audio to be detected is input into the voice authentication model to obtain a authenticity classification result of the audio to be detected.

[0029] Furthermore, in order to achieve the above-mentioned object, the present invention also provides a training device for a speech authentication model, the training device for a speech authentication model comprising:

[0030] An audio acquisition module, acquiring a first preset amount of false audio and a second preset amount of real audio;

[0031] A feature extraction module, which extracts intermediate features, duration features and abstract features of the false audio and the real audio respectively through a whisper pre-training model, a hubert pre-training model and a wav2vec2.0 pre-training model;

[0032] A feature fusion module, which fuses the intermediate features, duration features and abstract features to generate fused features, and inputs the fused features into an expert model and a gating network to obtain an output of the expert model and a weight of the output of the expert model;

[0033] The model training module trains the expert model and the gated network according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

[0034] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a training device for a speech authentication model, the training device for the speech authentication model comprising a memory, a processor, and a training program for the speech authentication model stored in the memory and executable on the processor, and the training program for the speech authentication model, when executed by the processor, implements the steps of the training method for the speech authentication model as described above.

[0035] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a training program for a speech authentication model is stored, and when the training program for the speech authentication model is executed by a processor, the steps of the training method for the speech authentication model as described above are implemented.

[0036] The present invention relates to a training method for a speech authentication model, which obtains a first preset number of false audios and a second preset number of real audios. Then, the intermediate features, duration features and abstract features of the false audios and the real audios are extracted respectively through a whisper pre-training model, a huber pre-training model and a wav2vec2.0 pre-training model. The intermediate features, duration features and abstract features are fused to generate fused features, and the fused features are input into an expert model and a gating network to obtain the output of the expert model and the weight of the expert model output. The expert model and the gating network are trained according to the output of the expert model and the weight of the expert model output to obtain a speech authentication model. The present invention combines a plurality of pre-training models to extract multi-level features, and performs feature fusion through an Attention mechanism, which can enhance the adaptability of the model to different types of forged speech, thereby improving the generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the training device for the speech authentication model of the present invention;

[0038] Figure 2 It is a flow chart of the first embodiment of the training method of the speech authentication model of the present invention;

[0039] Figure 3 The figure is a functional module diagram of a preferred embodiment of the training device for the speech authentication model of the present invention.

[0040] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0042] It should be noted that with the rapid development of artificial intelligence technology, text-to-speech (TTS) and voice conversion (VC) technologies have also made significant progress. These technologies have been widely used in many fields such as voice assistants, intelligent customer service, barrier-free communication, entertainment and education. Modern TTS and VC technologies can generate high-quality and highly natural speech, which is of great significance in improving user experience and application scenarios. However, the advancement of these technologies has also brought new challenges and risks, especially in the field of voice authentication.

[0043] Traditional speech authentication methods mainly rely on manually designed features, such as Mel-frequency cepstral coefficients (MFCC) and constant Q cepstral coefficients (CQCC). Although these features can capture certain speech information, their expressive power is limited and they are difficult to deal with complex forged speech. Manually designed features usually cannot fully reflect the deep information of speech, resulting in poor performance when facing advanced forgery techniques.

[0044] Neural network-based speech authentication methods use deep learning technology to automatically learn features from large amounts of data. This type of method shows high accuracy in specific scenarios, but its generalization ability is poor, and it is often difficult to effectively detect unseen forged speech. In addition, neural network models usually require a large amount of labeled data for training, which has high data acquisition costs, long training time, and high resource consumption.

[0045] With the development of generative adversarial networks (GANs) and other advanced generative models, the quality and naturalness of forged voices have been continuously improved, making the task of voice forgery detection more difficult. Modern forged voice technology can generate audio that is extremely similar to real voices, which not only poses a potential threat to users, but also brings huge challenges to financial security, identity authentication, and public safety.

[0046] Based on the above-mentioned defects, the present invention provides a training device for a speech authentication model. Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the training device for the speech authentication model of the present invention.

[0047] like Figure 1As shown, the training device of the voice authentication model may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art will understand that Figure 1 The hardware structure of the training device for the speech authentication model shown in the figure does not constitute a limitation on the training device for the speech authentication model, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0049] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a training program for a voice authentication model. The operating system is a program for managing and controlling the training equipment and software resources of the voice authentication model, and supports the operation of the network communication module, the user interface module, the training program for the voice authentication model, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0050] exist Figure 1 In the hardware structure of the training device for the voice authentication model shown in the figure, the network interface 1004 is mainly used to connect to the background server and perform data communication with the background server; the user interface 1003 is mainly used to connect to the client and perform data communication with the client; the processor 1001 can call the training program of the voice authentication model stored in the memory 1005 and perform the following operations:

[0051] Acquire a first preset number of false audios, and acquire a second preset number of real audios;

[0052] The intermediate features, duration features and abstract features of the fake audio and the real audio are respectively extracted through the whisper pre-training model, the hubert pre-training model and the wav2vec2.0 pre-training model;

[0053] The intermediate features, duration features and abstract features are fused to generate fused features, and the fused features are input into the expert model and the gating network to obtain the output of the expert model and the weight of the output of the expert model;

[0054] The expert model and the gating network are trained according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

[0055] Furthermore, the step of fusing the intermediate features, duration features and abstract features to generate fused features includes:

[0056] The intermediate features, duration features and abstract features are fused using a multi-head attention mechanism. The intermediate features are used as Key (K) and Value (V), and the duration features and abstract features are used as Query (Q) for self-attention operation. The calculation formula is:

[0057]

[0058] Among them, Q stands for Query, K stands for Key, V stands for Value, and d k Indicates the dimension of the Key;

[0059] The fusion feature is generated according to the Attention output.

[0060] Furthermore, the step of inputting the fusion feature into the expert model and the gating network to obtain the expert model output and the weight of the expert model output includes:

[0061] The expert model includes three expert networks with different structures, and the fusion features are respectively input into the three expert networks to obtain the expert model output;

[0062] The gating network uses a softmax function to model the probability distribution, and the fusion feature is input into the gating network to obtain the weight output by the expert model.

[0063] Furthermore, the step of training the expert model and the gating network according to the expert model output and the weight of the expert model output includes:

[0064] Input the expert model output and the weight of the expert model output into two fully connected layers to obtain a predicted label;

[0065] The expert model and the gating network are trained according to the predicted label using a binary cross entropy loss function, and the parameters of the expert model and the gating network are updated.

[0066] Furthermore, the step of inputting the expert model output and the weight of the expert model output into two fully connected layers includes:

[0067] The expert model output is weightedly combined with the weight of the expert model output to generate a target feature output, and the target feature output is input into the two fully connected layers.

[0068] Furthermore, after the steps of obtaining a first preset number of false audios and obtaining a second preset number of real audios, the method further includes:

[0069] Noise is randomly added to the false audio and the real audio according to the first preset number and the second preset number to perform a data enhancement operation.

[0070] Furthermore, after the step of obtaining the voice authentication model, the method further includes:

[0071] The audio to be detected is obtained, and the audio to be detected is input into the voice authentication model to obtain a authenticity classification result of the audio to be detected.

[0072] The specific implementation of the training device for the speech authentication model of the present invention is basically the same as the various embodiments of the training method for the speech authentication model described below, and will not be described in detail here.

[0073] The present invention also provides a method for training a speech authentication model based on the training device for the speech authentication model.

[0074] Reference Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a method for training a speech authentication model according to the present invention.

[0075] The embodiment of the present invention provides an embodiment of a method for training a speech authentication model. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0076] In various embodiments of the method for training a speech authentication model, the execution subject is a training device for the speech authentication model.

[0077] The training method of the speech authentication model is applied to a training device of the speech authentication model, and the training method of the speech authentication model comprises:

[0078] Step S10, obtaining a first preset number of false audios, and obtaining a second preset number of real audios;

[0079] With the rapid development of artificial intelligence technology, text-to-speech (TTS) and voice conversion (VC) technologies have also made significant progress. These technologies have been widely used in many fields such as voice assistants, intelligent customer service, barrier-free communication, entertainment and education. Modern TTS and VC technologies can generate high-quality and highly natural speech, which is of great significance in improving user experience and application scenarios. However, the advancement of these technologies has also brought new challenges and risks, especially in the field of voice authentication.

[0080] Traditional speech authentication methods mainly rely on manually designed features, such as Mel-frequency cepstral coefficients (MFCC) and constant Q cepstral coefficients (CQCC). Although these features can capture certain speech information, their expressive power is limited and they are difficult to deal with complex forged speech. Manually designed features usually cannot fully reflect the deep information of speech, resulting in poor performance when facing advanced forgery techniques.

[0081] Neural network-based speech authentication methods use deep learning technology to automatically learn features from large amounts of data. This type of method shows high accuracy in specific scenarios, but its generalization ability is poor, and it is often difficult to effectively detect unseen forged speech. In addition, neural network models usually require a large amount of labeled data for training, which has high data acquisition costs, long training time, and high resource consumption.

[0082] With the development of generative adversarial networks (GANs) and other advanced generative models, the quality and naturalness of forged voices have been continuously improved, making the task of voice forgery detection more difficult. Modern forged voice technology can generate audio that is extremely similar to real voices, which not only poses a potential threat to users, but also brings huge challenges to financial security, identity authentication, and public safety.

[0083] The training method for a voice authentication model in this embodiment is applicable to a device having a function of the training method for a voice authentication model.

[0084] In this embodiment, data preparation is first completed to obtain false audio and real audio, and this embodiment does not specifically limit the number.

[0085] The audio data may be obtained by obtaining a first preset number of false audio samples from a public data set, and these samples may come from different types of speech generation technologies, such as a text-to-speech (TTS) system, a voice conversion (VC) system, speech generated by a generative adversarial network (GAN), etc. The first preset number may be adjusted according to actual needs, for example, 100,000 false audio samples may be selected as a reference.

[0086] A second preset number of real audio samples is obtained, which should include audio from different speakers, different contexts, and different recording environments. The second preset number can be adjusted according to actual needs. For example, 10,000 real audio samples can be selected as references to ensure that the model can learn enough real speech features.

[0087] In order to enhance the robustness and generalization ability of the model, data augmentation can be performed on fake and real audio. Specific methods include:

[0088] Add background noise: Randomly add different types of background noise to the audio, such as environmental noise, microphone noise, etc., to simulate various noise conditions in reality.

[0089] Change audio speed: Speed ​​up or slow down the audio to simulate different speaking speeds.

[0090] Shift Pitch: Shift the pitch of the audio to simulate different sound frequencies.

[0091] Cut and Splice: Randomly cut and splice audio to generate new audio clips.

[0092] After data enhancement, the number of fake audio and real audio should maintain a certain ratio to ensure the balance of the data set. For example, if 100,000 fake audio and 10,000 real audio are selected, noise and other enhancement processing can be randomly added in a ratio of 1:11.

[0093] Step S20, respectively extracting intermediate features, duration features, and abstract features of the false audio and the real audio through a whisper pre-training model, a huber pre-training model, and a wav2vec2.0 pre-training model;

[0094] In this embodiment, feature extraction may be performed in the following manner:

[0095] Whisper pre-trained model: Load the whisper pre-trained model, input fake audio and real audio, and extract intermediate features. These features capture the local spectral characteristics of the audio and help distinguish real audio from fake audio.

[0096] Hubert pre-trained model: Load the Hubert pre-trained model, input fake audio and real audio, and extract duration features. These features represent the temporal structure information of the audio and help detect anomalies in the temporal dimension of fake audio.

[0097] wav2vec2.0 pre-trained model: Load the wav2vec2.0 pre-trained model, input fake audio and real audio, and extract abstract features. These features contain high-level semantic information of the audio and help capture deeper forgery features.

[0098] Step S30, fusing the intermediate features, duration features and abstract features to generate fused features, inputting the fused features into an expert model and a gating network to obtain an output of the expert model and a weight of the output of the expert model;

[0099] In this embodiment, it is necessary to fuse the extracted intermediate features (whisper features), duration features (hubert features) and abstract features (wav2vec features).

[0100] One implementation method is to use a multi-head attention mechanism to fuse the extracted intermediate features (whisper features), duration features (hubert features) and abstract features (wav2vec features). The specific steps may include:

[0101] The self-attention operation is performed with the intermediate features as Key and Value, and the duration features and abstract features as Query. The calculation formula is:

[0102]

[0103] Concatenate the multiple results and project them into another vector to generate the final fused feature input.

[0104] Another implementation method is to perform feature concatenation and weighted averaging on the intermediate features (whisper features), duration features (hubert features) and abstract features (wav2vec features). The specific steps may include:

[0105] Feature concatenation: Whisper features, hubert features, and wav2vec features are directly concatenated together to form a larger feature vector. The concatenated feature vector contains the complete information of all three features.

[0106] Weighted average: For each feature, different weights are assigned according to its importance, and weighted average is performed. The specific formula is: fused_feature = α·whisper_feature + β·hubert_feature + γ·wav2vec_feature, where α, β and γ are weight coefficients, satisfying α+β+γ=1. These weights can be learned through training data.

[0107] After obtaining the fusion features, the fusion features are input into the expert model and the gating network to obtain the output of the expert model and the weight of the expert model output. The specific steps may include:

[0108] Expert model: The expert model can include multiple expert networks with different structures, each of which is responsible for further processing and extracting fusion features from different perspectives. For example, the expert model includes three expert networks with different structures, specifically:

[0109] Expert Model 1 (Expert1)

[0110] Structure: Shallow Convolutional Neural Network

[0111] Composition: convolution layer, four 2D convolution layers, each convolution layer is followed by an activation function (such as ReLU); pooling layer, maximum pooling layer, used for dimensionality reduction and extraction of local features.

[0112] Purpose: Capture local spectral features of audio, especially local anomalies in forged audio.

[0113] How it works: The low-level features of the audio are extracted through convolution operations, and the size of the feature map is reduced through the pooling layer to retain important local information.

[0114] Expert2

[0115] Structure: Fusion of residual blocks and FMS attention modules

[0116] Composition: residual block, three residual blocks, each residual block contains two convolutional layers and a shortcut connection; FMS attention module, used to adjust the weight of the feature map and emphasize important features.

[0117] Purpose: Capture deep features of audio, especially global anomalies in forged audio.

[0118] Working principle: Learn deeper features through residual blocks, and emphasize important features and suppress irrelevant information through the attention mechanism (FMS module).

[0119] Expert Model 3 (Expert3)

[0120] Structure: Three-layer feedforward neural network

[0121] Composition: fully connected layer, three fully connected layers, each layer is followed by an activation function (such as ReLU).

[0122] Purpose: Capture high-level abstract features of audio.

[0123] Working principle: The input features are mapped to a high-dimensional space through a fully connected layer to extract more complex feature representations.

[0124] In addition, the role of the gating network is to calculate the weights of each expert model output based on the input data. These weights represent the contribution of each expert model to the input data.

[0125] Input features: The gated network accepts fused features as input, which are obtained through multi-head attention mechanism or other feature fusion methods.

[0126] Calculate weights: The gating network uses the softmax function to calculate the weights of each expert model output. The softmax function normalizes the weights to a probability distribution, ensuring that the sum of all weights is 1. The specific formula is:

[0127]

[0128] Among them, a i represents the weight of the i-th expert model, W i Represents the weight matrix of the gating network, and input represents the fusion feature.

[0129] Step S40: training the expert model and the gating network according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

[0130] In this embodiment, the specific steps of training the expert model and the gating network according to the expert model output and weights may include:

[0131] Weighted combination: The output of the expert model is weighted and combined according to the weights generated by the gating network to generate the final feature output. The specific formula is:

[0132]

[0133] in, represents the output of the i-th expert model, a i is its corresponding weight.

[0134] Finally, the model is trained through the final feature output. The specific steps may include:

[0135] Input fusion features: Input the final feature output into two fully connected layers.

[0136] The first fully connected layer:

[0137] Structure: Contains the weight matrix W FC1 and bias b FC1 .

[0138] Formula: outputFC1 =ReLU(W FC1 final_output+b FC1 );

[0139] Among them, final_output is the weighted combined feature, and ReLU is the activation function.

[0140] The second fully connected layer:

[0141] Structure: Contains the weight matrix W FC2 and bias b FC2 ;

[0142] Formula: output FC2 =sigmoid(W FC2 Output FC1 +b FC2 );

[0143] Among them, output FC1 is the output of the first fully connected layer, and sigmoid is the activation function.

[0144] Choose the loss function: You can use the binary cross entropy loss function (BCEWithLogitLoss) as the loss function for training the model.

[0145] Loss function formula:

[0146]

[0147] in, is the sigmoid function, N is the total number of samples, yi is the actual label, x i is the predicted value, which is the output of the second fully connected layer FC2 .

[0148] Back-propagation algorithm: Use the back-propagation algorithm to calculate the gradient of the loss function.

[0149] Gradient calculation: Calculate the gradient of each parameter (including the weights and bias parameters of the expert model, gating network, and fully connected layer).

[0150] Parameter Update: Use gradient descent or its variants (such as Adam optimizer) to update the parameters.

[0151] Optimization process: The update formula for each parameter θ is:

[0152]

[0153] After training, a voice authentication model is obtained, which includes the following parts:

[0154] Whisper pre-trained model, HuBERT pre-trained model, wav2vec2.0 pre-trained model, expert model, gated network, fully connected layer.

[0155] The present invention obtains a first preset number of false audios and a second preset number of real audios. Then, the intermediate features, duration features and abstract features of the false audio and the real audio are extracted respectively through the whisper pre-training model, the hubert pre-training model and the wav2vec2.0 pre-training model. The intermediate features, duration features and abstract features are fused to generate fused features, and the fused features are input into the expert model and the gating network to obtain the output of the expert model and the weight of the expert model output. The expert model and the gating network are trained according to the output of the expert model and the weight of the expert model output to obtain a speech authentication model. The present invention combines a variety of pre-training models to extract multi-level features, and performs feature fusion through the Attention mechanism, which can enhance the adaptability of the model to different types of forged speech, thereby improving the generalization ability.

[0156] Furthermore, a second embodiment of the training method of the speech authentication model of the present invention is proposed. In the above step S30, the step of fusing the intermediate features, duration features and abstract features to generate fused features includes:

[0157] Step a1: Use the multi-head attention mechanism to fuse the intermediate features, duration features and abstract features, use the intermediate features as Key (K) and Value (V), and use the duration features and abstract features as Query (Q) to perform self-attention operation. The calculation formula is:

[0158]

[0159] Among them, Q stands for Query, K stands for Key, V stands for Value, and d k Indicates the dimension of the Key;

[0160] Step a2: generating the fusion feature according to the Attention output.

[0161] In this embodiment, the intermediate features, duration features and abstract features can be fused by using a multi-head attention mechanism. By using the multi-head attention mechanism, information of different features can be integrated to improve the model's ability to process complex data. The specific steps may include:

[0162] The self-attention operation is performed using the intermediate features as Key (K) and Value (V), and the duration features and abstract features as Query (Q).

[0163] Intermediate features (Key and Value): come from the Whisper pre-trained model and capture the local spectral characteristics of the audio.

[0164] Duration features and abstract features (Query): They come from the HuBERT pre-trained model and the wav2vec2.0 pre-trained model respectively, representing the temporal structure and high-level semantic information of the audio.

[0165] The calculation formula is:

[0166]

[0167] Generate the fused feature based on the Attention output: Calculate the dot product of Query and Key, and then divide by the square root of the Key dimension to get a scaled attention score matrix. Apply the softmax function to the attention score matrix to get the weight matrix. Multiply the weight matrix with the Value to get the weighted sum output, which is the fused feature.

[0168] Through the above steps, this embodiment can focus on different parts of the input at the same time, capture multi-scale feature information, improve the detection effect, and integrate the features of different pre-trained models to generate a richer and more comprehensive feature representation, thereby enhancing the stability of the model under different forged audio types.

[0169] Furthermore, a third embodiment of the training method of the speech authentication model of the present invention is proposed. In the above step S30, the step of inputting the fusion feature into the expert model and the gating network to obtain the expert model output and the weight of the expert model output includes:

[0170] Step b1, the expert model includes three expert networks with different structures, and the fusion features are respectively input into the three expert networks to obtain the expert model output;

[0171] Step b2, the gating network uses a softmax function to model the probability distribution, inputs the fusion feature into the gating network, and obtains the weight output by the expert model.

[0172] In this embodiment, the expert model includes three expert networks with different structures, specifically:

[0173] Expert Model 1 (Expert1)

[0174] Structure: Shallow Convolutional Neural Network

[0175] Composition: convolution layer, four 2D convolution layers, each convolution layer is followed by an activation function (such as ReLU); pooling layer, maximum pooling layer, used for dimensionality reduction and extraction of local features.

[0176] Purpose: Capture local spectral features of audio, especially local anomalies in forged audio.

[0177] How it works: The low-level features of the audio are extracted through convolution operations, and the size of the feature map is reduced through the pooling layer to retain important local information.

[0178] Expert2

[0179] Structure: Fusion of residual blocks and FMS attention modules

[0180] Composition: residual block, three residual blocks, each residual block contains two convolutional layers and a shortcut connection; FMS attention module, used to adjust the weight of the feature map and emphasize important features.

[0181] Purpose: Capture deep features of audio, especially global anomalies in forged audio.

[0182] Working principle: Learn deeper features through residual blocks, and emphasize important features and suppress irrelevant information through the attention mechanism (FMS module).

[0183] Expert Model 3 (Expert3)

[0184] Structure: Three-layer feedforward neural network

[0185] Composition: fully connected layer, three fully connected layers, each layer is followed by an activation function (such as ReLU).

[0186] Purpose: Capture high-level abstract features of audio.

[0187] Working principle: The input features are mapped to a high-dimensional space through a fully connected layer to extract more complex feature representations.

[0188] The fused features are input into each expert model. Each expert model processes the fused features and generates a specific output. For each expert model, its output is calculated.

[0189] The output of each expert model is Where i represents the number of the expert model.

[0190] In addition, the role of the gating network is to calculate the weights of each expert model output based on the input data. These weights represent the contribution of each expert model to the input data. The gating network uses a fully connected layer and a softmax function to model the probability distribution.

[0191] Input features: The gated network accepts fused features as input, which are obtained through multi-head attention mechanism or other feature fusion methods.

[0192] Calculate weights: The gating network uses the softmax function to calculate the weights of each expert model output. The softmax function normalizes the weights to a probability distribution, ensuring that the sum of all weights is 1. The specific formula is:

[0193]

[0194] Among them, a i represents the weight of the i-th expert model, W i Represents the weight matrix of the gating network, and input represents the fusion feature.

[0195] Through the above method, this embodiment can help the model comprehensively utilize the strengths of different expert models and improve the ability to distinguish the authenticity of audio; at the same time, by dynamically allocating weights through the gating network, the model can adaptively adjust the contribution of each expert model according to the input data, so that it can perform well on different types of audio.

[0196] Furthermore, a fourth embodiment of the training method of the speech authentication model of the present invention is proposed. In the above step S40, the step of training the expert model and the gating network according to the expert model output and the weight of the expert model output includes:

[0197] Step c1, inputting the expert model output and the weight of the expert model output into two fully connected layers to obtain a predicted label;

[0198] Step c2: training the expert model and the gating network using a binary cross entropy loss function according to the predicted label, and updating the parameters of the expert model and the gating network.

[0199] In this embodiment, the output of each expert model is weighted and combined with the weights calculated by the gating network to generate the final fused feature output. The fused feature output is input into the first fully connected layer for processing, and the intermediate feature representation after nonlinear transformation is output. The intermediate feature representation is input into the second fully connected layer, and the final predicted label is output. Through the nonlinear transformation of the fully connected layer, the feature representation is further optimized and the classification performance is improved; the cascade operation of the two fully connected layers can better capture the complex feature relationship and improve the accuracy of the predicted label.

[0200] The steps of model training and parameter updating may include:

[0201] Calculate the loss function: Use the binary cross entropy loss function to calculate the error between the predicted label and the actual label. The binary cross entropy loss function can effectively measure the difference between the predicted value and the actual value and accurately guide the optimization of the model parameters. The specific formula is:

[0202]

[0203] Among them, σ(x i ) is the predicted label, y i is the actual label, and N is the total number of samples.

[0204] Backpropagation: The gradient of the loss function is calculated using the backpropagation algorithm.

[0205] Parameter update: Use gradient descent or its variant optimization algorithm (such as Adam) to update the parameters of the expert model and the gating network. Through back propagation and gradient descent algorithms, the model parameters are continuously adjusted to continuously improve the classification performance of the model.

[0206] This embodiment obtains the prediction label by inputting the expert model output and weights into two fully connected layers, and uses the binary cross entropy loss function for training and parameter update, so as to optimize the feature representation and improve the classification ability of the model; accurately measure the prediction error and accurately guide the model optimization; improve the overall performance and adaptability of the model, so that it performs better in practical applications.

[0207] Furthermore, a fifth embodiment of the training method of the speech authentication model of the present invention is proposed. In the above step c1, the step of inputting the expert model output and the weight of the expert model output into two fully connected layers includes:

[0208] Step d1, performing weighted combination of the expert model output and the weight of the expert model output to generate a target feature output, and inputting the target feature output into the two fully connected layers.

[0209] In this embodiment, the step of generating target feature output may include:

[0210] The output of the expert model is weighted and combined according to the weights generated by the gating network to generate the final feature output. The specific formula is:

[0211]

[0212] Among them, final_output is the target feature output, represents the output of the i-th expert model, a i is its corresponding weight.

[0213] The steps of inputting the target feature output into two fully connected layers may include:

[0214] The first fully connected layer:

[0215] Structure: Contains the weight matrix W FC1 and bias b FC1 .

[0216] Formula: output FC1 =ReLU(W FC1 final_output+b FC1 );

[0217] Among them, final_output is the weighted combined feature, and ReLU is the activation function.

[0218] The second fully connected layer:

[0219] Structure: Contains the weight matrix W FC2 and bias b FC2 ;

[0220] Formula: output FC2 =sigmoid(W FC2 Output FC1 +b FC2 );

[0221] Among them, output FC1 is the output of the first fully connected layer, and sigmoid is the activation function.

[0222] This embodiment can better capture complex feature relationships and improve the classification ability of the model through the cascade operation of two fully connected layers; at the same time, through the nonlinear transformation of ReLU and sigmoid activation functions, the model can process complex nonlinear features and improve classification accuracy.

[0223] Furthermore, a sixth embodiment of the method for training a voice authentication model of the present invention is proposed. In the above step S10, after the steps of obtaining a first preset number of false audios and obtaining a second preset number of real audios, the method further includes:

[0224] Step e1: randomly adding noise to the false audio and the real audio according to the first preset number and the second preset number to perform a data enhancement operation.

[0225] In this embodiment, after the steps of obtaining a first preset amount of false audio and obtaining a second preset amount of real audio, the following steps may also be included:

[0226] Randomly add noise: According to the first preset number of fake audio and the second preset number of real audio, a certain proportion of audio samples are randomly selected for noise addition. The noise type can include environmental noise, white noise, background music, etc. to simulate different recording environments.

[0227] Noise adding method: For the selected audio samples, noise is mixed into the audio according to a certain signal-to-noise ratio (SNR), ensuring that the intensity of the noise is within a reasonable range and does not cover the main features of the original audio.

[0228] Example: For an audio sample x(t) and noise n(t), the mixed audio can be expressed as:

[0229] x noisy (t) = x(t) + α·n(t)

[0230] Among them, x noisy (t) represents the audio signal after noise is added; x(t) represents the original audio signal; n(t) represents the noise signal; α is the noise intensity coefficient, which is adjusted according to the required signal-to-noise ratio.

[0231] Data augmentation operation: All selected audio samples are processed through the above noise addition method to generate an augmented dataset. Ensure that the augmented dataset still maintains the balance of the original dataset, that is, the ratio of fake audio to real audio remains unchanged.

[0232] This embodiment enhances the diversity of the data set by randomly adding noise, so that the model can adapt to different noise environments and improve the robustness to noise interference; the data enhancement operation can increase the variability of the training data, effectively reduce the risk of overfitting of the model, and improve the generalization ability; by adding different types of noise, various situations in the real recording environment are simulated, so that the model has better performance in practical applications.

[0233] Furthermore, a seventh embodiment of the method for training a voice authentication model of the present invention is provided, which includes, after the above step S40, the following steps:

[0234] Step f1, obtaining the audio to be detected, inputting the audio to be detected into the voice authentication model, and obtaining the authenticity classification result of the audio to be detected.

[0235] In this embodiment, the voice audio may be detected by a trained voice authentication model to determine the authenticity. The specific steps may include:

[0236] Get the audio to be tested: Get the audio samples to be tested from the actual application scenario. These audios can come from various sources such as user recordings, phone recordings, voice assistants, etc. Perform necessary preprocessing operations on the audio to be tested, such as noise reduction, removal of silent segments, volume normalization, etc., to ensure that the audio quality meets the model input requirements.

[0237] Input the audio to be detected into the speech authentication model: Use the trained feature extraction module (such as Whisper, HuBERT, wav2vec2.0 pre-trained model) to extract the intermediate features, duration features and abstract features of the audio to be detected. Fuse the extracted features through the multi-head attention mechanism or other fusion methods to generate fused features. Input the fused features into the expert model and the gated network to obtain the output of the expert model and its weights. Perform a weighted combination of the expert model output and the weight to generate the target feature output, which is then input into two fully connected layers to obtain the final predicted label.

[0238] The output of the model (the result after the sigmoid activation function) is used to obtain the authenticity classification probability of the audio to be tested. According to the predicted probability, a threshold (such as 0.5) is set to determine the authenticity of the audio to be tested. Audio above the threshold is real audio, and audio below the threshold is fake audio.

[0239] This embodiment obtains the audio to be detected and inputs it into a trained voice authentication model, so as to efficiently and accurately classify the authenticity of the audio. It can be applied to a variety of practical application scenarios, such as voice authentication, telephone fraud prevention, voice assistant security, etc.

[0240] The invention also provides a training device for a voice authentication model.

[0241] Reference Figure 3 , Figure 3 The functional module diagram of the first embodiment of the training device for the speech authentication model of the present invention is shown in FIG. The training device for the speech authentication model comprises:

[0242] An audio acquisition module, acquiring a first preset amount of false audio and a second preset amount of real audio;

[0243] A feature extraction module, which extracts intermediate features, duration features, and abstract features of the fake audio and the real audio respectively through a whisper pre-training model, a huber pre-training model, and a wav2vec2.0 pre-training model;

[0244] A feature fusion module, which fuses the intermediate features, duration features and abstract features to generate fused features, and inputs the fused features into an expert model and a gating network to obtain an output of the expert model and a weight of the output of the expert model;

[0245] The model training module trains the expert model and the gated network according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

[0246] In addition, an embodiment of the present invention further provides a storage medium.

[0247] The storage medium stores a training program for the speech authentication model. When the training program for the speech authentication model is executed by the processor, the steps of the training method for the speech authentication model as described above are implemented.

[0248] The specific implementation of the storage medium of the present invention is basically the same as the various embodiments of the training method of the above-mentioned speech authentication model, and will not be repeated here.

[0249] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

[0250] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0251] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk.

[0252] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0253] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training a speech authentication model, characterized in that: The training method of the voice authentication model comprises the following steps: Acquire a first preset number of false audios, and acquire a second preset number of real audios; The intermediate features, duration features and abstract features of the fake audio and the real audio are respectively extracted through the whisper pre-training model, the hubert pre-training model and the wav2vec2.0 pre-training model; The intermediate features, duration features and abstract features are fused to generate fused features, and the fused features are input into the expert model and the gating network to obtain the output of the expert model and the weight of the output of the expert model; The expert model and the gating network are trained according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

2. The method for training a voice authentication model as claimed in claim 1, wherein: The step of fusing the intermediate features, the duration features and the abstract features to generate fused features comprises: The intermediate features, duration features and abstract features are fused using a multi-head attention mechanism. The intermediate features are used as Key (K) and Value (V), and the duration features and abstract features are used as Query (Q) for self-attention operation. The calculation formula is: Among them, Q stands for Query, K stands for Key, V stands for Value, and d k Indicates the dimension of the Key; The fusion feature is generated according to the Attention output.

3. The method for training a voice authentication model as claimed in claim 1, wherein: The step of inputting the fusion feature into the expert model and the gating network to obtain the expert model output and the weight of the expert model output comprises: The expert model includes three expert networks with different structures, and the fusion features are respectively input into the three expert networks to obtain the expert model output; The gating network uses a softmax function to model the probability distribution, and the fusion feature is input into the gating network to obtain the weight output by the expert model.

4. The method for training a voice authentication model as claimed in claim 1, wherein: The step of training the expert model and the gating network according to the expert model output and the weight of the expert model output comprises: Inputting the expert model output and the weight of the expert model output into two fully connected layers to obtain a predicted label; The expert model and the gating network are trained according to the predicted label using a binary cross entropy loss function, and the parameters of the expert model and the gating network are updated.

5. The method for training a voice authentication model as claimed in claim 4, wherein: The step of inputting the expert model output and the weight of the expert model output into two fully connected layers comprises: The expert model output is weightedly combined with the weight of the expert model output to generate a target feature output, and the target feature output is input into the two fully connected layers.

6. The method for training a voice authentication model as claimed in claim 1, wherein: After the steps of obtaining a first preset number of false audios and obtaining a second preset number of real audios, the method further includes: Noise is randomly added to the false audio and the real audio according to the first preset number and the second preset number to perform a data enhancement operation.

7. The method for training a voice authentication model as claimed in claim 1, wherein: After the step of obtaining the voice authentication model, the method further includes: The audio to be detected is obtained, and the audio to be detected is input into the voice authentication model to obtain a authenticity classification result of the audio to be detected.

8. A training device for a speech authentication model, characterized in that: The training device of the speech authentication model comprises: An audio acquisition module, acquiring a first preset amount of false audio and a second preset amount of real audio; A feature extraction module, which extracts intermediate features, duration features, and abstract features of the fake audio and the real audio respectively through a whisper pre-training model, a huber pre-training model, and a wav2vec2.0 pre-training model; A feature fusion module, which fuses the intermediate features, duration features and abstract features to generate fused features, and inputs the fused features into an expert model and a gating network to obtain an output of the expert model and a weight of the output of the expert model; The model training module trains the expert model and the gated network according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

9. A training device for a speech authentication model, characterized in that: The training device for the speech authentication model includes a memory, a processor, and a training program for the speech authentication model stored in the memory and executable on the processor. When the training program for the speech authentication model is executed by the processor, the steps of the training method for the speech authentication model as described in any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores a training program for a speech authentication model. When the training program for the speech authentication model is executed by the processor, the steps of the training method for a speech authentication model according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Forged voice detection method based on multi-feature fusion and device thereof

    CN113488073A

  • Pseudo voice detection method and device, computer equipment and storage medium

    CN115273814A

  • Voice authentic identification method and device based on self-supervised learning, and storage medium

    CN115662441A

  • Self-supervised voice authentic identification training method and system based on multi-center single classification

    CN117612562A

  • Voice spoofing detection method based on feature-enhanced attention mechanism

    CN118298832A

Cited By

  • Synthetic speech detection method and device, storage medium and electronic equipment

    CN119517008A

  • Parkinson's disease auxiliary detection method, device, equipment and medium

    CN120766728A