Voice authentication model training method and device, equipment and storage medium

Abstract

The present application relates to a kind of training methods of speech authentication model, by obtaining the first preset quantity of false audio, obtains the second preset quantity of real audio.By whisper pre-training model, hubert pre-training model and wav2vec2.0 pre-training model respectively extract the intermediate feature, duration feature and abstract feature of false audio and real audio.The intermediate feature, duration feature and abstract feature are fused to generate fusion features, which are input into expert model and gated network to obtain expert model output and expert model output weight.Expert model and gated network are trained according to expert model output and expert model output weight to obtain speech authentication model.The present application combines various pre-training models to extract multi-level features, and the features are fused through Attention mechanism, which can enhance the adaptability of the model to different types of fake speech, thereby improving the generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a speech authentication model training method and device, equipment and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, text-to-speech (TTS) and voice conversion (VC) technologies have also made significant progress. These technologies have been widely applied in voice assistants, intelligent customer service, barrier-free communication, entertainment and education, and many other fields. Modern TTS and VC technologies can generate high-quality, highly natural speech, which is of great significance in improving user experience and application scenarios. However, the progress of these technologies has also brought new challenges and risks, especially in the field of speech authentication.

[0003] Traditional speech authentication methods mainly rely on hand-designed features such as Mel-frequency cepstral coefficients (MFCC) and constant Q transform cepstral coefficients (CQCC). Although these features can capture some speech information, their expressive ability is limited and they are difficult to deal with complex counterfeit speech. Hand-designed features often cannot fully reflect the deep information of speech, resulting in poor performance when facing advanced counterfeit technology.

[0004] Neural network-based speech authentication methods use deep learning technology to automatically learn features from a large amount of data. This type of method performs well in specific scenarios, but its generalization ability is poor and it often fails to effectively detect unseen counterfeit speech. In addition, neural network models usually require a large amount of labeled data for training, which is costly in terms of data acquisition and time-consuming in terms of training, and resource consumption is large.

[0005] With the development of generative adversarial networks (GAN) and other advanced generative models, the quality and naturalness of counterfeit speech are constantly improving, making the speech authentication task more difficult. Modern counterfeit speech technology can generate audio that is very similar to real speech, which not only poses a potential threat to users, but also poses a great challenge to financial security, identity authentication, and public safety. SUMMARY

[0006] The main purpose of the present application is to provide a speech authentication model training method, device, equipment and storage medium, which aims to solve the technical problems of weak generalization ability and low detection accuracy of existing speech authentication technology.

[0007] To achieve the above-mentioned purpose, the present application provides a speech authentication model training method, which comprises the following steps:

[0008] obtaining a first preset number of false audios and a second preset number of real audios;

[0009] extracting intermediate features, duration features and abstract features of the false audios and the real audios through whisper pre-training models, hubert pre-training models and wav2vec2.0 pre-training models respectively;

[0010] fusing the intermediate features, the duration features and the abstract features to generate fusion features, inputting the fusion features into an expert model and a gating network to obtain an expert model output and a weight of the expert model output;

[0011] training the expert model and the gating network according to the expert model output and the weight of the expert model output to obtain a voice authentication model.

[0012] Optionally, the step of fusing the intermediate features, the duration features and the abstract features to generate fusion features comprises:

[0013] fusing the intermediate features, the duration features and the abstract features by using a multi-head attention mechanism, taking the intermediate features as Key (K) and Value (V), and taking the duration features and the abstract features as Query (Q) to perform self-attention operation, and the calculation formula is:

[0014]

[0015] wherein, Q represents Query, K represents Key, V represents Value, d k represents the dimension of Key;

[0016] generating the fusion features according to the Attention output.

[0017] Optionally, the step of inputting the fusion features into an expert model and a gating network to obtain an expert model output and a weight of the expert model output comprises:

[0018] the expert model comprises three expert networks with different structures, the fusion features are input into the three expert networks to obtain the expert model output;

[0019] the gating network uses a softmax function to model a probability distribution, the fusion features are input into the gating network to obtain the weight of the expert model output.

[0020] Optionally, the step of training the expert model and the gating network according to the expert model output and the weight of the expert model output comprises:

[0021] input the expert model output and a weight of the expert model output into two fully connected layers to obtain a predicted label;

[0022] train the expert model and the gating network according to the predicted label by using a binary cross-entropy loss function, and update parameters of the expert model and the gating network.

[0023] Optionally, the step of inputting the expert model output and a weight of the expert model output into two fully connected layers comprises:

[0024] combining the expert model output and the weight of the expert model output by weighting, to generate a target feature output, and inputting the target feature output into the two fully connected layers.

[0025] Optionally, after the steps of obtaining a first preset number of fake audios and a second preset number of real audios, the method further comprises:

[0026] adding noise to the fake audios and the real audios according to the first preset number and the second preset number, to perform a data enhancement operation.

[0027] Optionally, after the step of obtaining the speech anti-fake model, the method further comprises:

[0028] obtaining a to-be-detected audio, inputting the to-be-detected audio into the speech anti-fake model, and obtaining a true or false classification result of the to-be-detected audio.

[0029] Further, to achieve the above object, the application further provides a training device of a speech anti-fake model, which comprises:

[0030] an audio acquisition module, configured to obtain a first preset number of fake audios and a second preset number of real audios;

[0031] a feature extraction module, configured to extract intermediate features, duration features and abstract features of the fake audios and the real audios by using a whisper pre-training model, a hubert pre-training model and a wav2vec2.0 pre-training model respectively;

[0032] a feature fusion module, configured to fuse the intermediate features, the duration features and the abstract features to generate fusion features, and input the fusion features into an expert model and a gating network to obtain an expert model output and a weight of the expert model output;

[0033] a model training module, configured to train the expert model and the gating network according to the expert model output and the weight of the expert model output, to obtain a speech anti-fake model.

[0034] Further, to achieve the above object, the present application provides a speech authentication model training device, which comprises a memory, a processor, and a speech authentication model training program stored in the memory and executable on the processor, and the speech authentication model training program implements the steps of the speech authentication model training method when executed by the processor.

[0035] Further, to achieve the above object, the present application provides a storage medium, which stores a speech authentication model training program, and the speech authentication model training program implements the steps of the speech authentication model training method when executed by a processor.

[0036] The present application relates to a speech authentication model training method, which comprises obtaining a first preset number of false audios and a second preset number of real audios. Then, intermediate features, duration features and abstract features of the false audios and the real audios are extracted by using whisper pre-training model, hubert pre-training model and wav2vec2.0 pre-training model respectively. The intermediate features, the duration features and the abstract features are fused to generate fusion features, and the fusion features are input into an expert model and a gating network to obtain expert model output and a weight of the expert model output. The expert model and the gating network are trained according to the expert model output and the weight of the expert model output, and a speech authentication model is obtained. The present application combines multiple pre-training models to extract multi-level features, and fuses the features through an Attention mechanism, which can enhance the adaptability of the model to different types of fake speech and improve the generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The structure diagram of the hardware running environment of the device related to the speech authentication model training device embodiment of the present application is shown in the figure.

[0038] Figure 2 The flowchart of the first embodiment of the speech authentication model training method of the present application is shown in the figure.

[0039] Figure 3 The functional module diagram of the preferred embodiment of the speech authentication model training device of the present application is shown in the figure.

[0040] The implementation, functional characteristics and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0042] It should be noted that with the rapid development of artificial intelligence technology, text-to-speech (TTS) and voice conversion (VC) technologies have also made significant progress. These technologies have been widely used in voice assistants, intelligent customer service, barrier-free communication, entertainment and education, and many other fields. Modern TTS and VC technologies can generate high-quality, highly natural speech, which is of great significance in improving user experience and application scenarios. However, the progress of these technologies has also brought new challenges and risks, especially in the field of voice authentication.

[0043] Traditional voice authentication methods mainly rely on hand-designed features such as Mel-frequency cepstral coefficients (MFCC) and constant Q transform cepstral coefficients (CQCC). Although these features can capture some speech information, their expressive ability is limited and they are difficult to deal with complex counterfeit speech. Hand-designed features usually cannot fully reflect the deep information of speech, resulting in poor performance when facing advanced counterfeit technology.

[0044] Neural network-based voice authentication methods use deep learning technology to automatically learn features from a large amount of data. This kind of method shows high accuracy in specific scenarios, but its generalization ability is poor, and it is often difficult to effectively detect unobserved counterfeit speech. In addition, neural network models usually need a large amount of labeled data for training, which has high data acquisition cost, long training time, and large resource consumption.

[0045] With the development of generative adversarial networks (GAN) and other advanced generative models, the quality and naturalness of counterfeit speech are constantly improving, making the voice authentication task more difficult. Modern counterfeit speech technology can generate audio that is very similar to real speech, which not only poses a potential threat to users, but also brings great challenges to financial security, identity authentication, and public safety.

[0046] Based on the above defects, the present application provides a voice authentication model training device, referring to Figure 1 , Figure 1 The structural diagram of the hardware running environment of the device for the embodiment scheme of the voice authentication model training device of the present application.

[0047] As Figure 1As shown, the training device of the voice authentication model can include a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection communication between these components. The user interface 1003 can include a display screen (Display), an input unit such as a keyboard (Keyboard), and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory) such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art can understand that Figure 1 The hardware structure of the training device of the voice authentication model shown in the foregoing embodiments is not a limitation on the training device of the voice authentication model, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0049] As Figure 1 As shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a voice authentication model training program. The operating system is a program that manages and controls the software resources of the voice authentication model training device, supports the running of the network communication module, the user interface module, the voice authentication model training program, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0050] In Figure 1 In the hardware structure of the voice authentication model training device shown in the foregoing embodiments, the network interface 1004 is mainly used to connect a background server and communicate data with the background server; the user interface 1003 is mainly used to connect a client and communicate data with the client; and the processor 1001 can call the voice authentication model training program stored in the memory 1005 and perform the following operations:

[0051] Obtain a first preset number of false audios and a second preset number of real audios;

[0052] Extract intermediate features, duration features, and abstract features of the false audios and the real audios through whisper pre-training models, hubert pre-training models, and wav2vec2.0 pre-training models, respectively;

[0053] fusing the intermediate feature, the duration feature and the abstract feature to generate a fusion feature, inputting the fusion feature into an expert model and a gating network to obtain an expert model output and a weight of the expert model output;

[0054] training the expert model and the gating network according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

[0055] Further, the step of fusing the intermediate feature, the duration feature and the abstract feature to generate a fusion feature comprises:

[0056] fusing the intermediate feature, the duration feature and the abstract feature by using a multi-head attention mechanism, taking the intermediate feature as Key (K) and Value (V), and taking the duration feature and the abstract feature as Query (Q) to perform self-attention operation, and the calculation formula is:

[0057]

[0058] wherein Q represents Query, K represents Key, V represents Value, d k represents the dimension of Key;

[0059] generating the fusion feature according to the Attention output.

[0060] Further, the step of inputting the fusion feature into an expert model and a gating network to obtain an expert model output and a weight of the expert model output comprises:

[0061] The expert model comprises three expert networks with different structures, and the fusion feature is inputted into the three expert networks to obtain the expert model output.

[0062] The gating network uses a softmax function to model a probability distribution, and the fusion feature is inputted into the gating network to obtain the weight of the expert model output.

[0063] Further, the step of training the expert model and the gating network according to the expert model output and the weight of the expert model output comprises:

[0064] inputting the expert model output and the weight of the expert model output into two fully connected layers to obtain a predicted label;

[0065] training the expert model and the gating network according to the predicted label by using a binary cross-entropy loss function to update the parameters of the expert model and the gating network.

[0066] Further, the step of inputting the expert model output and the weight of the expert model output into two fully connected layers comprises:

[0067] The expert model output and the weight of the expert model output are weighted and combined to generate a target feature output, and the target feature output is input into the two fully connected layers.

[0068] Further, after the steps of obtaining a first preset number of false audios and obtaining a second preset number of real audios, the method further comprises:

[0069] According to the first preset number and the second preset number, noise is added to the false audios and the real audios to perform a data enhancement operation.

[0070] Further, after the step of obtaining the speech authentication model, the method further comprises:

[0071] Obtaining a to-be-detected audio, inputting the to-be-detected audio into the speech authentication model, and obtaining a true or false classification result of the to-be-detected audio.

[0072] The specific implementation of the speech authentication model training device of the present application is basically the same as that of each embodiment of the speech authentication model training method described below, and will not be repeated here.

[0073] The present application also provides a speech authentication model training method based on the speech authentication model training device described above.

[0074] Reference Figure 2 , Figure 2 The flowchart of the first embodiment of the speech authentication model training method of the present application.

[0075] The embodiments of the speech authentication model training method provided by the present application need to be explained, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0076] In each embodiment of the speech authentication model training method, the execution subject is a speech authentication model training device.

[0077] The speech authentication model training method is applied to a speech authentication model training device, and the speech authentication model training method comprises:

[0078] Step S10, obtaining a first preset number of false audios and a second preset number of real audios;

[0079] With the rapid development of artificial intelligence technology, text-to-speech (TTS) and voice conversion (VC) technologies have also made significant progress. These technologies have been widely applied in voice assistants, intelligent customer service, barrier-free communication, entertainment, education and other fields. Modern TTS and VC technologies can generate high-quality, highly natural speech, which is of great significance in improving user experience and application scenarios. However, the progress of these technologies has also brought new challenges and risks, especially in the field of voice authentication.

[0080] Traditional voice authentication methods mainly rely on manually designed features such as Mel-frequency cepstral coefficients (MFCC) and constant Q transform cepstral coefficients (CQCC). Although these features can capture some speech information, their expressive ability is limited and they are difficult to deal with complex counterfeit speech. Manually designed features usually cannot fully reflect the deep information of speech, resulting in poor performance when facing advanced counterfeit technology.

[0081] Neural network-based voice authentication methods use deep learning technology to automatically learn features from a large amount of data. This kind of method shows high accuracy in specific scenarios, but its generalization ability is poor, and it is often difficult to effectively detect unobserved counterfeit speech. In addition, neural network models usually need a large amount of labeled data for training, which has high data acquisition cost, long training time and large resource consumption.

[0082] With the development of generative adversarial networks (GAN) and other advanced generative models, the quality and naturalness of counterfeit speech are constantly improving, making the voice authentication task more difficult. Modern counterfeit speech technology can generate audio that is very similar to real speech, which not only poses a potential threat to users, but also poses a great challenge to financial security, identity authentication and public safety.

[0083] The training method of the voice authentication model in this embodiment is applicable to devices with the function of training method of voice authentication model.

[0084] In this embodiment, first, data preparation is completed, and false audio and real audio are obtained. The number of this embodiment is not specifically limited.

[0085] The audio data can be obtained from a public dataset to obtain a first preset number of false audio samples. These samples can come from different types of speech generation technology, such as text-to-speech (TTS) systems, voice conversion (VC) systems, and generative adversarial networks (GAN) generated speech. The first preset number can be adjusted according to actual needs, for example, 100,000 false audio samples can be selected as a reference.

[0086] Obtain a second preset number of real audio samples, which should include audio from different speakers, different contexts, and different recording environments. The second preset number can be adjusted according to actual needs, for example, 10,000 real audio samples can be selected as a reference to ensure that the model can learn enough real speech features.

[0087] To enhance the robustness and generalization ability of the model, data augmentation can be performed on the fake audio and real audio. The specific methods include:

[0088] Add background noise: Randomly add different types of background noise, such as environmental noise, microphone noise, etc., to simulate various noise conditions in reality.

[0089] Change the speed of the audio: speed up or slow down the audio to simulate different speaking speeds.

[0090] Change the pitch: Change the pitch of the audio to simulate different sound frequencies.

[0091] Cut and splice: randomly cut and splice the audio to generate new audio segments.

[0092] After data augmentation, the number of fake audio and real audio should maintain a certain ratio to ensure the balance of the data set. For example, if 100,000 fake audio and 10,000 real audio are selected, the ratio of random noise and other augmentation processing can be 1:11.

[0093] Step S20, extract the intermediate features, duration features and abstract features of the fake audio and the real audio through the whisper pre-training model, the hubert pre-training model and the wav2vec2.0 pre-training model respectively;

[0094] In this embodiment, feature extraction is performed, and the specific method can include:

[0095] whisper pre-training model: load the whisper pre-training model, input the fake audio and the real audio, and extract the intermediate features. These features capture the local spectral features of the audio, which helps to distinguish between real audio and fake audio.

[0096] hubert pre-training model: load the hubert pre-training model, input the fake audio and the real audio, and extract the duration features. These features represent the time structure information of the audio, which helps to detect the abnormalities of the fake audio in the time dimension.

[0097] wav2vec2.0 pre-trained model: load the wav2vec2.0 pre-trained model, input the fake audio and real audio, and extract abstract features. These features contain high-level semantic information of the audio, which helps to capture deeper fake features.

[0098] Step S30, the intermediate features, duration features and abstract features are fused to generate fused features, and the fused features are input into the expert model and the gating network to obtain the expert model output and the weight of the expert model output;

[0099] In this embodiment, the extracted intermediate features (whisper features), duration features (hubert features) and abstract features (wav2vec features) need to be fused.

[0100] One implementation is to use a multi-head attention mechanism to fuse the extracted intermediate features (whisper features), duration features (hubert features) and abstract features (wav2vec features). The specific steps can include:

[0101] Take the intermediate features as Key and Value, and take the duration features and abstract features as Query to perform self-attention operation, and the calculation formula is:

[0102]

[0103] Concatenate the results of multi-head and project them into another vector to generate the final fused feature input.

[0104] Another implementation is to perform feature concatenation and weighted averaging on the intermediate features (whisper features), duration features (hubert features) and abstract features (wav2vec features). The specific steps can include:

[0105] Feature concatenation: directly concatenate the whisper features, hubert features and wav2vec features to form a larger feature vector. The concatenated feature vector contains complete information of all three features.

[0106] Weighted averaging: for each feature, give different weights according to its importance and perform weighted averaging. The specific formula is: fused_feature = α·whisper_feature + β·hubert_feature + γ·wav2vec_feature, where α, β and γ are weight coefficients, and α + β + γ = 1. These weights can be learned through training data.

[0107] After obtaining the fusion features, input the fusion features into the expert model and the gating network to obtain the expert model output and the weight of the expert model output. The specific steps can include:

[0108] Expert model: The expert model can include multiple expert networks with different structures, and each expert model is responsible for further processing and extracting fusion features from different angles. For example, the expert model includes three expert networks with different structures, specifically:

[0109] Expert model 1 (Expert1)

[0110] Structure: shallow convolutional neural network

[0111] Composition: convolutional layer, four 2D convolutional layers, each followed by an activation function (such as ReLU); pooling layer, max pooling layer, used for dimension reduction and local feature extraction.

[0112] Function: Capture local spectral features of audio, especially local anomalies in fake audio.

[0113] Working principle: Extract low-level features of audio through convolution operation, and reduce the size of feature map through pooling layer to retain important local information.

[0114] Expert model 2 (Expert2)

[0115] Structure: combines residual blocks and FMS attention modules

[0116] Composition: residual block, three residual blocks, each containing two convolutional layers and a shortcut connection; FMS attention module, used to adjust the weight of the feature map and emphasize important features.

[0117] Function: Capture deep features of audio, especially global anomalies in fake audio.

[0118] Working principle: Learn deeper features through residual blocks, while emphasizing important features and suppressing irrelevant information through attention mechanism (FMS module).

[0119] Expert model 3 (Expert3)

[0120] Structure: three-layer feedforward neural network

[0121] Composition: fully connected layer, three fully connected layers, each followed by an activation function (such as ReLU).

[0122] Function: Capture high-level abstract features of audio.

[0123] Working principle: The input features are mapped to a high-dimensional space through a fully connected layer, extracting more complex feature representations.

[0124] In addition, the role of the gating network is to calculate the weights of each expert model output based on the input data, which represent the contribution of each expert model to the input data.

[0125] Input features: The gating network accepts fusion features as input, which are obtained through multi-head attention mechanism or other feature fusion methods.

[0126] Calculate weights: The gating network uses the softmax function to calculate the weights of each expert model output. The softmax function normalizes the weights into a probability distribution, ensuring that the sum of all weights is 1. The specific formula is:

[0127]

[0128] where a i represents the weight of the i-th expert model, W i represents the weight matrix of the gating network, and input represents the fusion features.

[0129] Step S40, according to the expert model output and the weight of the expert model output, the expert model and the gating network are trained to obtain a voice authentication model.

[0130] In this embodiment, the specific steps of training the expert model and the gating network according to the expert model output and the weight can include:

[0131] Weighted combination: The output of the expert model is weighted and combined according to the weight generated by the gating network to generate the final feature output. The specific formula is:

[0132]

[0133] where, represents the output of the i-th expert model, a i is its corresponding weight.

[0134] Finally, the model is trained through the final feature output, and the specific steps can include:

[0135] Input fusion features: The final feature output is input to two fully connected layers.

[0136] First fully connected layer:

[0137] Structure: Contains weight matrix W FC1 and bias b FC1 .

[0138] Formula: outputFC1 =ReLU(W FC1 ·final_output+b FC1 );

[0139] Here, final_output is the weighted combined feature, and ReLU is the activation function.

[0140] The second fully connected layer:

[0141] Structure: Includes a weight matrix W FC2 and bias b FC2 ;

[0142] Formula: output FC2 =sigmoid(W FC2 •output FC1 +b FC2 );

[0143] Where, output FC1 It is the output of the first fully connected layer, and sigmoid is the activation function.

[0144] Choosing a loss function: The binary cross-entropy loss function (BCEWithLogitLoss) can be used as the loss function for training the model.

[0145] Loss function formula:

[0146]

[0147] in, It is the sigmoid function, where N is the total number of samples, yi is the actual label, and x is the number of samples. i The predicted value is the output of the second fully connected layer. FC2 .

[0148] Backpropagation algorithm: The backpropagation algorithm is used to calculate the gradient of the loss function.

[0149] Gradient calculation: Calculate the gradient of each parameter (including the weights and biases of the expert model, gated network, and fully connected layers).

[0150] Parameter update: Use gradient descent or its variants (such as the Adam optimizer) to update the parameters.

[0151] Optimization process: The update formula for each parameter θ is:

[0152]

[0153] The trained speech authentication model includes the following components:

[0154] Whisper pre-training model, HuBERT pre-training model, wav2vec2.0 pre-training model, expert model, gating network, full connection layer.

[0155] The present application obtains a first preset number of false audios and a second preset number of real audios. Then, the intermediate features, the duration features and the abstract features of the false audios and the real audios are extracted by the whisper pre-training model, the hubert pre-training model and the wav2vec2.0 pre-training model respectively. The intermediate features, the duration features and the abstract features are fused to generate fusion features. The fusion features are input into the expert model and the gating network to obtain the expert model output and the weight of the expert model output. The expert model and the gating network are trained according to the expert model output and the weight of the expert model output, and a voice identification model is obtained. The present application combines multiple pre-training models to extract multi-level features, and performs feature fusion through the Attention mechanism, which can enhance the adaptability of the model to different types of fake voices and improve the generalization ability.

[0156] Further, the second embodiment of the training method of the voice identification model is proposed. In step S30, the step of fusing the intermediate features, the duration features and the abstract features to generate fusion features includes:

[0157] Step a1, the intermediate features, the duration features and the abstract features are fused by using the multi-head attention mechanism, the intermediate features are taken as Key (K) and Value (V), the duration features and the abstract features are taken as Query (Q) to perform self-attention operation, and the calculation formula is:

[0158]

[0159] Wherein, Q represents Query, K represents Key, V represents Value, d k represents the dimension of Key;

[0160] Step a2, the fusion features are generated according to the Attention output.

[0161] In this embodiment, the intermediate features, the duration features and the abstract features can be fused by using the multi-head attention mechanism. Through the multi-head attention mechanism, the information of different features is integrated to improve the processing ability of the model to complex data. The specific steps can include:

[0162] The intermediate features are taken as Key (K) and Value (V), and the duration features and the abstract features are taken as Query (Q) to perform self-attention operation.

[0163] Intermediate features (Key and Value): from the Whisper pre-trained model, capturing the local spectral features of the audio.

[0164] Duration and abstract features (Query): from the HuBERT pre-trained model and the wav2vec2.0 pre-trained model, respectively, representing the temporal structure and high-level semantic information of the audio.

[0165] The calculation formula is:

[0166]

[0167] The fusion features are generated according to the Attention output: calculate the dot product of Query and Key, then divide by the square root of the Key dimension to get a scaled attention score matrix. Apply the softmax function to the attention score matrix to get a weight matrix. Multiply the weight matrix with the Value to get the weighted sum output, which is the fusion feature.

[0168] Through the above steps, the embodiment can simultaneously focus on different parts of the input, capture multi-scale feature information, improve the effect of authentication, and fuse features of different pre-trained models to generate more rich and comprehensive feature representations, enhancing the stability of the model under different types of fake audio.

[0169] Further, the third embodiment of the training method of the voice authentication model is proposed, and the step of inputting the fusion features into the expert model and the gating network in step S30 to obtain the expert model output and the weight of the expert model output comprises:

[0170] Step b1, the expert model comprises three expert networks with different structures, and the fusion features are input into the three expert networks to obtain the expert model output;

[0171] Step b2, the gating network uses a softmax function to model the probability distribution, and the fusion features are input into the gating network to obtain the weight of the expert model output.

[0172] In this embodiment, the expert model comprises three expert networks with different structures, specifically:

[0173] Expert model 1 (Expert1)

[0174] Structure: shallow convolutional neural network

[0175] Composition: convolutional layer, four 2D convolutional layers, each followed by an activation function (such as ReLU); pooling layer, max pooling layer, used for dimension reduction and local feature extraction.

[0176] Function: Capture local spectral features of audio, especially local anomalies in fake audio.

[0177] Working Principle: Extract low-level features of audio through convolution operation, then reduce the size of feature map through pooling layer to preserve important local information.

[0178] Expert Model 2 (Expert2)

[0179] Structure: Combines residual blocks and FMS attention modules

[0180] Composition: Residual blocks, three residual blocks, each containing two convolution layers and a shortcut connection; FMS attention modules, used to adjust the weights of feature maps, emphasizing important features.

[0181] Function: Capture deep-level features of audio, especially global anomalies in fake audio.

[0182] Working Principle: Learn deeper features through residual blocks, while emphasizing important features and suppressing irrelevant information through attention mechanisms (FMS modules).

[0183] Expert Model 3 (Expert3)

[0184] Structure: Three-layer feedforward neural network

[0185] Composition: Fully connected layers, three fully connected layers, each followed by an activation function (such as ReLU).

[0186] Function: Capture high-level abstract features of audio.

[0187] Working Principle: Map input features to high-dimensional space through fully connected layers to extract more complex feature representations.

[0188] The fusion features are input into each expert model. Each expert model processes the fusion features and generates a specific output. For each expert model, calculate its output.

[0189] Let the output of each expert model be where i represents the number of expert models.

[0190] In addition, the role of the gating network is to calculate the weights of each expert model output according to the input data, which represents the contribution of each expert model to the input data. The gating network uses a fully connected layer and a softmax function to model the probability distribution.

[0191] Input features: The gating network accepts fusion features as input, which are obtained through multi-head attention mechanisms or other feature fusion methods.

[0192] Calculate weights: The gating network uses the softmax function to calculate the weight of each expert model output. The softmax function normalizes the weights into a probability distribution, ensuring that the sum of all weights is 1. The specific formula is:

[0193]

[0194] where a i represents the weight of the i-th expert model, W i represents the weight matrix of the gating network, and input represents the fusion feature.

[0195] The above method can help the model to comprehensively utilize the strengths of different expert models and improve the discrimination ability of audio authenticity. At the same time, through the dynamic allocation of weights by the gating network, the model can adaptively adjust the contribution of each expert model according to the input data, so as to perform well on different types of audio.

[0196] Further, the fourth embodiment of the training method of the speech authentication model is proposed. In the step S40, the step of training the expert model and the gating network according to the expert model output and the weight of the expert model output includes:

[0197] Step c1, inputting the expert model output and the weight of the expert model output into two fully connected layers to obtain a predicted label;

[0198] Step c2, training the expert model and the gating network according to the predicted label using a binary cross-entropy loss function, and updating the parameters of the expert model and the gating network.

[0199] In this embodiment, the output of each expert model and the weight calculated by the gating network are weighted and combined to generate the final fusion feature output. The fusion feature output is input into the first fully connected layer for processing, and the intermediate feature representation after nonlinear transformation is output. The intermediate feature representation is input into the second fully connected layer, and the final predicted label is output. Through the nonlinear transformation of the fully connected layer, the feature representation is further optimized, and the classification performance is improved; the cascade operation of the two fully connected layers can better capture the complex feature relationship and improve the accuracy of the predicted label.

[0200] The step of training and updating the parameters of the model can include:

[0201] Calculate the loss function: use the binary cross-entropy loss function to calculate the error between the predicted label and the actual label. The binary cross-entropy loss function can effectively measure the difference between the predicted value and the actual value, and accurately guide the optimization of the model parameters. The specific formula is:

[0202]

[0203] where σ(x i ) is the predicted label, y i is the actual label, and N is the total number of samples.

[0204] Backpropagation: Calculate the gradient of the loss function using the backpropagation algorithm.

[0205] Parameter update: Update the parameters of the expert model and the gating network using gradient descent or its variants such as Adam. Through backpropagation and gradient descent algorithm, the model parameters are constantly adjusted to improve the classification performance of the model.

[0206] This embodiment can optimize feature representation, improve the classification ability of the model, accurately measure the prediction error, accurately guide the model optimization, improve the overall performance and adaptability of the model, and make it perform better in practical applications by obtaining the predicted label by inputting the expert model output and the weight into two fully connected layers and training and updating the parameters using the binary cross-entropy loss function.

[0207] Further, the fifth embodiment of the training method of the voice authentication model is proposed, and the step c1 above includes:

[0208] Step d1: Weighted combination of the expert model output and the weight of the expert model output to generate a target feature output, and inputting the target feature output into the two fully connected layers.

[0209] In this embodiment, the step of generating a target feature output can include:

[0210] The output of the expert model is weighted and combined according to the weight generated by the gating network to generate the final feature output. The specific formula is:

[0211]

[0212] where final_output is the target feature output, represents the output of the i-th expert model, and a i is its corresponding weight.

[0213] The step of inputting the target feature output into the two fully connected layers can include:

[0214] The first fully connected layer:

[0215] Structure: Contains a weight matrix W FC1 and a bias b FC1 .

[0216] Formula: output FC1 = ReLU(W FC1 · final_output + b FC1 );

[0217] Where final_output is the weighted combined features, and ReLU is the activation function.

[0218] Second fully connected layer:

[0219] Structure: contains weight matrix W FC2 and bias b FC2 ;

[0220] Formula: output FC2 = sigmoid(W FC2 · output FC1 + b FC2 );

[0221] Where output FC1 is the output of the first fully connected layer, and sigmoid is the activation function.

[0222] This embodiment can better capture complex feature relationships and improve the classification ability of the model through the cascading operation of the two fully connected layers. At the same time, through the nonlinear transformation of the ReLU and sigmoid activation functions, the model can process complex nonlinear features and improve the classification accuracy.

[0223] Further, the sixth embodiment of the training method of the voice authentication model is proposed. After the steps of obtaining a first preset number of false audios and a second preset number of real audios in the above step S10, the method further comprises:

[0224] Step e1, according to the first preset number and the second preset number, randomly adding noise to the false audio and the real audio, and performing a data enhancement operation.

[0225] In this embodiment, after the steps of obtaining a first preset number of false audios and a second preset number of real audios, the following steps can be further included:

[0226] Randomly adding noise: according to the first preset number of false audios and the second preset number of real audios, randomly selecting a certain proportion of audio samples to perform noise adding operation. The noise types can include environmental noise, white noise, background music, etc. to simulate different recording environments.

[0227] Noise adding method: For selected audio samples, mix noise into the audio at a certain signal-to-noise ratio (SNR), ensuring that the intensity of the noise is within a reasonable range and does not cover the main features of the original audio.

[0228] Example: For an audio sample x(t) and noise n(t), the mixed audio can be represented as:

[0229] x noisy (t)=x(t)+α·n(t)

[0230] Where x noisy (t) represents the audio signal after adding noise; x(t) represents the original audio signal; n(t) represents the noise signal; and α is the noise intensity coefficient, which is adjusted according to the required signal-to-noise ratio.

[0231] Data augmentation operation: Through the above noise adding method, all selected audio samples are processed to generate an augmented data set. Ensure that the augmented data set still maintains the balance of the original data set, i.e. the ratio of false audio and real audio remains unchanged.

[0232] This embodiment enhances the diversity of the data set by randomly adding noise, so that the model can adapt to different noise environments and improve the robustness to noise interference; the data augmentation operation can increase the variability of the training data, effectively reduce the risk of model overfitting, and improve the generalization ability; by adding different types of noise, various situations in the real recording environment are simulated, so that the model has better performance in actual application.

[0233] Further, the seventh embodiment of the training method of the speech authentication model is proposed, and after the step S40, the method further comprises:

[0234] Step f1, obtaining a to-be-detected audio, inputting the to-be-detected audio into the speech authentication model, and obtaining a true or false classification result of the to-be-detected audio.

[0235] In this embodiment, the trained speech authentication model can be used to detect speech audio and determine its authenticity. The specific steps can include:

[0236] Obtaining a to-be-detected audio: obtaining a to-be-detected audio sample from an actual application scenario. These audios can come from user recordings, telephone recordings, voice assistants and other sources. Necessary preprocessing operations are performed on the to-be-detected audio, such as noise reduction, removing silent segments, volume normalization, etc., to ensure that the audio quality meets the model input requirements.

[0237] The audio to be detected is input into the speech authentication model: the trained feature extraction module (such as Whisper, HuBERT, wav2vec2.0 pre-training model) is used to extract the intermediate features, duration features and abstract features of the audio to be detected. The extracted features are fused by a multi-head attention mechanism or other fusion methods to generate fused features. The fused features are input into the expert model and the gating network to obtain the output of the expert model and the weight thereof. The output of the expert model and the weight are combined by weighting to generate target feature output, which is then input into two fully connected layers to obtain the final prediction label.

[0238] The authenticity classification probability of the audio to be detected is obtained through the output of the model (the result after the sigmoid activation function). According to the prediction probability, a threshold (such as 0.5) is set to determine the authenticity of the audio to be detected. The audio to be detected is higher than the threshold, and the audio to be detected is lower than the threshold.

[0239] The embodiment can efficiently and accurately classify the authenticity of the audio by obtaining the audio to be detected and inputting it into the trained speech authentication model, and can be applied to various practical application scenarios such as voice authentication, telephone fraud prevention, voice assistant security, etc.

[0240] The application also provides a speech authentication model training device.

[0241] Reference Figure 3 , Figure 3 The function module schematic diagram of the first embodiment of the speech authentication model training device of the application is shown in the figure. The speech authentication model training device comprises:

[0242] An audio acquisition module acquires a first preset number of false audios and a second preset number of real audios.

[0243] A feature extraction module extracts intermediate features, duration features and abstract features of the false audios and the real audios through a whisper pre-training model, a hubert pre-training model and a wav2vec2.0 pre-training model.

[0244] A feature fusion module fuses the intermediate features, duration features and abstract features to generate fused features, and inputs the fused features into an expert model and a gating network to obtain an expert model output and a weight of the expert model output.

[0245] A model training module trains the expert model and the gating network according to the expert model output and the weight of the expert model output to obtain a speech authentication model.

[0246] In addition, the embodiment of the application also provides a storage medium.

[0247] The storage medium stores a training program of the voice authentication model, and the training program of the voice authentication model, when executed by the processor, implements the steps of the voice authentication model training method described above.

[0248] The storage medium specific embodiments of the present application are basically the same as the voice authentication model training method described above, and will not be repeated here.

[0249] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative, not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims. Any equivalent structure or equivalent flow conversion made by using the contents of the present application specification and drawings, or directly or indirectly applied to other related technical fields, all belong to the protection of the present application.

[0250] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, program segment or part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0251] In addition, each of the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each of the modules can exist independently, or two or more modules can be integrated to form an independent part. When the functions are realized in the form of software functional modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0252] It should be noted that, in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0253] The above is only various embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a voice anti-spoofing model, characterized in that, The training method of the voice authentication model comprises the following steps: Obtain a first preset number of false audios and a second preset number of real audios; Extract intermediate features, duration features, and abstract features of the false audios and the real audios through whisper pre-training models, hubert pre-training models, and wav2vec2.0 pre-training models respectively; Fuse the intermediate features, the duration features, and the abstract features to generate fused features, input the fused features into an expert model and a gating network to obtain an expert model output and a weight of the expert model output; Train the expert model and the gating network according to the expert model output and the weight of the expert model output to obtain a voice authentication model.

2. The method of claim 1, wherein the voice authentication model is trained using a plurality of voice data of a plurality of speakers, and the plurality of voice data is obtained from a plurality of voice samples of the plurality of speakers. The step of fusing the intermediate features, the duration features, and the abstract features to generate fused features comprises: Fuse the intermediate features, the duration features, and the abstract features using a multi-head attention mechanism, take the intermediate features as Key (K) and Value (V), and take the duration features and the abstract features as Query (Q) to perform self-attention operation, and the calculation formula is: wherein Q represents Query, K represents Key, V represents Value, d k represents the dimension of Key; Generate the fused features according to the Attention output.

3. The method of claim 1, wherein the voice authentication model is trained by using a voice of a user of the electronic device. The step of inputting the fused features into the expert model and the gating network to obtain an expert model output and a weight of the expert model output comprises: The expert model comprises three expert networks with different structures, the fused features are input into the three expert networks to obtain the expert model output; The gating network uses a softmax function to model a probability distribution, the fused features are input into the gating network to obtain the weight of the expert model output.

4. The method of claim 1, wherein the voice authentication model is trained by using a voice of a user of the electronic device. The step of training the expert model and the gating network according to the expert model output and the weight of the expert model output comprises: Input the expert model output and the weight of the expert model output into two fully connected layers to obtain a predicted label; Train the expert model and the gating network using a binary cross-entropy loss function according to the predicted label to update parameters of the expert model and the gating network.

5. The method of training a voice authentication model of claim 4, wherein, The step of inputting the expert model output and the weight of the expert model output into two fully connected layers comprises: Weighted combine the expert model output and the weight of the expert model output to generate a target feature output, and input the target feature output into the two fully connected layers.

6. The method of training a voice authentication model of claim 1, wherein, After the steps of obtaining a first preset number of false audios and a second preset number of real audios, the method further comprises: Add noise to the false audios and the real audios according to the first preset number and the second preset number to perform a data enhancement operation.

7. The method of training a voice authentication model of claim 1, wherein, After the step of obtaining a voice authentication model, the method further comprises: Obtain a to-be-detected audio, input the to-be-detected audio into the voice authentication model to obtain a true or false classification result of the to-be-detected audio.

8. A training device for a voice authentication model, characterized in that, The training device of the voice authentication model comprises: An audio acquisition module that acquires a first preset number of false audios and a second preset number of real audios; The feature extraction module extracts intermediate features, duration features and abstract features of the false audio and the real audio through a whisper pre-training model, a hubert pre-training model and a wav2vec2.0 pre-training model respectively. The feature fusion module fuses the intermediate features, the duration features and the abstract features to generate fusion features, inputs the fusion features into an expert model and a gating network, and obtains an expert model output and a weight of the expert model output. The model training module trains the expert model and the gating network according to the expert model output and the weight of the expert model output, and obtains a speech authentication model.

9. A device for training a voice anti-spoofing model, comprising: The training device of the speech authentication model comprises a memory, a processor and a training program of the speech authentication model stored on the memory and executable on the processor. When the training program of the speech authentication model is executed by the processor, the steps of the training method of the speech authentication model according to any one of claims 1-7 are implemented.

10. A storage medium, characterized by The storage medium stores a training program of the speech authentication model. When the training program of the speech authentication model is executed by the processor, the steps of the training method of the speech authentication model according to any one of claims 1-7 are implemented.

Images