Voice authentic identification method and related equipment

Abstract

The invention belongs to the field of artificial intelligence and big data, is applied to the field of smart cities, and relates to a voice authentic identification method, which comprises the following steps: acquiring a voice data set, and training a pre-trained voice authentic identification model through the voice data set to obtain a trained voice authentic identification model; carrying out vocoder classification processing on a target audio through the trained voice authentic identification model, and determining a corresponding vocoder to which the target audio belongs; performing feature extraction processing on the target audio in a vocoder corresponding to the target audio to obtain a target audio feature; and performing authentic identification processing on the target audio features, and determining an authentic identification processing result of the target audio. The invention further provides a voice authentic identification device, computer equipment and a storage medium. According to the invention, the problems of low accuracy and poor generalization of an existing speech authentication method in a real environment are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a voice authentication method and related equipment. Background Art

[0002] With the development of deepfake technology, it has become easier to generate highly realistic fake human voice audio. The mainstream speech generation methods are mainly divided into two forms. One is to synthesize the speech audio of the target task through a file, that is, TTS. The other is to convert the source speaker's voice into the target speaker's voice and keep the audio content unchanged, that is, voice conversion VC. The development of convolutional neural networks has made the generated speech more and more realistic. Therefore, a speech authentication method is urgently needed to solve the problem of low accuracy and poor generalization of existing speech authentication methods in real environments. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a speech authentication method and related equipment to solve the problems of low accuracy and poor generalization of existing speech authentication methods in real environments. Its main purpose is to classify the input audio by introducing vocoder features in the model, and to improve the accuracy of speech authentication by using vocoder feature recognition, and to effectively improve the detection accuracy without increasing the complexity of reasoning.

[0004] In order to solve the above technical problems, the present application provides a method for detecting fake voices, which adopts the following technical solutions:

[0005] Acquire a speech data set, and train a pre-trained speech authentication model using the speech data set to obtain a trained speech authentication model;

[0006] Performing vocoder classification processing on the target audio through the trained speech authentication model to determine the corresponding vocoder to which the target audio belongs;

[0007] Performing feature extraction processing on the target audio in a corresponding vocoder to which the target audio belongs to obtain target audio features;

[0008] Perform authentication processing on the target audio feature to determine an authentication result of the target audio.

[0009] Furthermore, the speech data set includes target generated audio and annotations of the target generated audio, and the acquiring of the speech data set includes:

[0010] Get sample real audio and different categories of vocoders;

[0011] Performing audio generation processing on the sample real audio in the different categories of vocoders to obtain generated audio of different categories;

[0012] Calculating the correlation between the generated audio and the sample real audio, and determining a correlation threshold corresponding to the type of the vocoder, and determining the generated audio whose correlation is greater than the correlation threshold as the target generated audio;

[0013] Determine the annotation of the target generated audio, wherein the annotation of the target generated audio is composed of two parts: the type of the vocoder generating the audio and fake.

[0014] Furthermore, the calculating the correlation between the generated audio and the sample real audio includes:

[0015] Calculating the high pitch similarity between the generated audio and the sample real audio;

[0016] Calculating the bass similarity between the generated audio and the sample real audio;

[0017] Calculating the lexical similarity between the generated audio and the sample real audio;

[0018] Calculating the global semantic similarity between the generated audio and the sample real audio;

[0019] The high-pitched similarity, the low-pitched similarity, the lexical similarity and the global semantic similarity are weightedly fused to obtain the correlation between the generated audio and the sample real audio.

[0020] Furthermore, the determining the correlation threshold corresponding to the type of the vocoder includes:

[0021] Determining weights corresponding to categories of the vocoders based on generation accuracy rates of the vocoders of different categories;

[0022] Determine the effective duration and the number of effective phonemes in the real audio of the sample, wherein the effective duration is the sum of the durations of the effective phonemes, and the effective phonemes are factors of the person;

[0023] The weight, the effective duration, and the number of effective phonemes are weighted and summed to obtain the correlation threshold corresponding to the type of the vocoder.

[0024] Furthermore, before the pre-trained speech authentication model is trained by the speech data set to obtain a trained speech authentication model, the method further includes:

[0025] Obtaining a pre-trained speech authentication model, adding a vocoder feature extraction module to the pre-trained speech authentication model, wherein the vocoder feature extraction module performs pre-emphasis processing on the vocoder features of different categories to obtain the vocoder features after the pre-emphasis processing;

[0026] Sending the pre-emphasized vocoder features to an instance normalization layer and a parameterized analysis filter bank to process them into a time-frequency representation, thereby obtaining a vocoder feature time-frequency representation;

[0027] The characteristic time-frequency representation of the vocoder passes through three backbone blocks with residual connections, wherein the connection mode of the three backbone blocks with residual connections is that the output of the first backbone block is used as the input of the second backbone block, and the output of the first backbone block and the output of the second backbone block are added as the input of the third backbone block, wherein the output of the first backbone block is pooled, and the output of the first backbone block, the output of the second backbone block, and the output of the third backbone block after the pooling operation are simultaneously sent to the convolution layer as inputs for feature extraction;

[0028] The inputs of the three backbone blocks with residual connections are added together, and connected to a one-dimensional convolutional layer and two fully connected layers to obtain the output of the vocoder.

[0029] Furthermore, feature processing is performed on the target audio in a corresponding vocoder to which the target audio belongs to obtain target audio features, including:

[0030] Performing time-frequency analysis processing on the target audio in a corresponding vocoder to which the target audio belongs, to obtain the target audio after the time-frequency analysis processing;

[0031] The target audio processed by the time-frequency analysis is input into three backbone blocks with residual connections for feature extraction processing to obtain target audio features.

[0032] Furthermore, the target audio after the time-frequency analysis is input into three backbone blocks with residual connections for feature extraction to obtain target audio features, including:

[0033] Inputting the target audio after the time-frequency analysis processing into the first backbone block for first feature extraction processing, and outputting a first target audio feature to be processed;

[0034] Inputting the first target audio feature to be processed into a second backbone block to extract a second feature, and outputting a second target audio feature to be processed;

[0035] Inputting the first target audio feature to be processed and the second target audio feature to be processed into a third backbone block to perform a third feature extraction process to obtain a third target audio feature to be processed;

[0036] The first target audio feature to be processed, the second target audio feature to be processed, and the third target audio feature to be processed are input into a convolutional layer for fourth feature extraction processing to obtain a target audio feature.

[0037] In order to solve the above technical problems, the present application also provides a voice authentication device, which adopts the following technical solution:

[0038] A training module is used to obtain a speech data set, and train a pre-trained speech authentication model using the speech data set to obtain a trained speech authentication model;

[0039] A classification processing module is used to perform vocoder classification processing on the target audio through the trained speech authentication model to determine the corresponding vocoder to which the target audio belongs;

[0040] An extraction processing module, used for performing feature extraction processing on the target audio in a corresponding vocoder to which the target audio belongs, to obtain target audio features;

[0041] The authentication processing module is used to perform authentication processing on the target audio features and determine the authentication processing result of the target audio.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0043] The computer device includes a memory and a processor, the memory stores computer-readable instructions, and the processor implements the steps of any one of the voice authentication methods described in the embodiments of the present application when executing the computer-readable instructions.

[0044] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0045] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the voice authentication method described in any one of the embodiments of the present invention are implemented.

[0046] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: obtaining a speech data set, training a pre-trained speech authentication model through the speech data set to obtain a trained speech authentication model, performing vocoder classification processing on the target audio through the trained speech authentication model, determining the corresponding vocoder to which the target audio belongs, performing feature extraction processing on the target audio in the corresponding vocoder to which the target audio belongs to obtain the target audio feature, performing authentication processing on the target audio feature, and determining the authentication processing result of the target audio, by inputting the target audio into the model, using a normalization layer and a parameterized analysis filter group to process the audio into a time-frequency representation, and then sending it to three backbone blocks with residual connections for feature extraction, and then sending the feature to a GRU layer for information aggregation, and finally obtaining the authentication processing result of the target audio through a fully connected layer and a softmax function, thereby solving the problems of low accuracy and poor generalization of speech authentication in real environments in the existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0049] Figure 2 A flowchart of an embodiment of a voice authentication method according to the present application;

[0050] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S201;

[0051] Figure 4 yes Figure 3 A flowchart of a specific implementation method of calculating the correlation between the generated audio and the sample real audio in step S2014;

[0052] Figure 5 yes Figure 3 A flowchart of a specific embodiment of determining the correlation threshold corresponding to the type of the vocoder in step S2014;

[0053] Figure 6 yes Figure 2 A flowchart of another specific embodiment of the method for voice authentication in step 2;

[0054] Figure 7 yes Figure 2A flowchart of a specific implementation method of step S203;

[0055] Figure 8 yes Figure 7 A flowchart of a specific implementation method of step S2032;

[0056] Fig. 9 is a structural diagram of the connection method of three backbone blocks according to the present application;

[0057] Fig.10 is a structural schematic diagram of an embodiment of a voice authentication device according to the present application;

[0058] Fig.11 yes Fig.10 A schematic diagram of a structure of a specific implementation of the acquisition module 1001;

[0059] Fig.12 yes Fig.11 A structural diagram of a specific implementation of the first determination submodule 10013;

[0060] Fig.13 yes Fig.11 A structural diagram of another specific implementation of the first determination submodule 10013;

[0061] Fig.14 yes Fig.10 A structural diagram of a specific implementation of a voice authentication device 1000;

[0062] Fig.15 yes Fig.10 A schematic diagram of a specific implementation of the extraction processing module 1003;

[0063] Fig.16 yes Fig.15 A schematic diagram of a specific implementation of the extraction processing submodule 10032;

[0064] Fig.17 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0066] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0068] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0069] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0070] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0071] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0072] It should be noted that the voice authentication method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the voice authentication device is generally arranged in the server / terminal device.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0074] Continue to refer Figure 2 , shows a flow chart of an embodiment of a voice authentication method according to the present application. The voice authentication method comprises the following steps:

[0075] Step S201: Acquire a speech data set, and train a pre-trained speech authentication model using the speech data set to obtain a trained speech authentication model.

[0076] In this embodiment, the electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the voice authentication request of the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0077] The above-mentioned terminal devices may be terminal devices equipped with remote interaction functions such as personal assistants and intelligent customer service.

[0078] The above-mentioned voice authentication method can be applied to online document authentication, online document management and other services.

[0079] Specifically, the electronic device can be used in different industries such as medical voice authentication, property insurance voice authentication, finance voice authentication, and store voice authentication.

[0080] The above-mentioned speech data set includes target generated audio and annotations of the target generated audio; the above-mentioned target generated audio includes different categories of vocoder generated audio; the above-mentioned vocoder generated audio is generated by sample real audio and different categories of vocoders; the correlation between the above-mentioned target generated audio and the sample real audio is greater than a preset correlation threshold; the above-mentioned correlation threshold is determined according to the content length of the sample real audio and the category of the vocoder.

[0081] The annotation of the target generated audio is composed of two parts: the type of vocoder that generates it and fake. Different vocoders have different characteristics, such as timbre, pronunciation speed, etc. The fake can be understood as a generated sound signal, which is generated by the vocoder algorithm, rather than a real sound.

[0082] The above-mentioned vocoder is a device that converts sound signals into digital signals for processing and transmission. Its function is to convert natural speech into a machine-readable form. The vocoder can convert sound signals into digital signals by sampling, quantizing and encoding the sound signals, so that the computer system can analyze and process them. The vocoder can be applied to speech synthesis, speech communication, speech recognition, etc. In speech synthesis, the vocoder can convert text data into machine-readable speech signals to generate natural and smooth speech output; in speech communication, the vocoder can convert the sound signals of both parties of the call into digital signals for transmission and processing, thereby achieving high-quality speech communication; in speech recognition, the vocoder can convert natural speech into digital signals so that the computer system can recognize and understand it.

[0083] The pre-trained voice authentication model is an untrained voice authentication model, and can be a model for voice authentication built based on deep learning or machine learning technology. The pre-trained voice authentication model can be a model based on the SignatureNet model, a convolutional neural network (CNN), a long short-term memory network (LSTM), etc.

[0084] The above training can be supervised training. Supervised training is a machine learning training process based on labeled data. Its core is to build a speech authentication model by learning the mapping relationship between input and output.

[0085] The trained speech authentication model is a model obtained through learning with training data from a large number of real speech data sets, and is a speech authentication model that can judge the authenticity of new speech data.

[0086] Furthermore, the spectrum of the audio generated by the vocoder is not exactly the same as the spectrum of the real audio. The spectrum difference between the audio generated by each type of vocoder and the original audio is different, and the audio generated by each type of vocoder has its own unique implicit features.

[0087] In one embodiment, about 120,000 audios generated by the above-mentioned different vocoders can be selected. In order to ensure that the number of audios generated by each type of vocoder remains roughly balanced, about 20,000 real audios are also selected, totaling 140,000 audios to form a speech data set, where the generated audio of each audio is composed of the type of vocoder that generates it and fake, and the annotation of the real audio is composed of two real parts. The ratio of training and test data is 6:1.

[0088] It should be noted that during the training process, the parameters of the model will be adjusted according to the supervisory signals on the speech dataset, so that the model can better identify the authenticity of different speech.

[0089] Step S202: Perform vocoder classification processing on the target audio through the trained speech authentication model to determine the corresponding vocoder to which the target audio belongs.

[0090] In this embodiment, the target audio refers to the audio that needs to be authenticated by voice.

[0091] Specifically, the target audio may be input into a trained speech authentication model to determine the corresponding vocoder to which the target audio belongs.

[0092] The above-mentioned vocoder classification process can be understood as a process of further classifying the target audio to determine the type of vocoder to which the target audio belongs.

[0093] Furthermore, a vocoder feature extraction module is added based on the speech authentication model. The vocoder feature extraction module can assist the speech authentication model in classifying the audio, ensuring that the extracted features can accurately capture the different statistical characteristics of the vocoder, making it more sensitive to different vocoder features.

[0094] The above-mentioned vocoder features include fundamental frequency features, harmonic features, phase features, etc. The fundamental frequency feature is the feature of the sound source, that is, the position of each pulse in the pulse sequence generated when the vocal cords vibrate; the spectrum parameter is modulated by the vocal tract (laryngeal cavity, oral cavity, lips, teeth, etc.), which can produce different vowels and consonants; the phase feature is the phase information of the audio signal in the frequency domain, and the phase feature describes the rotation direction and amplitude of the audio signal waveform.

[0095] Specifically, the process of training the feature extraction module of the vocoder includes: first, the original waveform of the original speech data set is pre-emphasized, and then it is sent to the instance normalization layer and the parameterized analysis filter group to process it into a time-frequency representation, and then the representation passes through three backbone blocks with residual connections. The connection method of the three backbone blocks is that the output of the first block is used as the input of the second block, and the output of the second block is added as the input of the third block. In order to reduce the overfitting of the model, the output of the first block is subjected to the maximum pooling operation, and then the output of the other two blocks is simultaneously used as input to a convolution layer with a batch normalization position. Among them, the structure of each backbone block is similar to the backbone block of ECAPA-TDNN, and the AFMS module is used instead of the squeeze excitation, and the maximum pooling operation is performed before the AFMS. Then the inputs of the three backbone blocks are added, followed by a one-dimensional convolution layer, and two fully connected layers to obtain the output of the vocoder. It should be noted that the convolution layer and the fully connected layer in the training process are only used in the training process, and are not used in the reasoning (false detection processing) stage, mainly to improve the efficiency of the speech false detection model and accelerate the reasoning speed. For network structures with large parameters in convolutional layers and fully connected layers, if they are also used in the inference phase, it will lead to increased computation and memory consumption, thereby reducing the inference speed of the entire speech authentication model. In addition, the parameters of the convolutional layers and fully connected layers have been learned in the training phase, and these parameters can be directly used for inference, avoiding repeated calculations during the inference process.

[0096] The above three backbone blocks with residual connections can be understood as using three neural network backbone structures with residual connections (such as the Inception module in the convolutional neural network CNN) to extract audio features. The above residual connection is a skip connection that allows the gradient to propagate directly to deeper layers, thereby alleviating the gradient vanishing problem. The above Inception module mainly solves the problem of CNN when processing inputs of different sizes and improves the efficiency and performance of the model.

[0097] The above-mentioned ECAPA-TDNN is a time delay neural network (Time Delay Neural Network), which is characterized by introducing time delay and gating units into the traditional recurrent neural network (RNN) structure.

[0098] The above-mentioned AFMS module is the attention mechanism (Auxiliary Feature Matrix Sum). The main function of the AFMS module is to introduce the attention mechanism in the feature interaction process to learn the influence of different features on the prediction results.

[0099] Furthermore, in order to more accurately capture different vocoder features to identify vocoder types, a multi-classification loss function is used to constrain the vocoder feature extraction module. That is, the features of the vocoder are identified and the vocoder category is output (all categories are all vocoder categories + 1). Through vocoder identification, the feature extraction module is optimized to make the network more sensitive to vocoder features, and the probability that the input audio belongs to a certain vocoder category is obtained through the softmax function. The loss function used is:

[0100]

[0101] Among them, y i Indicates that the audio belongs to the i-th type of vocoder, p i It represents the probability that the audio belongs to the i-th type of vocoder, and C-1 is the number of all categories.

[0102] The outputs of the three backbone networks are fed into the GRU layer (Gated Recurrent Unit layer), which aggregates the frame-level representation into the speech-level representation, and then connects a fully connected layer to finally use the softmax function to get the predicted result. In this step, the application uses a binary classification loss function to distinguish between true and false audio. The vocoder features are used to assist the binary classification task to further improve the accuracy of speech authentication.

[0103] The binary classification loss function is:

[0104] Loss2=-[ylog(σ(x n ))+(1-y n )log(1-σ(x n ))]

[0105] Where y is the vocoder to which the audio belongs, σ(x n ) is the sigmoid function, which maps x to the (0,1) interval: y n is the vocoder to which the audio is predicted.

[0106] The loss function of the trained speech authentication model is Loss = λLoss1+(1-λ)Loss2, where λ is an adjustable hyperparameter used to control the different weights of the two-class classification and multi-classification branches.

[0107] Step S203: performing feature extraction processing on the target audio in the vocoder corresponding to the target audio to obtain the target audio feature.

[0108] In this embodiment, the feature extraction process described above can be understood as a process of extracting a representative feature vector representation from the target audio.

[0109] Specifically, the target audio may be preprocessed, such as removing silent parts, reducing noise, etc. According to the type of vocoder to which the target audio belongs, a corresponding feature extraction algorithm is selected, such as short-time energy, etc. The feature extraction algorithm is applied to the target audio to obtain the features of the target audio.

[0110] The target audio features are a set of values ​​extracted from the original target audio signal and are used to describe the intrinsic characteristics of the audio. The target audio features can be derived from the time domain or frequency domain analysis of the audio and can reflect the key information of the audio data.

[0111] Step S204: perform authentication processing on the target audio features to determine the authentication processing result of the target audio.

[0112] In this embodiment, the above-mentioned authentication process can be understood as a process of identifying and confirming false audio.

[0113] In this embodiment, the target audio is input into the trained speech authentication model, and the target audio is processed into a time-frequency representation using a normalization layer and a parametric analysis filter bank (paramFbank), and then sent to three backbone blocks with residual connections for feature extraction, and then the features are sent to the GRU layer (gated recurrent unit layer) for information aggregation. Finally, the final authentication result of the predicted target audio is obtained through a fully connected layer and a softmax function.

[0114] The above GRU (Gated Recurrent Unit) is a recurrent neural network structure that can model and predict sequence data. The GRU layer (Gated Recurrent Unit layer) integrates the features extracted from the three backbone blocks with residual connections to form more global context information.

[0115] The Softmax function is a commonly used activation function, which is used to normalize a numerical vector into a probability distribution vector, and the sum of each probability is 1.

[0116] In this embodiment, the present application adds a module of vocoder recognition on the basis of the voice authentication model, so that the voice authentication model can more accurately capture the features of the vocoder while distinguishing true and false audio, and makes the feature extractor sensitive to the vocoder features, which will make it easier to distinguish true and false audio. The present application uses the idea of ​​multi-task training, integrating binary classification and multi-classification tasks into one model, both of which share weights, and uses vocoder feature recognition to improve the accuracy of voice authentication, and can effectively improve the goal of detection accuracy without increasing the complexity of reasoning. The present invention can be applied to business scenarios such as financial fraud.

[0117] The present application obtains a speech data set, trains a pre-trained speech authentication model through the speech data set, obtains a trained speech authentication model, performs vocoder classification processing on the target audio through the trained speech authentication model, determines the vocoder to which the target audio belongs, performs feature extraction processing on the target audio in the vocoder to which the target audio belongs, obtains the target audio feature, performs authentication processing on the target audio feature, and determines the authentication processing result of the target audio, thereby solving the problems of low accuracy and poor generalization of speech authentication in real environments in existing methods.

[0118] Continue to refer Figure 3 , shown in Figure 2 Flow chart of a specific implementation of step S201 in FIG. Step S201 specifically includes the following steps:

[0119] Step S2011: Obtain sample real audio and different types of vocoders.

[0120] In this embodiment, the sample real audio may be understood as real audio that has not been modified or edited.

[0121] The above-mentioned different categories of vocoders include mel-GAN, full-band-melGAN, multiband-GAN, HiFi-GAN, parallel-GAN based on GAN, waveRNN based on autoregressive model and DiffWave based on diffusion model.

[0122] The above-mentioned vocoder is a device that converts sound signals into digital signals for processing and transmission. Its function is to convert natural speech into a machine-readable form. The vocoder can convert sound signals into digital signals by sampling, quantizing and encoding the sound signals, so that the computer system can analyze and process them.

[0123] Step S2012: Perform audio generation processing on the sample real audio in different categories of vocoders to obtain different categories of generated audio.

[0124] In this embodiment, the above audio generation process can be understood as a process of performing audio processing on real audio samples to generate audio of different categories.

[0125] Step S2013, calculate the correlation between the generated audio and the sample real audio, and determine the correlation threshold corresponding to the type of the vocoder, and determine the generated audio with a correlation greater than the sound correlation threshold as the target generated audio.

[0126] In this embodiment, a speech analysis technique may be used to calculate the correlation between the generated audio and the sample real audio, for example, using techniques such as Cepstrum Similarity (CCS) or sound event detection to calculate the similarity of the two audios.

[0127] Step S2014: Determine the annotation of the target generated audio.

[0128] In this embodiment, the annotation of the target generated audio is composed of two parts: the type of the vocoder that generates it and fake. Different vocoders have different characteristics, such as timbre, pronunciation speed, etc. The fake can be understood as a generated sound signal, which is generated by the vocoder algorithm rather than a real sound.

[0129] The present application obtains sample real audio and different categories of vocoders, and performs audio generation processing on the sample real audio in different categories of vocoders to obtain different categories of generated audio, and then calculates the correlation between the generated audio and the sample real audio, and determines the correlation threshold corresponding to the category of the vocoder, and determines the generated audio with a correlation greater than the correlation threshold as the target generated audio, determines the label of the target generated audio, so as to better manage and use the generated audio data and improve the quality and acceptability of the generated audio.

[0130] Continue to refer Figure 4 , showing that Figure 3 Flow chart of a specific implementation of step S2014 in FIG. Step S2014 specifically includes the following steps:

[0131] Step S20141, calculating the high-pitched similarity between the generated audio and the sample real audio.

[0132] In this embodiment, the treble similarity can be understood as the similarity of the treble parts in the two audio signals. The similarity between the generated audio and the sample real audio can be measured by comparing the frequency, amplitude and other characteristics of the treble parts in the two audios.

[0133] Furthermore, the audio signal can be converted into a spectrogram using a spectrum analysis algorithm to view the relative strength of different frequency components. By comparing the spectrograms of the generated audio and the real audio, their similarity in the high-pitched part can be determined. Spectrum analysis algorithms include Fast Fourier Transform (FFT) and Short Time Fourier Transform (STFT).

[0134] The audio signal can be converted into a cepstrum through the cepstrum analysis algorithm to view the phase information of the audio signal. By comparing the cepstrum of the generated audio and the real audio, their similarity in the high-pitched part can be determined. Cepstrum analysis algorithms include Mel-frequency cepstrum coefficients (MFCC) and linear prediction cepstrum coefficients (LPCC).

[0135] Step S20142: Calculate the bass similarity between the generated audio and the sample real audio.

[0136] In this embodiment, the above-mentioned bass similarity can be understood as the similarity of the bass parts in the two audio signals. The similarity between the generated audio and the sample real audio can be measured by comparing the frequency, amplitude and other characteristics of the bass parts in the two audios.

[0137] Furthermore, the audio signal can be converted into a spectrogram by a spectrum analysis algorithm to see the relative strength of different frequency components. By comparing the spectrograms of the generated audio and the real audio, their similarity in the bass part can be determined. Spectrum analysis algorithms include fast Fourier transform (FFT) and short-time Fourier transform (STFT).

[0138] The audio signal can be converted into a cepstrum through the cepstrum analysis algorithm to view the phase information of the audio signal. By comparing the cepstrum of the generated audio and the real audio, their similarity in the bass part can be determined. Cepstrum analysis algorithms include Mel-frequency cepstrum coefficients (MFCC) and linear prediction cepstrum coefficients (LPCC).

[0139] Step S20143: Calculate the lexical similarity between the generated audio and the sample real audio.

[0140] In this embodiment, lexical similarity can be understood as the degree of similarity between two sentences or texts in terms of grammatical structure, vocabulary selection, and semantic content of words or phrases. In natural language processing, lexical similarity is usually used to measure the similarity between two sentences at the lexical level. The calculation of lexical similarity is usually based on information such as word co-occurrence, part-of-speech tagging, and syntactic structure. Cosine similarity, Jaccard similarity, edit distance, etc. can be used to calculate the similarity between the generated audio and the sample real audio at the lexical level.

[0141] Step S20144: Calculate the global semantic similarity between the generated audio and the sample real audio.

[0142] In this embodiment, the global speech similarity can be understood as evaluating the similarity between the two speech segments in terms of overall content. A pre-trained language model, such as BERT, GPT-2, etc., can be used to convert text into semantic representations and calculate the similarity scores between them.

[0143] Step S20145: perform weighted fusion processing on the high-pitched similarity, the low-pitched similarity, the lexical similarity and the global semantic similarity to obtain the correlation between the generated audio and the real audio of the sample.

[0144] In this embodiment, after obtaining the treble similarity, bass similarity, lexical similarity and global semantic similarity, the electronic device can obtain the correlation between the generated audio and the sample real audio by weighted fusion processing of the treble similarity, bass similarity, lexical similarity and global semantic similarity.

[0145] The correlation between the generated audio and the real audio of the sample is obtained by weighted fusion processing of high-pitched similarity, low-pitched similarity, lexical similarity and global semantic similarity.

[0146] Specifically, the treble similarity, bass similarity, lexical similarity and global semantic similarity are weighted and fused to obtain the correlation score between the generated audio and the sample real audio. Each treble similarity, bass similarity, lexical similarity and global semantic similarity is assigned a weight value to indicate its relative importance in the final correlation score. The above weighted fusion is to weighted average the treble similarity, bass similarity, lexical similarity and global semantic similarity to form a new correlation. Specifically, first, determine the weight coefficient of each similarity index, and the weight coefficient of each similarity index can be determined by techniques such as cross-validation; then, multiply the value of each similarity index by the corresponding weight coefficient; finally, add all the products to obtain a comprehensive score, which reflects the correlation between the generated audio and the sample real audio. The higher the comprehensive score, the higher the correlation between the generated audio and the sample real audio.

[0147] This application calculates the treble similarity between the generated audio and the sample real audio, calculates the bass similarity between the generated audio and the sample real audio, calculates the lexical similarity between the generated audio and the sample real audio, calculates the global semantic similarity between the generated audio and the sample real audio, and determines the correlation between the generated audio and the sample real audio based on the treble similarity, bass similarity, lexical similarity and global semantic similarity, so as to help users better evaluate the quality of the generated audio.

[0148] Continue to refer Figure 5 , showing that Figure 3 Flow chart of a specific implementation of step S2014 in FIG. Step S2014 specifically includes the following steps:

[0149] Step S20146: Determine the weight corresponding to the type of vocoder based on the generation accuracy of the vocoders of different types.

[0150] In this embodiment, the higher the generation accuracy of the vocoder, the lower the weight. This is because the greater the role of the vocoder in speech synthesis, the higher the quality of the speech it generates should be, so its weight needs to be reduced to balance the weight distribution of the entire speech synthesis system.

[0151] The above generation accuracy can be understood as the correlation between the generated audio generated by the vocoder and the real audio of the sample.

[0152] The above weights are used to measure the performance of different categories of vocoders in generating audio. The weight value can indicate the importance of the vocoder in the overall evaluation.

[0153] Step S20147: Determine the effective duration and the number of effective phonemes in the sample real audio.

[0154] In this embodiment, the effective duration is the total duration of effective phonemes, and the effective phonemes are the phonemes of a person, that is, the voice features of a person.

[0155] The phonemes of the above-mentioned effective representation personnel are meaningful for communication and understanding of speech content, rather than meaningless noise or mispronunciation.

[0156] Furthermore, the sample real audio can be preprocessed, and then effective features can be extracted from the preprocessed sample real audio, and finally the extracted features can be analyzed and counted to determine the effective phonemes therein. The above preprocessing includes noise removal, silence removal, volume balance, etc. to ensure the quality and accuracy of the data. The above extracted features can be extracted using Mel Frequency Cepstral Coefficients (MFCC), spectrogram, etc. The analysis and statistics can be performed by methods such as cluster analysis or factor analysis.

[0157] Step S20148: Perform weighted sum processing on the weight, effective duration, and effective phoneme quantity to obtain a correlation threshold corresponding to the type of the vocoder.

[0158] In this embodiment, the correlation threshold corresponding to the above-mentioned vocoder category is obtained by weighted summation according to the weight, the effective duration and the number of effective phonemes.

[0159] The weighted summation process described above can be understood as a process of multiplying each data by a weight and then adding them together when calculating the sum of a set of data.

[0160] Specifically, a weight value is assigned to the weight, effective duration, and effective number of phonemes corresponding to the category of the vocoder. For example, the weight value corresponding to the category of the vocoder can be assigned to 0.5, the effective duration value can be assigned to 0.3, and the effective number of phonemes can be assigned to 0.2. These weight values ​​can be adjusted according to the importance and application scenario of the vocoder. Then, the weight, effective duration, and effective number of phonemes corresponding to the category of the vocoder are multiplied by their corresponding weight values ​​to obtain the weighted values. Next, based on the default correlation threshold, the weighted value is compared with the default correlation threshold. If the weighted value is greater than the default correlation threshold, it is considered that the vocoder is correlated with the target audio, otherwise it is considered that there is no correlation between them.

[0161] Furthermore, the weight, effective duration, and number of effective phonemes corresponding to the category of the vocoder can be weighted and summed to determine the correlation threshold corresponding to the category of the vocoder. The correlation threshold represents the performance standard that the vocoder should achieve under this category. If the actual correlation of the vocoder exceeds this correlation threshold, it is considered that its performance under this category is qualified. The performance of the vocoder under different categories can be accurately evaluated through the correlation threshold, and the accuracy of the vocoder evaluation is improved.

[0162] The present application determines the weights corresponding to the categories of vocoders based on the generation accuracy of different categories of vocoders, determines the effective duration and the number of effective phonemes in the real audio of the sample, and performs weighted summation processing on the weights, effective duration and the number of effective phonemes to obtain the relevance threshold corresponding to the category of vocoders. The relevance threshold corresponding to the category of vocoders can be determined more accurately, thereby improving work efficiency.

[0163] Continue to refer Figure 6 , showing Figure 2 A flowchart of a specific implementation method of the voice authentication method in FIG. The voice authentication method specifically includes the following steps:

[0164] Step S601, obtaining a pre-trained speech authentication model, adding a vocoder feature extraction module to the pre-trained speech authentication model, performing pre-emphasis processing on different types of vocoder features in the vocoder feature extraction module, and obtaining pre-emphasized vocoder features.

[0165] In this embodiment, the pre-trained voice authentication model is a model for voice authentication built based on deep learning or machine learning technology, and can be a model based on the SignatureNet model, convolutional neural network (CNN), long short-term memory network (LSTM), etc.

[0166] The above-mentioned vocoder feature extraction module can assist the speech authentication model in classifying the audio, ensuring that the extracted features can accurately capture the different statistical characteristics of the vocoder, making it more sensitive to different vocoder features.

[0167] Furthermore, it is necessary to train the vocoder feature extraction module. The training process includes: first, the original waveform of the original vocoder speech data set is pre-emphasized, and then it is sent to the instance normalization layer and the parameterized analysis filter group to process it into a time-frequency representation, and then the representation passes through three backbone blocks with residual connections. The connection method of the three backbone blocks is that the output of the first block is used as the input of the second block, and at the same time, it is added to the output of the second block as the input of the third block. In order to reduce the overfitting of the model, the output of the first block is subjected to the maximum pooling operation, and then it is sent to a convolution layer with a batch normalization position as input together with the outputs of the other two blocks. Among them, the structure of each backbone block is similar to the backbone block of ECAPA-TDNN, and the AFMS module is used instead of the squeeze excitation, and the maximum pooling operation is performed before the AFMS. Then the inputs of the three backbone blocks are added, followed by a one-dimensional convolution layer, and two fully connected layers to obtain the output of the vocoder. It should be noted that the convolutional layers and fully connected layers in the training process are only used in the training process, and not in the inference (authentication processing) stage, mainly to improve the efficiency of the speech authentication model and accelerate the inference speed. For network structures with large parameters of convolutional layers and fully connected layers, if they are also used in the inference stage, it will lead to increased calculation and memory consumption, thereby reducing the inference speed of the entire speech authentication model. Moreover, the parameters of the convolutional layers and fully connected layers have been learned in the training stage, and these parameters can be directly used for inference, avoiding repeated calculations in the inference process.

[0168] The above-mentioned ECAPA-TDNN is a time delay neural network (Time Delay Neural Network), which is characterized by introducing time delay and gating units into the traditional recurrent neural network (RNN) structure.

[0169] The above-mentioned AFMS module is the attention mechanism (Auxiliary Feature Matrix Sum). The main function of the AFMS module is to introduce the attention mechanism in the feature interaction process to learn the influence of different features on the prediction results.

[0170] The above-mentioned vocoder features include fundamental frequency features, harmonic features, phase features, etc. The fundamental frequency feature is the feature of the sound source, that is, the position of each pulse in the pulse sequence generated when the vocal cords vibrate; the spectrum parameter is modulated by the vocal tract (laryngeal cavity, oral cavity, lips, teeth, etc.), which can produce different vowels and consonants; the phase feature is the phase information of the audio signal in the frequency domain, and the phase feature describes the rotation direction and amplitude of the audio signal waveform.

[0171] The above-mentioned pre-emphasis processing can be understood as a processing method for pre-processing the original signal by increasing the amplitude of the high-frequency component so that the frequency response of the signal is flatter when it is processed at the receiving end. Pre-emphasis processing can enhance the energy of high-frequency signals, thereby improving the quality of audio signals and reducing distortion and noise.

[0172] Step S602: Send the pre-emphasized vocoder features to the instance normalization layer and the parameterized analysis filter bank to process them into a time-frequency representation, thereby obtaining a vocoder feature time-frequency representation.

[0173] In this embodiment, the instance normalization layer can accelerate the training process and improve the performance of the model, and the instance normalization layer can normalize the vocoder features.

[0174] The above-mentioned parametric analysis filter bank can determine the spectral characteristics of the vocoder features for subsequent processing.

[0175] The above time-frequency representation is a method of representing a signal in time and frequency dimensions. The time-frequency representation can be used to describe the energy distribution and changes of a signal. By feeding the vocoder features into a parametric analysis filter bank, the time-frequency representation of the vocoder features can be obtained.

[0176] Step S603: The feature time-frequency representation of the vocoder is passed through three backbone blocks with residual connections.

[0177] In this embodiment, the above three backbone blocks with residual connections can be understood as using three neural network backbone structures with residual connections (such as the Inception module in the convolutional neural network CNN) to extract audio features. The above residual connection is a skip connection that allows the gradient to propagate directly to deeper layers, thereby alleviating the gradient vanishing problem. The above Inception mainly solves the problem of CNN when processing inputs of different sizes and improves the efficiency and performance of the model.

[0178] The three backbone blocks are connected in such a way that the output of the first block is used as the input of the second block, and the output of the first backbone block and the output of the second block are added as the input of the third block. In order to reduce the overfitting of the model, the output of the first block is subjected to the maximum pooling operation, and the output of the first backbone block after the pooling operation and the outputs of the other two blocks are simultaneously fed into a convolutional layer with batch normalization positions. The structure of each backbone block is similar to that of the backbone block of ECAPA-TDNN, using the AFMS module instead of the squeeze excitation, and performing the maximum pooling operation before the AFMS.

[0179] Step S604: Add the inputs of the three backbone blocks with residual connections, connect a one-dimensional convolutional layer, and two fully connected layers to obtain the output result of the vocoder.

[0180] In this embodiment, the one-dimensional convolution layer uses a one-dimensional filter to perform a convolution operation on the input vector, thereby extracting feature information of the vocoder feature time-frequency representation.

[0181] The above fully connected layer uses fully connected neurons to perform nonlinear transformation on the input feature vector to obtain the output result of the vocoder.

[0182] The convolutional layers and fully connected layers in the training process are only used in the training process, and not in the inference (authentication processing) stage, mainly to improve the efficiency of the speech authentication model and speed up the inference speed. For network structures with large convolutional and fully connected layer parameters, if they are also used in the inference stage, it will lead to increased computation and memory consumption, thereby reducing the inference speed of the entire speech authentication model. Moreover, the parameters of the convolutional and fully connected layers have been learned in the training stage, and these parameters can be directly used for inference, avoiding repeated calculations during the inference process.

[0183] The present application obtains a pre-trained speech authentication model, adds a vocoder feature extraction module to the pre-trained speech authentication model, performs pre-emphasis processing on different categories of vocoder features in the vocoder feature extraction module to obtain pre-emphasized vocoder features, sends the pre-emphasized vocoder features to an instance normalization layer and a parameterized analysis filter group to process them into a time-frequency representation to obtain a vocoder feature time-frequency representation, passes the vocoder feature time-frequency representation through three backbone blocks with residual connections, adds the inputs of the three backbone blocks with residual connections, and then connects a one-dimensional convolution layer and two fully connected layers to obtain the output result of the vocoder, and trains the vocoder feature extraction module to ensure that the extracted features can accurately capture the different statistical characteristics of the vocoder, thereby improving the accuracy and efficiency of speech processing tasks.

[0184] Continue to refer Figure 7 , showing Figure 2Flow chart of a specific implementation manner of step S203 in FIG. Step S203 specifically includes the following steps:

[0185] Step S2031: Perform time-frequency analysis on the target audio in the vocoder corresponding to the target audio to obtain the target audio after the time-frequency analysis.

[0186] In this embodiment, the above-mentioned time-frequency analysis processing is used to convert non-stable signals (such as sound, vibration, etc.) from the time domain to the frequency domain for analysis and processing. The time-frequency analysis processing extracts the characteristics of the signal in the time domain and frequency domain by performing time-frequency transformation on the signal, so as to better understand the nature and characteristics of the signal.

[0187] Step S2032: Input the target audio after the time-frequency analysis processing into three backbone blocks with residual connections for feature extraction processing to obtain target audio features.

[0188] In this embodiment, the above three backbone blocks with residual connections can be understood as using three neural network backbone structures with residual connections (such as the Inception module in the convolutional neural network CNN) to extract audio features. The above residual connection is a skip connection that allows the gradient to propagate directly to deeper layers, thereby alleviating the gradient vanishing problem. The three backbone block structures with residual connections help improve the performance and generalization ability of the model.

[0189] The present application obtains the target audio after time-frequency analysis by performing time-frequency analysis on the target audio in the corresponding vocoder to which the target audio belongs, and inputs the target audio after time-frequency analysis into three backbone blocks with residual connections for feature extraction to obtain the target audio features, which can effectively improve the accuracy and reliability of feature extraction, thereby improving the accuracy and efficiency of speech processing tasks.

[0190] Continue to refer Figure 8 , showing Figure 7 Flow chart of a specific implementation manner of step S2032 in FIG. Step S2032 specifically includes the following steps:

[0191] Step S20321: Input the target audio after the time-frequency analysis processing into the first backbone block for first feature extraction processing, and output the first target audio feature to be processed.

[0192] Step S20322: input the first target audio feature to be processed into the second backbone block for second feature extraction processing, and output to obtain the second target audio feature to be processed.

[0193] Step S20323: input the first target audio feature to be processed and the second target audio feature to be processed into the third backbone block for third feature extraction processing to obtain the third target audio feature to be processed.

[0194] Step S20324: input the first target audio feature to be processed, the second target audio feature to be processed, and the third target audio feature to be processed into the convolution layer for fourth feature extraction processing to obtain the target audio feature.

[0195] In this embodiment, the target audio after time-frequency analysis is input into three backbone blocks with residual connections for feature extraction to obtain target audio features. The three backbone blocks with residual connections can be understood as using three neural network backbone structures with residual connections (such as the Inception module in the convolutional neural network CNN) to extract audio features. The above-mentioned residual connection is a skip connection that allows the gradient to propagate directly to deeper layers, thereby alleviating the gradient vanishing problem. The three backbone block structures with residual connections help improve the performance and generalization ability of the model.

[0196] The output of the first backbone block is used as the input of the second backbone block, and is added to the output of the second backbone block as the input of the third backbone block. This connection method can make the three backbone blocks interconnected and enhance the propagation and transfer of features.

[0197] The present application inputs the target audio after time-frequency analysis processing into the first backbone block for first feature extraction processing, outputs the first target audio feature to be processed, inputs the first target audio feature to be processed into the second backbone block for second feature extraction processing, outputs the second target audio feature to be processed, inputs the first target audio feature to be processed and the second target audio feature to be processed into the third backbone block for third feature extraction processing, obtains the third target audio feature to be processed, inputs the first target audio feature to be processed, the second target audio feature to be processed and the third target audio feature to be processed into the convolution layer for fourth feature extraction processing, obtains the target audio feature, which can effectively improve the accuracy and reliability of feature extraction.

[0198] like Fig. 9 As shown, a structural diagram of the connection method of the three backbone blocks according to the present application is shown. Specifically, the output of the first block is used as the input of the second block, and is added to the output of the second block as the input of the third block. In order to reduce the overfitting of the model, the output of the first block is subjected to a maximum pooling operation, and then is fed into a convolutional layer with a batch normalization position as input together with the outputs of the other two blocks. Among them, the structure of each backbone block is similar to the backbone block of ECAPA-TDNN, using the AFMS module instead of the squeeze excitation, and performing a maximum pooling operation before the AFMS.

[0199] In this embodiment, the output of the first backbone block is used as the input of the second backbone block, and is added to the output of the second backbone block as the input of the third backbone block. This connection method can make the three backbone blocks interconnected and enhance the propagation and transmission of features.

[0200] The above-mentioned ECAPA-TDNN is a time delay neural network (Time Delay Neural Network), which is characterized by introducing time delay and gating units into the traditional recurrent neural network (RNN) structure.

[0201] The above-mentioned AFMS module is the attention mechanism (Auxiliary Feature Matrix Sum). The main function of the AFMS module is to introduce the attention mechanism in the feature interaction process to learn the influence of different features on the prediction results.

[0202] In order to reduce the risk of overfitting of the model, the output of the first backbone block is subjected to the maximum pooling operation, and then fed into a convolutional layer with batch normalization position together with the outputs of the other two backbone blocks as input. This can reduce the complexity of the model and improve the generalization ability.

[0203] The connection method of the three backbone blocks with residual connections in the present application can effectively extract the feature representation of the speech signal and enhance the robustness and reliability of the model.

[0204] This embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0205] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application can be applied to the fields of artificial intelligence and big data, thereby promoting the construction of smart cities.

[0206] This application belongs to the field of smart cities, and this solution can promote the construction of smart cities.

[0207] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0208] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0209] Further references Fig.10 , as a response to the above Figure 2 The present application provides an embodiment of a voice authentication device, and the voice authentication device embodiment is Figure 2 Corresponding to the embodiment of the voice authentication method shown, the voice authentication device can be specifically applied to various electronic devices.

[0210] like Fig.10 As shown, the speech authentication device 1000 of this embodiment includes: a training module 1001, a classification processing module 1002, an extraction processing module 1003 and an authentication processing module 1004. Among them:

[0211] The training module 1001 is used to obtain a speech data set, and train the pre-trained speech authentication model using the speech data set to obtain a trained speech authentication model;

[0212] The classification processing module 1002 is used to perform vocoder classification processing on the target audio through the trained speech authentication model to determine the corresponding vocoder to which the target audio belongs;

[0213] An extraction processing module 1003 is used to perform feature extraction processing on the target audio in a corresponding vocoder to which the target audio belongs, to obtain target audio features;

[0214] The authentication processing module 1004 is used to perform authentication processing on the target audio features and determine the authentication processing result of the target audio.

[0215] In this embodiment, a speech data set is obtained, a pre-trained speech authentication model is trained by the speech data set to obtain a trained speech authentication model, a vocoder classification process is performed on the target audio by the trained speech authentication model, the vocoder to which the target audio belongs is determined, a feature extraction process is performed on the target audio in the vocoder to which the target audio belongs to obtain the target audio feature, authentication process is performed on the target audio feature, and the authentication processing result of the target audio is determined, thereby solving the problem that the existing speech authentication method has low accuracy and poor generalization in real environments.

[0216] See also Fig.11 ,for Fig.10 The training module 1001 is a schematic diagram of a specific implementation scheme, wherein the training module 1001 includes an acquisition submodule 10011, a generation processing submodule 10012, a first determination submodule 10013, and a second determination submodule 10014.

[0217] An acquisition submodule 10011 is used to acquire sample real audio and different types of vocoders;

[0218] The generation processing submodule 10012 is used to perform audio generation processing on the sample real audio in the vocoders of different categories to obtain generated audio of different categories;

[0219] A first determination submodule 10013 is used to calculate the correlation between the generated audio and the sample real audio, and determine the correlation threshold corresponding to the type of the vocoder, and determine the generated audio with a correlation greater than the correlation threshold as the target generated audio;

[0220] The second determination submodule 10014 is used to determine the label of the target generated audio, and the label of the target generated audio is composed of two parts: the type of the vocoder that generates it and fake.

[0221] In this embodiment, sample real audio and different categories of vocoders are obtained, and the sample real audio is subjected to audio generation processing in different categories of vocoders to obtain different categories of generated audio, and then the correlation between the generated audio and the sample real audio is calculated, and the correlation threshold corresponding to the category of the vocoder is determined. If the correlation is greater than the correlation threshold corresponding to the category of the vocoder, the generated audio is determined as the target generated audio, and the label of the target generated audio is determined, so as to better manage and use the generated audio data and improve the quality and acceptability of the generated audio.

[0222] See also Fig.12 ,for Fig.11 The first determination submodule 10013 is a schematic diagram of a specific implementation scheme, wherein the first determination submodule 10013 includes a first calculation unit 100131, a second calculation unit 100132, a third calculation unit 100133, a fourth calculation unit 100134 and a first determination unit 100135.

[0223] A first calculation unit 100131 is used to calculate the high-pitched similarity between the generated audio and the sample real audio;

[0224] A second calculation unit 100132 is used to calculate the bass similarity between the generated audio and the sample real audio;

[0225] The third calculation unit 100133 is used to calculate the lexical similarity between the generated audio and the sample real audio;

[0226] A fourth calculation unit 100134 is used to calculate the global semantic similarity between the generated audio and the sample real audio;

[0227] The first determination unit 100135 is used to perform weighted fusion processing on the treble similarity, the bass similarity, the lexical similarity and the global semantic similarity to determine the correlation between the generated audio and the sample real audio.

[0228] In this embodiment, by calculating the treble similarity between the generated audio and the sample real audio, calculating the bass similarity between the generated audio and the sample real audio, calculating the lexical similarity between the generated audio and the sample real audio, and calculating the global semantic similarity between the generated audio and the sample real audio, and weightedly fusion processing is performed on the treble similarity, bass similarity, lexical similarity and global semantic similarity to obtain the correlation between the generated audio and the sample real audio, which can help users better evaluate the quality of the generated audio.

[0229] See also Fig.13 ,for Fig.11 The first determination submodule 10013 is a schematic diagram of a specific implementation scheme, wherein the first determination submodule 10013 further includes a second determination unit 100136, a third determination unit 100137, and a fourth determination unit 100138.

[0230] A second determining unit 100136 is used to determine the weight corresponding to the category of the vocoder based on the generation accuracy of the vocoder of different categories, wherein the higher the generation accuracy, the lower the weight;

[0231] The third determining unit 100137 is used to determine the effective duration and the number of effective phonemes in the sample real audio, where the effective duration is the total duration of the effective phonemes, and the effective phonemes are the phonemes of the person;

[0232] The fourth determining unit 100138 is used to perform weighted sum processing on the weight, the effective duration and the number of effective phonemes to obtain the correlation threshold corresponding to the type of the vocoder.

[0233] In this embodiment, based on the generation accuracy of different categories of vocoders, the weights corresponding to the categories of vocoders are determined, the effective duration and the number of effective phonemes in the real audio of the sample are determined, and the weights, effective duration and the number of effective phonemes are weighted and summed to obtain the correlation threshold corresponding to the category of vocoders. The correlation threshold corresponding to the category of vocoders can be determined more accurately, thereby improving work efficiency.

[0234] See also Fig.14 ,for Fig.10 The structure diagram of a specific implementation of the voice authentication device 1000 is shown in FIG. 1 , wherein the voice authentication device 1000 further includes a pre-emphasis processing module 1005, a time-frequency processing module 1006, a processing module 1007 and an output module 1008. Among them:

[0235] A pre-emphasis processing module 1005 is used to obtain a pre-trained speech authentication model, add a vocoder feature extraction module to the pre-trained speech authentication model, and perform pre-emphasis processing on the vocoder features of different categories in the vocoder feature extraction module to obtain the vocoder features after pre-emphasis processing;

[0236] A time-frequency processing module 1006 is used to send the vocoder features after the pre-emphasis processing to an instance normalization layer and a parameterized analysis filter bank to process them into a time-frequency representation, thereby obtaining a vocoder feature time-frequency representation;

[0237] The processing module 1007 is used to pass the characteristic time-frequency representation of the vocoder through three backbone blocks with residual connections, wherein the connection mode of the three backbone blocks with residual connections is that the output of the first backbone block is used as the input of the second backbone block, and the output of the first backbone block and the output of the second backbone block are added as the input of the third backbone block, wherein the output of the first backbone block is pooled, and the output of the first backbone block, the output of the second backbone block, and the output of the third backbone block after the pooling operation are simultaneously sent to the convolution layer as inputs for feature extraction;

[0238] The output module 1008 is used to add the inputs of the three backbone blocks with residual connections, connect a one-dimensional convolutional layer, and two fully connected layers to obtain the output result of the vocoder.

[0239] In this embodiment, a pre-trained speech authentication model is obtained, a vocoder feature extraction module is added to the pre-trained speech authentication model, different categories of vocoder features are pre-emphasized in the vocoder feature extraction module to obtain pre-emphasized vocoder features, the pre-emphasized vocoder features are sent to an instance normalization layer and a parameterized analysis filter group to process them into a time-frequency representation to obtain a vocoder feature time-frequency representation, the vocoder feature time-frequency representation is passed through three backbone blocks with residual connections, the inputs of the three backbone blocks with residual connections are added, and a one-dimensional convolution layer and two fully connected layers are connected to obtain the output result of the vocoder, and the vocoder feature extraction module is trained to ensure that the features extracted by it can accurately capture different statistical characteristics of the vocoder, thereby improving the accuracy and efficiency of the speech processing task.

[0240] See also Fig.15 ,for Fig.10 The structural diagram of a specific implementation of the extraction processing module 1003, the extraction processing module 1003 includes an analysis processing submodule 10031 and an extraction processing submodule 10032.

[0241] The analysis and processing submodule 10031 is used to perform time-frequency analysis on the target audio in the corresponding vocoder to which the target audio belongs, to obtain the target audio after the time-frequency analysis;

[0242] The extraction processing submodule 10032 is used to input the target audio after the time-frequency analysis processing into three backbone blocks with residual connections for feature extraction processing to obtain target audio features.

[0243] In this embodiment, by performing time-frequency analysis on the target audio in the corresponding voice coder to which the target audio belongs, the target audio after time-frequency analysis is obtained, and the target audio after time-frequency analysis is input into three backbone blocks with residual connections for feature extraction processing to obtain the target audio features, which can effectively improve the accuracy and reliability of feature extraction, thereby improving the accuracy and efficiency of speech processing tasks.

[0244] See also Fig.16 ,for Fig.14 A schematic diagram of a specific implementation of the extraction processing submodule 10032, wherein the extraction processing submodule 10032 includes a first feature extraction unit 100321, a second feature extraction unit 100322, a third feature extraction unit 100323 and a fourth feature extraction unit 100324.

[0245] A first feature extraction unit 100321 is used to input the target audio after the time-frequency analysis processing into a first backbone block for first feature extraction processing, and output a first target audio feature to be processed;

[0246] A second feature extraction unit 100322 is used to input the first target audio feature to be processed into a second backbone block for second feature extraction processing, and output a second target audio feature to be processed;

[0247] A third feature extraction unit 100323 is used to input the first target audio feature to be processed and the second target audio feature to be processed into a third backbone block for performing a third feature extraction process to obtain a third target audio feature to be processed;

[0248] The fourth feature extraction unit 100324 is used to input the first target audio feature to be processed, the second target audio feature to be processed, and the third target audio feature to be processed into the convolution layer for fourth feature extraction processing to obtain the target audio feature.

[0249] In this embodiment, the target audio after time-frequency analysis is input into the first backbone block for a first feature extraction process, and the first target audio feature to be processed is output; the first target audio feature to be processed is input into the second backbone block for a second feature extraction process, and the second target audio feature to be processed is output; the first target audio feature to be processed and the second target audio feature to be processed are input into the third backbone block for a third feature extraction process, and the third target audio feature to be processed is obtained; the first target audio feature to be processed, the second target audio feature to be processed, and the third target audio feature to be processed are input into the convolution layer for a fourth feature extraction process to obtain the target audio feature, which can effectively improve the accuracy and reliability of feature extraction.

[0250] To solve the above technical problems, the present application also provides a computer device. Fig.17 , Fig.17 This is a basic structural block diagram of the computer device in this embodiment.

[0251] The computer device 17 includes a memory 171, a processor 172, and a network interface 173 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 17 with components 171-173, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0252] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0253] The memory 171 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc.

[0254] In some embodiments, the memory 171 may be an internal storage unit of the computer device 17, such as a hard disk or memory of the computer device 17. In other embodiments, the memory 171 may also be an external storage device of the computer device 17, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 17. Of course, the memory 171 may also include both the internal storage unit of the computer device 17 and its external storage device. In this embodiment, the memory 171 is generally used to store the operating system and various application software installed on the computer device 17, such as computer-readable instructions of the voice authentication method, etc. In addition, the memory 171 may also be used to temporarily store various types of data that have been output or are to be output. In some embodiments, the processor 172 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 172 is generally used to control the overall operation of the computer device 17. In this embodiment, the processor 172 is used to run computer-readable instructions or process data stored in the memory 171, such as computer-readable instructions for running the voice authentication method.

[0255] The network interface 173 may include a wireless network interface or a wired network interface. The network interface 173 is generally used to establish a communication connection between the computer device 17 and other electronic devices.

[0256] In this embodiment, the impact data of the system in the knowledge base can be used to generate voice authentication data affected by the target system to be tested for the user, so that the user can intuitively obtain the impact range of the target system to be tested. According to the impact range of the target system to be tested, the impact of the change can be evaluated more accurately and quickly, irrelevant interference points can be eliminated, and regression use cases can be streamlined to avoid omissions of impact points in the early stage of the evaluation, resulting in the subsequent development of various systems. When they are jointly debugged, it is found that some associations have not been processed. In the mildest case, it will take up development time to supplement the logic. In the most serious case, the solution design may need to be overturned and redone, thereby improving the efficiency of project development.

[0257] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned voice authentication method.

[0258] In this embodiment, the impact data of the system in the knowledge base can be used to generate voice authentication data affected by the target system to be tested for the user, so that the user can intuitively obtain the impact range of the target system to be tested. According to the impact range of the target system to be tested, the impact of the change can be evaluated more accurately and quickly, irrelevant interference points can be eliminated, and regression use cases can be streamlined to avoid omissions of impact points in the early stage of the evaluation, resulting in the subsequent development of various systems. When they are jointly debugged, it is found that some associations have not been processed. In the mildest case, it will take up development time to supplement the logic. In the most serious case, the solution design may need to be overturned and redone, thereby improving the efficiency of project development.

[0259] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0260] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for detecting fake voice, characterized in that: The steps include: Acquire a speech data set, and train a pre-trained speech authentication model using the speech data set to obtain a trained speech authentication model; Performing vocoder classification processing on the target audio through the trained speech authentication model to determine the corresponding vocoder to which the target audio belongs; Performing feature extraction processing on the target audio in a corresponding vocoder to which the target audio belongs to obtain target audio features; Perform authentication processing on the target audio feature to determine an authentication result of the target audio.

2. The voice authentication method according to claim 1, characterized in that: The speech data set includes target generated audio and annotations of the target generated audio, and the acquiring of the speech data set includes: Get sample real audio and different categories of vocoders; Performing audio generation processing on the sample real audio in the different categories of vocoders to obtain generated audio of different categories; Calculating the correlation between the generated audio and the sample real audio, and determining a correlation threshold corresponding to the type of the vocoder, and determining the generated audio whose correlation is greater than the correlation threshold as the target generated audio; Determine the annotation of the target generated audio, wherein the annotation of the target generated audio is composed of two parts: the type of the vocoder generating the audio and fake.

3. The voice authentication method according to claim 2, characterized in that: The calculating the correlation between the generated audio and the sample real audio comprises: Calculating the high pitch similarity between the generated audio and the sample real audio; Calculating the bass similarity between the generated audio and the sample real audio; Calculating the lexical similarity between the generated audio and the sample real audio; Calculating the global semantic similarity between the generated audio and the sample real audio; The high-pitched similarity, the low-pitched similarity, the lexical similarity and the global semantic similarity are weightedly fused to obtain the correlation between the generated audio and the sample real audio.

4. The voice authentication method according to claim 3, characterized in that: The determining the correlation threshold corresponding to the type of the vocoder includes: Determining weights corresponding to categories of the vocoders based on generation accuracy rates of the vocoders of different categories; Determine the effective duration and the number of effective phonemes in the real audio of the sample, wherein the effective duration is the sum of the durations of the effective phonemes, and the effective phonemes are factors of the person; The weight, the effective duration, and the number of effective phonemes are weighted and summed to obtain the correlation threshold corresponding to the type of the vocoder.

5. The voice authentication method according to claim 4, characterized in that: Before training the pre-trained voice authentication model by using the voice data set to obtain the trained voice authentication model, the method further includes: Obtaining a pre-trained speech authentication model, adding a vocoder feature extraction module to the pre-trained speech authentication model, and performing pre-emphasis processing on the vocoder features of different categories in the vocoder feature extraction module to obtain the vocoder features after the pre-emphasis processing; Sending the pre-emphasized vocoder features to an instance normalization layer and a parameterized analysis filter bank to process them into a time-frequency representation, thereby obtaining a vocoder feature time-frequency representation; The characteristic time-frequency representation of the vocoder passes through three backbone blocks with residual connections, wherein the connection mode of the three backbone blocks with residual connections is that the output of the first backbone block is used as the input of the second backbone block, and the output of the first backbone block and the output of the second backbone block are added as the input of the third backbone block, wherein the output of the first backbone block is pooled, and the output of the first backbone block, the output of the second backbone block, and the output of the third backbone block after the pooling operation are simultaneously sent to the convolution layer as inputs for feature extraction; The inputs of the three backbone blocks with residual connections are added together, and connected to a one-dimensional convolutional layer and two fully connected layers to obtain the output of the vocoder.

6. The voice authentication method according to any one of claims 1 to 5, characterized in that: The target audio is subjected to feature processing in a vocoder corresponding to the target audio to obtain a target audio feature, including: Performing time-frequency analysis processing on the target audio in a corresponding vocoder to which the target audio belongs, to obtain the target audio after the time-frequency analysis processing; The target audio processed by the time-frequency analysis is input into three backbone blocks with residual connections for feature extraction to obtain target audio features.

7. The voice authentication method according to claim 6, characterized in that: The target audio after the time-frequency analysis is input into three backbone blocks with residual connections for feature extraction to obtain target audio features, including: Inputting the target audio after the time-frequency analysis processing into the first backbone block for first feature extraction processing, and outputting a first target audio feature to be processed; Inputting the first target audio feature to be processed into a second backbone block for second feature extraction processing, and outputting a second target audio feature to be processed; Inputting the first target audio feature to be processed and the second target audio feature to be processed into a third backbone block to perform a third feature extraction process to obtain a third target audio feature to be processed; The first target audio feature to be processed, the second target audio feature to be processed, and the third target audio feature to be processed are input into a convolutional layer for fourth feature extraction processing to obtain a target audio feature.

8. A voice authentication device, characterized in that: include: A training module is used to obtain a speech data set, and train a pre-trained speech authentication model using the speech data set to obtain a trained speech authentication model; A classification processing module is used to perform vocoder classification processing on the target audio through the trained speech authentication model to determine the corresponding vocoder to which the target audio belongs; An extraction processing module, used for performing feature extraction processing on the target audio in a corresponding vocoder to which the target audio belongs, to obtain target audio features; The authentication processing module is used to perform authentication processing on the target audio features and determine the authentication processing result of the target audio.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the voice authentication method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the voice authentication method according to any one of claims 1 to 7 are implemented.

Images