Synthetic speech detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a synthetic voice detection method and device, electronic equipment and a storage medium, and relates to the technical field of voice recognition, and the method comprises the steps: carrying out the preprocessing of a voice signal in response to a detection instruction for a received voice signal, and obtaining a target voice signal corresponding to the voice signal; acquiring a fusion model for the target voice signal; inputting the target voice signal into a fusion model for feature extraction to obtain at least two different types of voice features, recognizing the voice signal according to the voice features, and outputting a synthesis probability for the voice signal, the synthesis probability being used for representing a probability that the voice signal is a synthetic voice; and if the synthesis probability is greater than or equal to the preset threshold, determining that the voice signal is the synthesized voice, and outputting an abnormal prompt signal for the synthesized voice, thereby accurately identifying the type of the voice signal based on the fusion model, and ensuring the safety of user data.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a synthetic speech detection method, a synthetic speech detection device, an electronic device and a computer-readable storage medium. Background Art

[0002] With the advancement of speech synthesis technology, its application scope is becoming increasingly wide. From smart assistants to the entertainment industry, synthetic speech has gradually become an indispensable part of people's daily lives. However, the rapid development of this technology has also caused new security issues. Highly realistic synthetic speech may be used maliciously, causing corresponding losses to users or causing serious consequences such as information leakage. However, for speech recognition systems, the need to resist such high-level speech forgery attacks was not fully considered at the beginning of their design. Therefore, in the face of such threats, synthetic speech is often unable to be accurately recognized. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for detecting synthesized speech, so as to solve or partially solve the problem that synthesized speech cannot be accurately recognized.

[0004] The embodiment of the present invention discloses a method for detecting synthesized speech, comprising:

[0005] In response to a detection instruction for a received voice signal, preprocess the voice signal to obtain a target voice signal corresponding to the voice signal;

[0006] Acquire a fusion model for the target speech signal;

[0007] Inputting the target speech signal into the fusion model for feature extraction to obtain at least two different types of speech features, and identifying the speech signal according to the speech features, and outputting a synthesis probability for the speech signal, wherein the synthesis probability is used to characterize the probability that the speech signal is a synthesized speech;

[0008] If the synthesis probability is greater than or equal to a preset threshold, the speech signal is confirmed to be a synthesized speech, and an abnormal prompt signal for the synthesized speech is output.

[0009] In some feasible embodiments, the target speech signal is a spectrogram, and the preprocessing of the speech signal to obtain the target speech signal corresponding to the speech signal includes:

[0010] The speech signal is subjected to a signal sampling rate standardization process, and after the standardization process, the background noise of the speech signal is removed, and then the speech signal is converted into a fixed-length spectrogram.

[0011] In some feasible embodiments, the fusion model is a model combining a convolutional neural network and a long short-term memory network, the speech features include at least local features and time series features, and the target speech signal is input into the fusion model for feature extraction to obtain at least two different types of speech features, including:

[0012] Extracting local features corresponding to the target speech signal through the convolutional neural network;

[0013] The time series features corresponding to the target speech signal are extracted through the long short-term memory network.

[0014] In some feasible embodiments, the local features include at least Mel-frequency cepstral coefficients and linear prediction coding coefficients, and the extracting the local features corresponding to the target speech signal through the convolutional neural network includes:

[0015] Inputting the target speech signal into the convolutional neural network for feature extraction to obtain the Mel frequency cepstral coefficients and the linear prediction coding coefficients;

[0016] The Mel frequency cepstrum coefficient is calculated by the following formula:

[0017]

[0018] Wherein, M(n) represents the energy of the nth frequency point in the target speech signal, N is the total number of frequency points in the target speech signal, and k is the sequence number of the Mel frequency coefficient;

[0019] The linear prediction coding coefficients are determined by the following process:

[0020] The target speech signal is represented as s(n), n=0, 1, ..., N-1;

[0021] Get the prediction order p and define the covariance matrix C, the elements of the covariance matrix C are I, j = 1, 2, ..., p;

[0022] Define a vector d, the elements of which are I = 1, 2, ..., p;

[0023] Using the covariance matrix C and the vector d, construct a corresponding linear equation system Ca=d;

[0024] Where a=[a1,a2,...,a p ] T is the linear predictive coding coefficient.

[0025] In some feasible embodiments, the local features further include multimodal features, and the extracting the local features corresponding to the target speech signal through the convolutional neural network further includes:

[0026] Inputting the target speech signal into the convolutional neural network for time-frequency analysis to obtain corresponding time domain features and frequency domain features;

[0027] Obtaining the video to which the speech signal belongs, and inputting the video into the convolutional neural network for feature extraction, so as to obtain lip movement features corresponding to the lips of the speaker in the video;

[0028] The time domain features, the frequency domain features and the lip movement features are fused to obtain multimodal features corresponding to the target speech signal.

[0029] In some feasible embodiments, the time domain features include at least one of zero crossing rate, energy and zero crossing rate; the frequency domain features include at least one of spectral centroid and spectral bandwidth; the lip movement features include at least one of lip opening size, shape change and movement trajectory of the lip edge.

[0030] In some feasible embodiments, the step of recognizing the speech signal according to the speech feature and outputting a synthesis probability for the speech signal includes:

[0031] The speech signal is identified using the Mel-frequency cepstral coefficients, the linear prediction coding coefficients, and the multimodal features, and a synthesis probability for the speech signal is output.

[0032] The embodiment of the present invention further discloses a synthetic speech detection device, comprising:

[0033] A voice signal processing module, configured to pre-process the voice signal in response to a detection instruction for the received voice signal to obtain a target voice signal corresponding to the voice signal;

[0034] A model acquisition module, used to acquire a fusion model for the target speech signal;

[0035] A prediction module, configured to input the target speech signal into the fusion model for feature extraction, obtain at least two different types of speech features, identify the speech signal according to the speech features, and output a synthesis probability for the speech signal, wherein the synthesis probability is used to characterize the probability that the speech signal is a synthesized speech;

[0036] The prompt module is used to confirm that the speech signal is synthesized speech if the synthesis probability is greater than or equal to a preset threshold, and output an abnormal prompt signal for the synthesized speech.

[0037] In some feasible embodiments, the target speech signal is a spectrogram, and the speech signal processing module is specifically used for:

[0038] The speech signal is subjected to a signal sampling rate standardization process, and after the standardization process, the background noise of the speech signal is removed, and then the speech signal is converted into a fixed-length spectrogram.

[0039] In some feasible embodiments, the fusion model is a model combining a convolutional neural network and a long short-term memory network, the speech features include at least local features and time series features, and the prediction module is specifically used for:

[0040] Extracting local features corresponding to the target speech signal through the convolutional neural network;

[0041] The time series features corresponding to the target speech signal are extracted through the long short-term memory network.

[0042] In some feasible embodiments, the local features include at least Mel-frequency cepstral coefficients and linear prediction coding coefficients, and the prediction module is specifically used for:

[0043] Inputting the target speech signal into the convolutional neural network for feature extraction to obtain the Mel frequency cepstral coefficients and the linear prediction coding coefficients;

[0044] The Mel frequency cepstrum coefficient is calculated by the following formula:

[0045]

[0046] Wherein, M(n) represents the energy of the nth frequency point in the target speech signal, N is the total number of frequency points in the target speech signal, and k is the sequence number of the Mel frequency coefficient;

[0047] The linear prediction coding coefficients are determined by the following process:

[0048] The target speech signal is represented as s(n), n=0, 1, ..., N-1;

[0049] Get the prediction order p and define the covariance matrix C, the elements of the covariance matrix C are I, j = 1, 2, ..., p;

[0050] Define a vector d, the elements of which are I = 1, 2, ..., p;

[0051] Using the covariance matrix C and the vector d, construct a corresponding linear equation system Ca=d;

[0052] Where a=[a1,a2,...,a p ] T is the linear predictive coding coefficient.

[0053] In some feasible embodiments, the local features further include multimodal features, and the prediction module is further configured to:

[0054] Inputting the target speech signal into the convolutional neural network for time-frequency analysis to obtain corresponding time domain features and frequency domain features;

[0055] Obtaining the video to which the speech signal belongs, and inputting the video into the convolutional neural network for feature extraction, so as to obtain lip movement features corresponding to the lips of the speaker in the video;

[0056] The time domain features, the frequency domain features and the lip movement features are fused to obtain multimodal features corresponding to the target speech signal.

[0057] In some feasible embodiments, the time domain features include at least one of zero crossing rate, energy and zero crossing rate; the frequency domain features include at least one of spectral centroid and spectral bandwidth; the lip movement features include at least one of lip opening size, shape change and movement trajectory of the lip edge.

[0058] In some feasible embodiments, the prediction module is specifically used for:

[0059] The speech signal is identified using the Mel-frequency cepstral coefficients, the linear prediction coding coefficients, and the multimodal features, and a synthesis probability for the speech signal is output.

[0060] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0061] The memory is used to store computer programs;

[0062] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0063] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0064] The embodiments of the present invention include the following advantages:

[0065] In an embodiment of the present invention, it can be applied to a voice authentication system. In the process of voice authentication system recognizing voice, in response to a detection instruction for a received voice signal, the voice signal can be preprocessed to obtain a target voice signal corresponding to the voice signal, and then a fusion model for the target voice signal can be obtained. Then, the target voice signal is input into the fusion model for feature extraction to obtain at least two different types of voice features, and the voice signal is recognized according to the voice features, and a synthesis probability for the voice signal is output. The synthesis probability is used to characterize the probability that the voice signal is a synthesized voice. If the synthesis probability is greater than or equal to a preset threshold, the voice signal is confirmed to be a synthesized voice, and an abnormal prompt signal for the synthesized voice is output, so that the voice signal is feature extracted and predicted through the fusion model. Different networks in the fusion model can complement each other, so that the fusion model can analyze the voice signal more comprehensively, thereby improving the accuracy of voice signal recognition. At the same time, the probability of whether the voice signal is a synthesized voice is predicted through the fusion model. Based on the predicted synthesis probability, the voice type to which the voice signal belongs can be accurately identified, thereby effectively enhancing the accuracy of voice recognition, avoiding a series of security issues caused by synthetic voice, and ensuring the security of user data. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of the steps of a method for detecting synthesized speech provided in an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of a synthetic speech detection process provided in an embodiment of the present invention;

[0068] Figure 3 It is a structural block diagram of a synthetic speech detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] As an example, speech recognition systems were not designed to fully protect against such high-level speech forgery attacks. Therefore, when faced with such threats, they are often unable to accurately recognize synthetic speech.

[0071] In this regard, in the present invention, it is applied to a voice authentication system. In the process of voice recognition by the voice authentication system, the voice signal can be preprocessed in response to a detection instruction for the received voice signal to obtain a target voice signal corresponding to the voice signal, and then a fusion model for the target voice signal is obtained. Then, the target voice signal is input into the fusion model for feature extraction to obtain at least two different types of voice features, and the voice signal is recognized according to the voice features, and a synthesis probability for the voice signal is output. The synthesis probability is used to characterize the probability that the voice signal is a synthetic voice. If the synthesis probability is greater than or equal to a preset threshold, the voice signal is confirmed to be a synthetic voice, and an abnormal prompt signal for the synthetic voice is output, so that the voice signal is feature extracted and predicted through the fusion model. Different networks in the fusion model can complement each other, so that the fusion model can analyze the voice signal more comprehensively, thereby improving the accuracy of voice signal recognition. At the same time, the probability of whether the voice signal is a synthetic voice is predicted through the fusion model. Based on the predicted synthesis probability, the voice type to which the voice signal belongs can be accurately identified, thereby effectively enhancing the accuracy of voice recognition, avoiding a series of security issues caused by synthetic voice, and ensuring the security of user data.

[0072] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, some technical features involved in the embodiments of the present invention are explained and illustrated below:

[0073] Large speech models: usually refers to large-scale machine learning models used in speech recognition, speech synthesis or other speech processing tasks. These models are usually based on deep learning techniques, such as convolutional neural networks (CNN), long short-term memory networks (LSTM) or other recurrent neural networks (RNN), and are trained on a large amount of speech data. Such models can learn complex patterns in speech signals and can perform tasks such as converting speech to text (speech recognition) or converting text to speech (speech synthesis).

[0074] Synthesized speech: refers to speech produced by technical means rather than direct human speech. It can be achieved through different technologies, such as rule-based methods (such as concatenated synthesis or parameter synthesis) or machine learning-based methods (such as deep learning). Modern synthetic speech technology can produce very natural and realistic speech, so much so that it is sometimes difficult to distinguish it from real people. This technology is often used in smart assistants, audiobooks, automatic customer service systems and other fields. However, due to its high degree of realism, synthetic speech may cause corresponding losses to users or cause serious consequences such as information leakage. Therefore, the technology to detect synthetic speech and distinguish it from real speech is becoming increasingly important.

[0075] Reference Figure 1 , shows a flowchart of a method for detecting synthesized speech provided in an embodiment of the present invention, which may specifically include the following steps:

[0076] Step 101, in response to a detection instruction for a received voice signal, preprocess the voice signal to obtain a target voice signal corresponding to the voice signal;

[0077] Optionally, the recognition process of synthesized speech can be applied to systems such as voice authentication systems (hereinafter referred to as systems) that require user speech recognition. By recognizing the speech, it is determined whether the user is a "real person" and whether the speech is consistent with the user, etc., so as to reduce the impact of synthesized speech on the user.

[0078] In the embodiment of the present invention, when the system receives the corresponding voice signal, it can generate a detection instruction for the voice information, and detect the voice signal based on the detection instruction to determine whether the voice signal is a synthesized voice. Before the voice signal is formally detected, the voice signal can be preprocessed to reduce the noise in the voice signal and improve the accuracy of voice signal recognition.

[0079] In a specific implementation, the speech signal can be first subjected to signal sampling rate standardization processing, and after the standardization processing, the background noise of the speech signal can be removed, and then the speech signal can be converted into a spectrogram of a fixed length, thereby reducing the noise in the speech signal and improving the accuracy of speech signal recognition by processing the speech signal.

[0080] In one example, the original speech signal can be first subjected to sampling rate standardization, and by selecting a standard sampling rate (such as 16kHz or 8kHz), the speech signals of different sampling rates are unified to the standard sampling rate, so as to be achieved by resampling technology, ensuring that all speech signals have the same sampling rate. After the standardization process, the background noise of the speech signal is estimated, for example, the background noise can be estimated by short-time Fourier transform (STFT, Short-Time Fourier Transform) and noise spectrum estimation. Then, a noise elimination algorithm (such as spectral subtraction, Wiener filtering, noise reduction model based on deep learning, etc.) can be used to remove the background noise and enhance the clarity of the speech signal, so that the quality of the speech signal can be significantly improved by removing the background noise, and the interference of noise on subsequent speech recognition can be reduced. Next, the speech signal can be framed, usually 20ms to 30ms per frame, with an overlap rate of 50%, and then each frame is subjected to short-time Fourier transform to obtain a spectrogram. Furthermore, in order to ensure that the spectrograms of all speech signals have the same representation, the spectrograms can be standardized (such as mean variance normalization, etc.), and spectrograms of different lengths can be converted into spectrograms of fixed length through alignment techniques (such as dynamic time warping (DTW)), so as to effectively capture the frequency domain and time domain characteristics of the speech signal, which is convenient for subsequent speech recognition algorithm processing.

[0081] Through the above process, based on the steps of standardization, denoising, feature extraction, etc. of the speech signal, the noise in the speech signal can be significantly reduced, the overall quality of the speech signal can be improved, and the processed speech signal can have higher clarity and consistency, thereby improving the accuracy of the speech recognition model and reducing the possibility of misrecognition.

[0082] Step 102, obtaining a fusion model for the target speech signal;

[0083] For the pre-processed target speech signal, the target speech signal can be identified by the pre-trained fusion model. The fusion model can be used to predict the synthesis probability of whether the target speech signal is a synthesized speech. The higher the synthesis probability, the higher the possibility that the speech corresponding to the speech signal is a synthesized speech; conversely, the lower the possibility that the speech corresponding to the speech signal is a synthesized speech.

[0084] Step 103, inputting the target speech signal into the fusion model for feature extraction to obtain at least two different types of speech features, and identifying the speech signal according to the speech features, and outputting a synthesis probability for the speech signal, wherein the synthesis probability is used to characterize the probability that the speech signal is a synthesized speech;

[0085] In an embodiment of the present invention, by inputting the target speech signal into the fusion model for feature extraction, after obtaining at least two different types of speech features, the speech signal can be recognized according to the at least two different speech features, and the corresponding synthesis probability can be output based on the recognition result, so that the system can judge whether the speech is synthesized speech based on the synthesis probability, thereby performing feature extraction and prediction on the speech signal through the fusion model. The different networks in the fusion model can complement each other, so that the fusion model can analyze the speech signal more comprehensively, thereby improving the accuracy of speech signal recognition. At the same time, the probability of whether the speech signal is synthesized speech is predicted through the fusion model. Based on the predicted synthesis probability, the speech type to which the speech signal belongs can be accurately identified, thereby effectively enhancing the accuracy of speech recognition, avoiding a series of security issues caused by synthetic speech, and ensuring the security of user data.

[0086] Among them, the fusion model can at least be a model combining a convolutional neural network and a long short-term memory network. The convolutional neural network can be used to extract local features of speech signals, and the long short-term memory network can be used to extract time series features of speech signals. Specifically, by inputting the target speech signal into the fusion model, the local features corresponding to the target speech signal are extracted by the convolutional neural network, and the time series features corresponding to the target speech signal are extracted by the long short-term memory network. Then, the speech signal can be identified according to the local features and the time series features, and the synthesis probability corresponding to the speech signal can be determined.

[0087] It should be noted that, for a convolutional neural network, a structure including at least two convolutional layers can be constructed, each convolutional layer is followed by a maximum pooling layer for extracting local features; for a long short-term memory network model, it can include at least two layers of long short-term memory network units for capturing the time series characteristics of speech signals.

[0088] Optionally, the convolutional neural network and long short-term memory network models are pre-trained using a dataset containing at least one million speech clips, each of which is manually annotated to determine its authenticity.

[0089] Optionally, the training process of the convolutional neural network and the long short-term memory network model includes pre-training and fine-tuning. The pre-training stage uses a large amount of unlabeled speech data for unsupervised learning, and the fine-tuning stage uses labeled synthetic speech and real speech data for supervised learning.

[0090] Optionally, the dataset used for training the convolutional neural network and long short-term memory network models includes at least 10,000 hours of natural speech data and 10,000 hours of synthetic speech data, ensuring the diversity and robustness of the model training.

[0091] In a specific implementation, the local features may include at least Mel-frequency cepstral coefficients and linear prediction coding coefficients, and the target speech signal may be input into a convolutional neural network for feature extraction to obtain Mel-frequency cepstral coefficients and linear prediction coding coefficients;

[0092] Among them, the Mel-Frequency Cepstral Coefficients (MFCC) are calculated by the following formula:

[0093]

[0094] Wherein, M(n) represents the energy of the nth frequency point in the target speech signal, N is the total number of frequency points in the target speech signal, and k is the serial number of the Mel frequency coefficient;

[0095] The linear prediction coding coefficients are determined by the following process:

[0096] The target speech signal is represented as s(n), n=0, 1, ..., N-1;

[0097] Get the prediction order p and define the covariance matrix C. The elements of the covariance matrix C I, j = 1, 2, ..., p;

[0098] Define vector d, the elements of vector d I = 1, 2, ..., p;

[0099] Using the covariance matrix C and the vector d, the corresponding linear equation system Ca=d is constructed;

[0100] Where a=[a1,a2,...,a p ] T are the linear predictive coding coefficients.

[0101] In addition, when extracting local features corresponding to speech signals, convolutional neural networks can also use a multi-default feature fusion mechanism to extract multimodal features corresponding to speech signals. Specifically, the target speech signal can be input into the convolutional neural network for time-frequency analysis to obtain the corresponding time domain features and frequency domain features. At the same time, the video to which the speech signal belongs is obtained, and the video is input into the convolutional neural network for feature extraction to obtain the lip movement features corresponding to the lips of the speaker in the video. Finally, the time domain features, frequency domain features and lip movement features are fused to obtain the multimodal features corresponding to the target speech signal. By introducing the multimodal feature fusion mechanism, the time domain features, frequency domain features of the speech signal and the lip movement features extracted from the video source are combined to further improve the accuracy of distinguishing synthetic speech from real human voices. Even the most advanced synthetic speech technology is difficult to perfectly replicate the speaker's lip movements. Therefore, the fusion of these different dimensions of information can more accurately judge the authenticity of the speech, which not only improves the accuracy of detection, but also greatly reduces the false alarm rate. The multimodal fusion mechanism ensures that the system can still operate reliably in complex environments, thereby improving the user experience. Users can use the speech recognition function more safely and no longer worry about security issues.

[0102] Among them, the time domain features include at least one of zero crossing rate, energy and zero crossing rate; the frequency domain features include at least one of spectral centroid and spectral bandwidth; the lip movement features include at least one of lip opening size, shape change and movement trajectory of lip edge. It should be noted that the Mel frequency cepstral coefficients can also belong to the part of frequency domain features, and the extraction process of Mel frequency cepstral coefficients can refer to the description in the above embodiment, which will not be repeated here.

[0103] Through the above process, after the fusion model extracts the Mel-frequency cepstral coefficients, linear prediction coding coefficients and multimodal features corresponding to the target speech signal, the Mel-frequency cepstral coefficients, linear prediction coding coefficients and multimodal features can be used to identify the speech signal and output the synthesis probability for the speech signal. Thus, through the deep learning model combining the convolutional neural network and the long short-term memory network, the convolutional neural network is specifically used to extract the local features of the speech signal, while the long short-term memory network captures the time series features of the speech signal. The two complement each other and enable the model to analyze the speech samples more comprehensively. By training on large-scale real and synthetic speech data sets, the model is ensured to have a high degree of detection accuracy and robustness, which not only enhances the security of the speech authentication system, prevents unauthorized access, and ensures the security of user data, but also solves the potential risks brought by synthetic speech attacks, ensuring that service providers can provide users with more secure and reliable speech authentication services.

[0104] Step 104: If the synthesis probability is greater than or equal to a preset threshold, the speech signal is confirmed to be a synthesized speech, and an abnormal prompt signal for the synthesized speech is output.

[0105] In an embodiment of the present invention, a corresponding detection threshold can be pre-set for the synthesis probability. When the fusion model recognizes the speech signal and obtains the corresponding synthesis probability, the synthesis probability can be compared with the detection threshold to determine whether the speech signal is synthesized speech. Specifically, if the synthesis probability is greater than or equal to the preset threshold, the speech signal is confirmed to be synthesized speech, and an abnormal prompt signal for the synthesized speech is output; if the synthesis probability is less than the preset threshold, the speech signal is determined to be normal speech, so that the fusion model is used to extract and predict features of the speech signal. Different networks in the fusion model can complement each other, so that the fusion model can analyze the speech signal more comprehensively, thereby improving the accuracy of speech signal recognition. At the same time, the fusion model is used to predict the probability of whether the speech signal is synthesized speech. Based on the predicted synthesis probability, the speech type to which the speech signal belongs can be accurately identified, thereby effectively enhancing the accuracy of speech recognition, avoiding a series of security issues caused by synthetic speech, and ensuring the security of user data.

[0106] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0107] In an embodiment of the present invention, it can be applied to a voice authentication system. In the process of voice authentication system recognizing voice, in response to a detection instruction for a received voice signal, the voice signal can be preprocessed to obtain a target voice signal corresponding to the voice signal, and then a fusion model for the target voice signal can be obtained. Then, the target voice signal is input into the fusion model for feature extraction to obtain at least two different types of voice features, and the voice signal is recognized according to the voice features, and a synthesis probability for the voice signal is output. The synthesis probability is used to characterize the probability that the voice signal is a synthesized voice. If the synthesis probability is greater than or equal to a preset threshold, the voice signal is confirmed to be a synthesized voice, and an abnormal prompt signal for the synthesized voice is output, so that the voice signal is feature extracted and predicted through the fusion model. Different networks in the fusion model can complement each other, so that the fusion model can analyze the voice signal more comprehensively, thereby improving the accuracy of voice signal recognition. At the same time, the probability of whether the voice signal is a synthesized voice is predicted through the fusion model. Based on the predicted synthesis probability, the voice type to which the voice signal belongs can be accurately identified, thereby effectively enhancing the accuracy of voice recognition, avoiding a series of security issues caused by synthetic voice, and ensuring the security of user data.

[0108] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0109] As an example, see Figure 2 , shows a schematic diagram of the detection process of the synthesized speech provided in an embodiment of the present invention, which may specifically include the following processes:

[0110] S1, preprocessing the received speech signal to be detected, wherein the preprocessing includes signal sampling rate standardization, background noise removal, and converting the speech signal into a fixed-length spectrogram;

[0111] S2, extracting at least two types of features from the preprocessed speech signal, wherein at least one feature is a Mel frequency cepstral coefficient and the other feature is a linear prediction coding coefficient;

[0112] S3, using a model combining a convolutional neural network and a long short-term memory network to process the features, wherein the convolutional neural network is used to extract local features of the speech signal, and the long short-term memory network is used to capture time series features of the speech signal;

[0113] S4. According to the model output result, determining whether the speech signal is a synthesized speech by a set threshold value, wherein the threshold value is set by optimizing the balance between the true positive rate and the false positive rate;

[0114] S5. When the detection result shows that the speech is synthesized speech, an alarm signal is triggered.

[0115] In the specific implementation, the specific parameters of each layer of the convolutional neural network (CNN) and long short-term memory network (LSTM) model are set as follows:

[0116] Convolutional layer 1: The convolution kernel size is 3x3, the stride is 1, and the padding is 'same' to ensure that the output feature map has the same size as the input feature map. The activation function uses ReLU to increase the nonlinear feature expression capability.

[0117] Max pooling layer 1: The pooling window size is 2x2 and the step size is 2, which is used to reduce the spatial dimension of the feature map and reduce the amount of calculation;

[0118] Convolutional layer 2: The convolution kernel size is 5x5, the stride is 1, the padding is 'same', the feature map size remains unchanged, and ReLU is also used as the activation function;

[0119] Max pooling layer 2: The pooling window size is 2x2 and the step size is 2, which further compresses the feature map and enhances the translation invariance of the model;

[0120] LSTM layer 1: The number of hidden units is 256 to capture the time series characteristics of the speech signal;

[0121] LSTM layer 2: The number of hidden units is also 256, which further enhances the ability to capture time series features.

[0122] Training optimization: Use the Adam optimizer, the initial learning rate is 0.001, and adopt an exponential decay strategy, decaying once every 5 epochs, with a decay rate of 0.9, and the loss function uses the binary cross entropy loss function;

[0123] When actually deployed, the convolutional neural network (CNN) and long short-term memory network (LSTM) models must meet the requirement of processing at least 20 seconds of audio per second to ensure real-time performance; the model size should be controlled within 50MB to meet the storage requirements of mobile devices; in addition, the model's inference time on the CPU should not exceed 100 milliseconds to ensure fast response.

[0124] Specifically, in order to ensure that the synthetic speech detection model based on convolutional neural network (CNN) and long short-term memory network (LSTM) has good generalization ability and robustness, a diverse dataset was designed. The dataset contains at least one million speech clips from different genders (male, female), age groups (teenagers to the elderly), languages ​​(Mandarin, English, French, etc.) and dialects (various local dialects). Each clip has been strictly manually annotated to confirm its authenticity;

[0125] Data preprocessing includes unifying all audio signals to a sampling rate of 16KHz, using spectral subtraction to remove background noise, and converting speech signals into fixed-length Mel-frequency cepstral coefficient (MFCC) spectrograms for model processing;

[0126] The dataset is divided into training set, validation set and test set in the ratio of 70%, 15% and 15%. During the division process, it is ensured that the speech clips in the three sets are evenly distributed in terms of gender, age, language and dialect to avoid overfitting the model to the voices of a specific group.

[0127] Manual annotation follows strict annotation guidelines. All clips are initially marked as real or synthetic by professionals, and then reviewed by another expert. Any controversial annotations will be submitted to a third senior expert for final decision. In order to ensure the quality of annotation, annotators are regularly trained, and sampling inspections are used to monitor the consistency and accuracy of annotations.

[0128] During model training, accuracy, recall, and F1 score are used as the main evaluation indicators. By monitoring the changing trends of these indicators in real time, the model's hyperparameters, such as learning rate and batch size, are adjusted in a timely manner. In addition, the model structure is adjusted according to the performance of the validation set to ensure that the model's performance on the test set is optimal.

[0129] Specifically, in the feature extraction stage, in addition to calculating the Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC) coefficients, the following steps are also included:

[0130] Step 1: First, perform time-frequency analysis on the input speech signal to extract time domain features and frequency domain features. Time domain features include zero-crossing rate, energy, root mean square, etc. These features can reflect the instantaneous characteristics of the sound. Frequency domain features include power spectrum density, spectrum centroid, spectrum bandwidth, etc. These features can reveal the spectral characteristics of the speech signal.

[0131] Step 2: Next, extract the speaker's lip movement features from the synchronously recorded video, use facial key point detection technology to locate the speaker's lip area, track the changes of lip contour points over time, and calculate the position difference of contour points between adjacent frames to obtain the speed and acceleration of lip movement as motion features;

[0132] Step 3: Finally, a multimodal fusion network is designed by integrating time domain features, frequency domain features, and lip movement features through a fusion algorithm. The network contains three input branches, which process time domain features, frequency domain features, and lip movement features respectively. Each branch extracts high-level features through independent convolutional layers and fully connected layers, and then the features of the three branches are fused together through a shared fully connected layer. The fused features are used by subsequent classifiers to make the final classification decision between synthetic speech and real human voice.

[0133] Through the deep learning model that combines convolutional neural networks with long short-term memory networks, the convolutional neural network is specifically used to extract local features of speech signals, while the long short-term memory network captures the time series features of speech signals. The two complement each other, enabling the model to analyze speech samples more comprehensively. By training on large-scale real and synthetic speech data sets, the model is ensured to have high detection accuracy and robustness. This method not only enhances the security of the voice authentication system, prevents unauthorized access, and ensures the security of user data, but also solves the potential risks brought by synthetic speech attacks, ensuring that service providers can provide users with more secure and reliable voice authentication services.

[0134] By introducing a multimodal feature fusion mechanism, the time domain features, frequency domain features of the speech signal and the lip movement features extracted from the video source are combined to further improve the accuracy of distinguishing between synthetic speech and real human voice. Even the most advanced synthetic speech technology finds it difficult to perfectly replicate the speaker's lip movements. Therefore, the fusion of these different dimensions of information can more accurately judge the authenticity of the speech, which not only improves the accuracy of detection, but also greatly reduces the false alarm rate. The multimodal fusion mechanism ensures that the system can still operate reliably in complex environments, thereby improving the user experience. Users can use the speech recognition function with greater peace of mind and no longer worry about security issues.

[0135] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0136] Reference Figure 3 , shows a structural block diagram of a synthetic speech detection device provided in an embodiment of the present invention, which may specifically include the following modules:

[0137] The speech signal processing module 301 is used to pre-process the speech signal in response to a detection instruction for the received speech signal to obtain a target speech signal corresponding to the speech signal;

[0138] A model acquisition module 302, used to acquire a fusion model for the target speech signal;

[0139] The prediction module 303 is used to input the target speech signal into the fusion model for feature extraction, obtain at least two different types of speech features, identify the speech signal according to the speech features, and output a synthesis probability for the speech signal, where the synthesis probability is used to characterize the probability that the speech signal is a synthesized speech;

[0140] The prompt module 304 is used to confirm that the speech signal is a synthesized speech if the synthesis probability is greater than or equal to a preset threshold, and output an abnormal prompt signal for the synthesized speech.

[0141] In some feasible embodiments, the target speech signal is a spectrogram, and the speech signal processing module 301 is specifically used for:

[0142] The speech signal is subjected to a signal sampling rate standardization process, and after the standardization process, the background noise of the speech signal is removed, and then the speech signal is converted into a fixed-length spectrogram.

[0143] In some feasible embodiments, the fusion model is a model combining a convolutional neural network and a long short-term memory network, the speech features include at least local features and time series features, and the prediction module 303 is specifically used for:

[0144] Extracting local features corresponding to the target speech signal through the convolutional neural network;

[0145] The time series features corresponding to the target speech signal are extracted through the long short-term memory network.

[0146] In some feasible embodiments, the local features include at least Mel-frequency cepstral coefficients and linear prediction coding coefficients, and the prediction module 303 is specifically used for:

[0147] Inputting the target speech signal into the convolutional neural network for feature extraction to obtain the Mel frequency cepstral coefficients and the linear prediction coding coefficients;

[0148] The Mel frequency cepstrum coefficient is calculated by the following formula:

[0149]

[0150] Wherein, M(n) represents the energy of the nth frequency point in the target speech signal, N is the total number of frequency points in the target speech signal, and k is the sequence number of the Mel frequency coefficient;

[0151] The linear prediction coding coefficients are determined by the following process:

[0152] The target speech signal is represented as s(n), n=0, 1, ..., N-1;

[0153] Get the prediction order p and define the covariance matrix C, the elements of the covariance matrix C are I, j = 1, 2, ..., p;

[0154] Define a vector d, the elements of which are I = 1, 2, ..., p;

[0155] Using the covariance matrix C and the vector d, construct a corresponding linear equation system Ca=d;

[0156] Where a=[a1,a2,...,a p ] Tis the linear predictive coding coefficient.

[0157] In some feasible embodiments, the local features further include multimodal features, and the prediction module 303 is further configured to:

[0158] Inputting the target speech signal into the convolutional neural network for time-frequency analysis to obtain corresponding time domain features and frequency domain features;

[0159] Obtaining the video to which the speech signal belongs, and inputting the video into the convolutional neural network for feature extraction, so as to obtain lip movement features corresponding to the lips of the speaker in the video;

[0160] The time domain features, the frequency domain features and the lip movement features are fused to obtain multimodal features corresponding to the target speech signal.

[0161] In some feasible embodiments, the time domain features include at least one of zero crossing rate, energy and zero crossing rate; the frequency domain features include at least one of spectral centroid and spectral bandwidth; the lip movement features include at least one of lip opening size, shape change and movement trajectory of the lip edge.

[0162] In some feasible embodiments, the prediction module 303 is specifically used for:

[0163] The speech signal is identified using the Mel-frequency cepstral coefficients, the linear prediction coding coefficients, and the multimodal features, and a synthesis probability for the speech signal is output.

[0164] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] In addition, an embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned synthetic speech detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0166] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned synthetic speech detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0167] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0168] It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program codes.

[0169] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0173] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0174] The above is a detailed introduction to a method for detecting a synthetic speech and a device for detecting a synthetic speech provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting synthetic speech, characterized in that: include: In response to a detection instruction for a received voice signal, preprocess the voice signal to obtain a target voice signal corresponding to the voice signal; Acquire a fusion model for the target speech signal; Inputting the target speech signal into the fusion model for feature extraction to obtain at least two different types of speech features, and identifying the speech signal according to the speech features, and outputting a synthesis probability for the speech signal, wherein the synthesis probability is used to characterize the probability that the speech signal is a synthesized speech; If the synthesis probability is greater than or equal to a preset threshold, the speech signal is confirmed to be a synthesized speech, and an abnormal prompt signal for the synthesized speech is output.

2. The method according to claim 1, characterized in that The target speech signal is a spectrogram, and the preprocessing of the speech signal to obtain the target speech signal corresponding to the speech signal includes: The speech signal is subjected to a signal sampling rate standardization process, and after the standardization process, the background noise of the speech signal is removed, and then the speech signal is converted into a fixed-length spectrogram.

3. The method according to claim 1 or 2, characterized in that: The fusion model is a model combining a convolutional neural network and a long short-term memory network, the speech features include at least local features and time series features, and the target speech signal is input into the fusion model for feature extraction to obtain at least two different types of speech features, including: Extracting local features corresponding to the target speech signal through the convolutional neural network; The time series features corresponding to the target speech signal are extracted through the long short-term memory network.

4. The method according to claim 3, characterized in that The local features include at least Mel-frequency cepstral coefficients and linear prediction coding coefficients, and the extracting the local features corresponding to the target speech signal by the convolutional neural network includes: Inputting the target speech signal into the convolutional neural network for feature extraction to obtain the Mel frequency cepstral coefficients and the linear prediction coding coefficients; The Mel frequency cepstrum coefficient is calculated by the following formula: Wherein, M(n) represents the energy of the nth frequency point in the target speech signal, N is the total number of frequency points in the target speech signal, and k is the sequence number of the Mel frequency coefficient; The linear prediction coding coefficients are determined by the following process: The target speech signal is represented as s(n), n=0, 1, ..., N-1; Get the prediction order p and define the covariance matrix C, the elements of the covariance matrix C are I, j = 1, 2, ..., p; Define a vector d, the elements of which are I = 1, 2, ..., p; Using the covariance matrix C and the vector d, construct a corresponding linear equation system Ca=d; Where a=[a1,a2,...,a p ] T is the linear predictive coding coefficient.

5. The method according to claim 3, characterized in that: The local features also include multimodal features, and the extracting the local features corresponding to the target speech signal through the convolutional neural network also includes: Inputting the target speech signal into the convolutional neural network for time-frequency analysis to obtain corresponding time domain features and frequency domain features; Obtaining the video to which the speech signal belongs, and inputting the video into the convolutional neural network for feature extraction, so as to obtain lip movement features corresponding to the lips of the speaker in the video; The time domain features, the frequency domain features and the lip movement features are fused to obtain multimodal features corresponding to the target speech signal.

6. The method according to claim 5, characterized in that The time domain features include at least one of zero crossing rate, energy and zero crossing rate; the frequency domain features include at least one of spectral centroid and spectral bandwidth; the lip movement features include at least one of lip opening size, shape change and movement trajectory of the lip edge.

7. The method according to claim 5, characterized in that The step of recognizing the speech signal according to the speech feature and outputting a synthesis probability for the speech signal includes: The speech signal is identified using the Mel-frequency cepstral coefficients, the linear prediction coding coefficients, and the multimodal features, and a synthesis probability for the speech signal is output.

8. A synthetic speech detection device, characterized in that: include: A voice signal processing module, configured to pre-process the voice signal in response to a detection instruction for the received voice signal to obtain a target voice signal corresponding to the voice signal; A model acquisition module, used to acquire a fusion model for the target speech signal; A prediction module, configured to input the target speech signal into the fusion model for feature extraction, obtain at least two different types of speech features, identify the speech signal according to the speech features, and output a synthesis probability for the speech signal, wherein the synthesis probability is used to characterize the probability that the speech signal is a synthesized speech; The prompt module is used to confirm that the speech signal is synthesized speech if the synthesis probability is greater than or equal to a preset threshold, and output an abnormal prompt signal for the synthesized speech.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images