A method, device, and storage medium for detecting synthetic speech based on background noise

Abstract

The present invention discloses a method, apparatus and storage medium for detecting synthetic speech based on background noise, belonging to the technical field of deep learning. The method includes: using an encoder to extract a sample encoding containing composite features from the original audio; inputting the sample encoding into RVQs for feature extraction, using a latent element model to guide VQ1 to extract semantic features in the sample encoding, and stripping the semantic features from the composite features through the residual structure of RVQs to obtain pure acoustic features; using a natural language 3-generation model to guide VQ2 and VQ3 to extract prosodic features and speaker information in the pure acoustic features respectively, and stripping the prosodic features and speaker information from the pure acoustic features to obtain pure background noise information; inputting the background noise information into a detection model for speech detection to determine whether the original audio is synthetic audio. The detection efficiency and accuracy are improved, and the balance between privacy protection and data security is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and more specifically, relates to a synthetic speech detection method, device and storage medium based on background noise. Background Art

[0002] Deep learning is an important branch in the field of artificial intelligence. The core idea of this technology is to simulate human intelligence, interpret data, classify data, discover potential laws, etc. through multi-layer neural networks. Deep learning technology can automatically learn complex patterns and features from large-scale data and has been widely applied in multiple fields, such as the field of computer images, natural language processing, autonomous driving, medical image analysis, speech recognition, etc. In synthetic speech detection, deep learning is used to analyze and identify subtle differences in audio. These differences may be difficult for humans to distinguish, but can indicate whether the audio is synthesized or tampered with, thereby preventing forged content from being used for improper purposes and avoiding posing a security threat to society and individuals.

[0003] In the research of existing synthetic speech detection technologies, it usually focuses on the analysis of audio waveforms or spectral features, relying on extracting low-level features or high-level speech attributes from speech signals, and requires access to complete speech information. For example, a detection scheme based on deep ResNet proposed by Alzantot et al. performs score fusion for three different front-end features of MFCC, spectrogram, and CQCC, and trains a model to identify subtle features of synthetic speech for deepfake detection. Although these detection technologies are efficient and practical in preventing audio deepfakes, there may be a risk of exposing private speech content, especially in scenarios involving user privacy such as trade secrets and medical conditions, thus hindering the application of deepfake detection technology. Some detection technologies attempt to develop an audio deepfake detection framework for content privacy protection. Xinfeng Li et al. proposed a new framework SafeEar for detecting without accessing audio content. It designs a neural audio codec, focuses on extracting speaker-related features, and avoids exposing content privacy by only analyzing the acoustic information of speech without relying on semantic content.

[0004] However, these detection technologies for protecting audio content privacy may still leak user identities due to individual physiological characteristics such as accents, intonations, emotions, and timbres reflected in the acoustic information, affecting the balance between efficient detection and privacy protection and making it difficult to meet the growing detection requirements. Summary of the Invention

[0005] Aiming at the deficiencies of the related technologies, the purpose of the present invention is to provide a synthetic speech detection method, device and storage medium based on background noise, aiming to solve the technical problems that the existing synthetic speech detection methods lack content privacy protection and user feature protection for the detected audio.

[0006] To achieve the above object, the present invention provides a method for detecting synthetic speech based on background noise, including:

[0007] S1. An encoder is used to extract a sample encoding containing composite features from the original audio, and the composite features include semantic features and acoustic features;

[0008] S2. The sample encoding is input into a residual vector quantization model RVQs for feature extraction. A latent model is used to guide the first-layer quantization output VQ1 of RVQs to extract the semantic features in the sample encoding, and the semantic features are stripped from the composite features through the residual structure of RVQs to obtain pure acoustic features;

[0009] S3. A natural language generation model is used to guide the second quantization output VQ2 and the third-layer quantization output VQ3 of RVQs to extract the prosodic features and speaker information in the pure acoustic features respectively, and the prosodic features and speaker information are stripped from the pure acoustic features through the residual structure of RVQs to obtain pure background noise information;

[0010] S4. The background noise information is input into a detection model for speech detection to determine whether the original audio is synthetic audio.

[0011] Optionally, the encoder includes a one-dimensional convolutional layer, a residual block, a downsampling layer, an LSTM layer, and a convolutional layer;

[0012] The one-dimensional convolutional layer is used to extract features from the original audio signal, convert the original audio signal into an array form, and obtain preliminary features;

[0013] The residual block, downsampling layer, and LSTM layer are used to extract information across the audio feature space from the preliminary features and output composite features containing semantic features and acoustic features;

[0014] The convolutional layer is used to convert the dimension of the composite features into the dimension of the quantization input RVQ to obtain a sample encoding containing composite features.

[0015] Optionally, step S2 specifically includes:

[0016] S2.1. Map the output dimension of VQ1 to the same dimension as the semantic features extracted by the latent model;

[0017] S2.2. Use the average representation of all latent model layers as a semantic supervision signal, and supervise VQ1 to learn the content representation in the sample encoding through a loss function, extract semantic features, and obtain corresponding semantic tags; the loss function is:

[0018]

[0019] Among them, represents the quantization output of the VQ layer, represents the time step of the supervision signal, represents the cosine similarity, represents the sigmoid activation function, represents the projection matrix;

[0020] S2.3. Use the residual structure of RVQs to subtract the semantic features corresponding to the semantic tokens from the composite features to obtain pure acoustic features.

[0021] Optionally, step S3 specifically includes:

[0022] S3.1. Map the output dimension of VQ2 to the same dimension as the prosody features extracted by the natural language 3-generation model, and map the output dimension of VQ3 to the same dimension as the speaker features extracted by the natural language 3-generation model;

[0023] S3.2. Use the natural language 3-generation model to extract the prosody features and speaker information in the pure acoustic features from the second quantization output VQ2 and the third-layer quantization output VQ3 respectively, and perform supervised learning on VQ2 and VQ3 through a loss function to obtain the prosody token P and the speaker token ;

[0024] S3.3. Use the residual structure of RVQs to strip the prosody features and speaker information corresponding to the prosody token P and the speaker token from the pure acoustic features to obtain pure background noise information.

[0025] Optionally, after step S3, it further includes:

[0026] Transmit the pure background noise information to the remaining five quantizers of RVQs in sequence, and each quantizer refines the background noise information in sequence to enhance its feature representation.

[0027] Optionally, the detection model includes a bottleneck layer, a Transformer classifier, and an output layer;

[0028] The bottleneck layer is used to reduce the dimension of the obtained background noise features through one-dimensional convolution and batch normalization;

[0029] The Transformer classifier is used to pass the compressed background noise features through multiple attention heads to capture different long-range dependencies and dynamic spatial weighting information, and aggregate the output features of each attention head to obtain a unified attention spectrum;

[0030] The output layer is used to compare the attention spectrum with the attention spectrum corresponding to the background noise of the learned real audio and the attention spectrum corresponding to the background noise of the synthesized audio, and determine whether the original audio is synthesized audio.

[0031] In a second aspect, the present invention further provides a synthesized speech detection device based on background noise, which is used to execute the synthesized speech detection method described in any item of the first aspect, including:

[0032] A data preprocessing module, which is used to extract a sample encoding containing composite features from the original audio by using an encoder, and the composite features include semantic features and acoustic features;

[0033] An acoustic feature extraction module, which is used to input the sample encoding into a residual vector quantization model RVQs for feature extraction, use a latent element model to guide the first-layer quantization output VQ1 of RVQs to extract the semantic features in the sample encoding, and strip the semantic features from the composite features through the residual structure of RVQs to obtain pure acoustic features;

[0034] A background feature extraction module, which is used to use a natural language 3-generation model to guide the second quantization output VQ2 and the third-layer quantization output VQ3 of RVQs to extract the prosody features and speaker information in the pure acoustic features respectively, and strip the prosody features and speaker information from the pure acoustic features through the residual structure of RVQs to obtain pure background noise information;

[0035] A detection module, which is used to input the background noise information into a detection model for speech detection and determine whether the original audio is synthesized audio.

[0036] In a third aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the synthesized speech detection method described in any item of the first aspect.

[0037] Through the above technical solutions conceived by the present invention, compared with the prior art, the following beneficial effects can be achieved:

[0038] 1. The embodiment of the present invention provides a synthetic speech detection method based on background noise, which performs synthetic speech detection by only analyzing background noise information to determine whether the audio is deepfake. Since background noise does not contain semantic content such as context and acoustic information such as accent, intonation, emotion, and timbre, although privacy attackers can receive a series of background markers, the lack of semantic, prosodic, and speaker timbre clues hinders their ability to recover understandable content and infer user identity characteristics. In the feature extraction stage, different features are extracted and stripped hierarchically to promote the hierarchical decoupling of different information; the semantic features and acoustic features are decoupled, and further the prosody, speaker features, and background noise are decoupled in the acoustic features, gradually stripping the semantic features, prosody, and speaker features, and extracting purer background noise information for deepfake detection, improving the detection efficiency and accuracy. Compared with traditional detection methods, the method of the present invention realizes an efficient and accurate synthetic speech detection method, and also ensures the balance between privacy protection and data security, providing a new technical path for the application in the field of speech deepfake detection.

[0039] 2. The embodiment of the present invention provides a synthetic speech detection method based on background noise, which can also improve the adaptability to different types of forged speech. Traditional methods usually rely on the acoustic features of the speech itself, while the present invention avoids the adversarial attacks of these forgery methods by separating noise and speech features, significantly improving the detection ability of synthetic speech. In addition, it also has a certain degree of flexibility and can adapt to the needs of different devices or application environments by further optimizing the background noise information. For the speech collected in different recording environments, the characteristics of the background noise will be different, so the method has good adaptability and can process various changing background noises by appropriately adjusting parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic flowchart of a synthetic speech detection method based on background noise provided by an embodiment of the present invention.

[0041] Figure 2 is a framework diagram of a synthetic speech detection method based on background noise proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0043] The following describes the content involved in the above embodiments in combination with a preferred embodiment.

[0044] Example 1

[0045] The present invention provides a method for detecting synthetic speech based on background noise, including:

[0046] S1. An encoder is used to extract a sample encoding containing composite features from the original audio, and the composite features include semantic features and acoustic features;

[0047] S2. The sample encoding is input into a residual vector quantization model RVQs for feature extraction. A latent model is used to guide the first quantization output VQ1 of RVQs to extract the semantic features in the sample encoding, and the semantic features are stripped from the composite features through the residual structure of RVQs to obtain pure acoustic features;

[0048] S3. A natural language 3-generation model is used to guide the second quantization output VQ2 and the third quantization output VQ3 of RVQs to extract the prosody features and speaker information in the pure acoustic features respectively, and the prosody features and speaker information are stripped from the pure acoustic features through the residual structure of RVQs to obtain pure background noise information;

[0049] S4. The background noise information is input into a detection model for speech detection to determine whether the original audio is synthetic audio.

[0050] Reference Figure 2 , in an embodiment, a model for detecting synthetic speech based on background noise is constructed, including an encoder, a feature extraction module, and a detector module; the encoder is used to extract a sample encoding containing composite features from the original audio; the feature extraction module includes a residual vector quantization model RVQs, a latent model, and a natural language 3-generation model; the latent model guides the first quantization output VQ1 of RVQs to extract the semantic features in the sample encoding, and the natural language 3-generation model guides the second quantization output VQ2 and the third quantization output VQ3 of RVQs to extract the prosody features and speaker information in the pure acoustic features respectively. Thus, after the sample encoding passes through VQ1, VQ2, and VQ3 in sequence, the semantic features, prosody features, and speaker information are stripped to obtain pure background noise information; the detection model of the detector module includes a bottleneck layer, a Transformer classifier, and an output layer. The bottleneck layer reduces the dimension of the pure background noise information, the Transformer classifier aggregates the output features of each attention head to obtain a unified attention spectrum, and the output layer judges it based on the feature difference between the background noise of the real audio and the synthetic audio, so as to determine whether the original audio is synthetic audio. If it is synthetic audio, the output result is "forged", and if it is real audio, the output result is "real".

[0051] The present invention cleverly strips other feature information from the speech, thereby using background noise as the key clue for detecting synthesized speech. The encoder extracts composite features and uses the RVQs model to gradually separate them, finally obtaining background noise information and performing detection based on this. This solution is different from traditional speech recognition methods, focusing on the background noise features in the sound that are not easily forged, and can effectively improve the detection accuracy of synthesized speech.

[0052] Optionally, the encoder includes a one-dimensional convolutional layer, a residual block, a downsampling layer, an LSTM layer, and a convolutional layer;

[0053] The one-dimensional convolutional layer is used to extract features from the original audio signal, transform the original audio signal into an array form, and obtain preliminary features;

[0054] The residual block, downsampling layer, and LSTM layer are used to extract information across the audio feature space from the preliminary features and output composite features containing semantic features and acoustic features;

[0055] The convolutional layer is used to convert the dimension of the composite features into the dimension of the quantization input RVQ to obtain a sample encoding containing the composite features.

[0056] In the embodiment of the present invention, the dimension of the given original audio signal X is 1, the convolutional kernel size of the one-dimensional convolutional layer is 7, and the preliminary feature representation E is obtained after passing through the convolutional layer. The preliminary feature representation E is passed into four residual blocks, and the convolutional kernel sizes in the four residual blocks are 8, 5, 4, and 2 respectively. Then, through the downsampling layer and the bidirectional LSTM (Bi-LSTM), information across the audio feature space is extracted to obtain a rich composite feature E'; the composite feature E' contains semantic features and acoustic features. The composite feature E' is passed through the convolutional layer, and the exponential linear unit (ELU) and layer normalization are used in the convolutional layer to enhance the non-linear representation and the stability of the model, and the dimension of the feature E' is matched to be consistent with the RVQ input dimension, with a dimension of 256.

[0057] Optionally, step S2 specifically includes:

[0058] S2.1. Map the output dimension of VQ1 to the same dimension as the semantic features extracted by the latent model;

[0059] S2.2. Use the average representation of all latent model layers as the semantic supervision signal, and supervise VQ1 to learn the content representation in the sample encoding through the loss function, extract semantic features, and obtain corresponding semantic tags; the loss function is:

[0060]

[0061] Wherein, Represents the quantization output of the VQ layer, represents the time step of the supervision signal, represents the cosine similarity, represents the sigmoid activation function, represents the projection matrix;

[0062] S2.3. Use the residual structure of RVQs to subtract the semantic features corresponding to the semantic tokens from the composite features to obtain pure acoustic features.

[0063] Input the encoded feature E' into the quantizer RVQs for quantization. The feature dimension of the first-layer quantization output VQ1 is 1024, and it is mapped to the latent model feature dimension of 768 through a linear layer to achieve feature dimension matching.

[0064] Use the semantic feature F1 extracted by the latent model layer as the semantic supervision signal, and calculate the distillation loss for content representation supervised learning of VQ1 to obtain the semantic tokens of VQ1 . The loss function is:

[0065]

[0066] where, represents the quantization output of the VQ layer, represents the time step of the supervision signal, represents the cosine similarity, represents the sigmoid activation function, represents the projection matrix.

[0067] Use the residual structure of RVQs to decouple the semantic features and acoustic features. During the backpropagation process, adopt the gradient reversal technique to change the gradient direction by multiplying the acoustic feature gradient by -alpha, suppress the learning of acoustic features, and enhance the extraction effect of semantic tokens . Since RVQs has a residual structure, the semantic features corresponding to the semantic tokens output by VQ1 can be stripped from the composite representation to obtain pure acoustic information A without semantic information, realizing the decoupling of semantic features and pure acoustic features.

[0068] Optionally, step S3 specifically includes:

[0069] S3.1. Map the output dimension of VQ2 to the same dimension as the prosody features extracted by the natural language 3-generation model, and map the output dimension of VQ3 to the same dimension as the speaker features extracted by the natural language 3-generation model;

[0070] S3.2. Use the natural language 3rd generation model to extract the prosody features and speaker information in the pure acoustic features from the second quantization output VQ2 and the third quantization output VQ3 respectively, and perform supervised learning on VQ2 and VQ3 through a loss function to obtain the prosody label P and the speaker label ;

[0071] S3.3. Through the residual structure of RVQs, strip the corresponding prosody features and speaker information of the prosody label P and the speaker label from the pure acoustic features to obtain pure background noise information.

[0072] The output feature dimensions of VQ2 and VQ3 are 1024. Use a linear layer to map their dimensions to 256, which is consistent with the prosody features and speaker feature dimensions of the natural language 3rd generation model.

[0073] Use the prosody features F2 extracted by the natural language 3rd generation model as the prosody supervision signal, and perform prosody representation supervised learning on VQ2 by calculating the distillation loss to obtain the prosody label P of VQ2; use the speaker features F3 extracted by the natural language 3rd generation model as the speaker supervision signal, and perform speaker representation supervised learning on VQ3 by calculating the distillation loss to obtain the speaker label of VQ3 . Further, in the backpropagation process, use the gradient reversal technique to suppress the learning of other features respectively and enhance the extraction effects of the prosody label P and the speaker label Based on the second quantization output and the third quantization output, further decouple the prosody, speaker features and background noise in the pure acoustic features; based on the pure acoustic information A of the input VQ2, strip the prosody label P of VQ2 and the speaker label of VQ3 from it to obtain pure background noise information B.

[0074] Optionally, after step S3, it further includes:

[0075] Transmit the pure background noise information to the remaining five quantizers of RVQs in sequence, and each quantizer refines the background noise information in sequence to enhance its feature representation.

[0076] As Figure 2 shown, transmit the obtained background noise information B to the next five quantizers VQ4 to VQ8 to further refine the background noise information.

[0077] Optionally, the detection model includes a bottleneck layer, a Transformer classifier and an output layer;

[0078] The bottleneck layer is used to reduce the dimension of the obtained background noise features through one-dimensional convolution and batch normalization;

[0079] The Transformer classifier is used to pass the compressed background noise features through multiple attention heads to capture different long-range dependencies and dynamic spatial weighting information, and aggregate the output features of each attention head to obtain a unified attention spectrum;

[0080] The output layer is used to compare the attention spectrum with the corresponding attention spectra of the background noise of the learned real audio and the corresponding attention spectra of the background noise of the synthetic audio, and determine whether the original audio is synthetic audio.

[0081] The extracted background noise features B are passed through a bottleneck layer composed of 5 convolutional layers with a kernel size of 1 and 1 normalization layer to reduce their dimension from 5 dimensions to 1 dimension, which is convenient for subsequent layers to operate on the compact representation B1, and at the same time avoid overfitting. The compressed background noise features B1 are fed into the Transformer classifier. Through the best 8 attention heads, the model can more effectively participate in long-range feature interaction and dynamic spatial weighting, and parallel process different aspects of the input feature space; by using sinusoidal positional encoding, aggregate the output features of each attention head to obtain the attention spectrum B1'. The aggregated attention spectrum B1' is input into the output layer of two groups of Transformer classifiers to process the whole sequence and capture global dependencies. Each group of encoders contains two feed-forward networks, multi-head self-attention and layer normalization modules. Finally, a classification score score is output through a fully connected layer. The score score represents the similarity between the attention spectrum of the detected audio and the attention spectra of the learned real audio and synthetic audio. Combined with the best threshold threshold, the detected audio and the more similar one of the two are output as the classification result (using the threshold threshold to classify the output score score, classifying the samples higher than the threshold as synthetic audio, and the results lower than the threshold as real audio) to determine whether the audio is deepfake. Experiments prove that setting two groups of Transformer classifiers has the best effect in this embodiment. Using two independent Transformer classifiers can avoid a single classifier relying too much on a certain type of feature, thereby reducing the risk of overfitting; even if the learning ability of a certain group of classifiers is affected by some errors, the other group can make up for its deficiencies.

[0082] The prototype code of the method of the present invention is implemented in the Python language, and feasibility and effectiveness experimental verifications are carried out on the ASVspoof 2019 corpus and the ASVspoof 2021 corpus, involving the detection of samples generated by various means such as speech synthesis, voice conversion, replay, etc. Among them, the detection accuracy rate on the ASVspoof 2019 dataset reaches 89.68%, the detection equal error rate (EER) is as low as 0.054, and the tandem detection cost function (t-DCF) is as low as 0.159, indicating that the model has good classification performance and high detection efficiency, and verifying the feasibility and effectiveness of the solution of the present invention.

[0083] The solution of the present invention promotes the hierarchical decoupling of different information by hierarchically extracting and stripping different features in the feature extraction stage; decouples semantic features and pure acoustic features, further decouples prosody, speaker features and background noise in the pure acoustic features, gradually strips semantic features, prosody and speaker features, and extracts purer background noise information for deepfake detection; performs synthetic speech detection by only analyzing background noise information, judges whether the audio is a deepfake, and improves the detection efficiency and accuracy. At the same time, since the detected audio does not contain human voice information, it hinders the ability to restore understandable content and infer user identity features, ensuring the balance of privacy protection and data security, and providing a new technical path for the application in the field of voice deepfake detection.

[0084] Embodiment 2

[0085] The present invention also provides a synthetic speech detection device based on background noise for performing the synthetic speech detection method described in any one of Embodiments 1, including:

[0086] A data preprocessing module for extracting a sample code containing composite features from the original audio by using an encoder, where the composite features include semantic features and acoustic features;

[0087] An acoustic feature extraction module for inputting the sample code into a residual vector quantization model (RVQs) for feature extraction, using a latent element model to guide the first quantization output (VQ1) of the RVQs to extract semantic features in the sample code, and stripping the semantic features from the composite features through the residual structure of the RVQs to obtain pure acoustic features;

[0088] A background feature extraction module for using a NaturalSpeech3 model to guide the second quantization output (VQ2) and the third quantization output (VQ3) of the RVQs to extract prosody features and speaker information in the pure acoustic features respectively, and stripping the prosody features and speaker information from the pure acoustic features through the residual structure of the RVQs to obtain pure background noise information;

[0089] A detection module, configured to input the background noise information into a detection model for voice detection to determine whether the original audio is synthetic audio.

[0090] The synthetic voice detection device provided by an embodiment of the present invention is used to execute the synthetic voice detection method provided by any embodiment of the present invention, and has corresponding beneficial effects.

[0091] Embodiment III

[0092] The present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the synthetic voice detection method described in any item of Embodiment I.

[0093] It is easy for those skilled in the art to understand that the above are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting synthetic speech based on background noise, characterized in that, Including: S1. An encoder is used to extract sample encodings containing composite features from the original audio, and the composite features include semantic features and acoustic features; S2. The sample encodings are input into a Residual Vector Quantization model RVQs for feature extraction. A latent model is used to guide the first-layer quantization output VQ1 of the RVQs to extract the semantic features in the sample encodings, and the semantic features are stripped from the composite features through the residual structure of the RVQs to obtain pure acoustic features; S3. A natural language 3-generation model is used to guide the second quantization output VQ2 and the third-layer quantization output VQ3 of the RVQs to extract the prosody features and speaker information in the pure acoustic features respectively, and the prosody features and speaker information are stripped from the pure acoustic features through the residual structure of the RVQs to obtain pure background noise information; S4. The background noise information is input into a detection model for speech detection to determine whether the original audio is synthetic audio.

2. The synthetic speech detection method according to claim 1, wherein The encoder includes a one-dimensional convolutional layer, a residual block, a downsampling layer, an LSTM layer, and a convolutional layer; The one-dimensional convolutional layer is used to extract features from the original audio signal, transform the original audio signal into an array form, and obtain preliminary features; The residual block, downsampling layer, and LSTM layer are used to extract information across the audio feature space from the preliminary features and output composite features containing semantic features and acoustic features; The convolutional layer is used to convert the dimension of the composite features into the dimension of the quantization input RVQ to obtain sample encodings containing composite features.

3. The synthetic speech detection method according to claim 1, wherein Step S2 specifically includes: S2.

1. Map the output dimension of VQ1 to the same dimension as the semantic features extracted by the latent model; S2.

2. Use the average representation of all latent model layers as a semantic supervision signal, and supervise VQ1 to learn the content representation in the sample encodings through a loss function, extract semantic features, and obtain corresponding semantic tokens; the loss function is: Among them, represents the quantization output of the VQ layer, represents the time step of the supervision signal, represents the cosine similarity, represents the sigmoid activation function, represents the projection matrix; S2.

3. Use the residual structure of the RVQs to subtract the semantic features corresponding to the semantic tokens from the composite features to obtain pure acoustic features.

4. The synthetic speech detection method according to claim 1, wherein Step S3 specifically includes: S3.

1. Map the output dimension of VQ2 to the same dimension as the prosody features extracted by the natural language 3-generation model, and map the output dimension of VQ3 to the same dimension as the speaker features extracted by the natural language 3-generation model; S3.

2. Use the natural language 3rd generation model to extract the prosody features and speaker information in the pure acoustic features from the second quantization output VQ2 and the third quantization output VQ3 respectively, and perform supervised learning on VQ2 and VQ3 through a loss function to obtain the prosody label P and the speaker label ; S3.

3. Stripping the corresponding prosodic features and speaker information of the prosody markers P and speaker markers from the clean acoustic features through the residual structure of RVQs to obtain clean background noise information. ​ 5. A method for detecting synthesized speech based on background noise according to claim 1, characterized in that, After step S3, it further includes: The pure background noise information is sequentially passed to the remaining five quantizers of the RVQs, and each quantizer sequentially refines the background noise information to enhance its feature representation.

6. The synthetic speech detection method based on background noise according to claim 1, wherein The detection model includes a bottleneck layer, a Transformer classifier, and an output layer; The bottleneck layer is used to reduce the dimension of the obtained background noise features through one-dimensional convolution and batch normalization; The Transformer classifier is used to pass the compressed background noise features through multiple attention heads, capture different long-range dependency relationships and dynamic spatial weighting information, and aggregate the output features of each attention head to obtain a unified attention spectrum; The output layer is used to compare the attention spectrum with the attention spectrum corresponding to the background noise of the learned real audio and the attention spectrum corresponding to the background noise of the synthesized audio, and determine whether the original audio is synthesized audio.

7. A synthetic speech detection device based on background noise, which is used to execute the synthetic speech detection method described in any one of claims 1-6, and is characterized in that It includes: A data preprocessing module, which is used to extract a sample encoding containing composite features from the original audio by using an encoder, and the composite features include semantic features and acoustic features; An acoustic feature extraction module, which is used to input the sample encoding into a residual vector quantization model RVQs for feature extraction, use a HuBERT model to guide the first quantization output VQ1 of RVQs to extract the semantic features in the sample encoding, and strip the semantic features from the composite features through the residual structure of RVQs to obtain pure acoustic features; A background feature extraction module, which is used to use a natural language 3-generation model to guide the second quantization output VQ2 and the third quantization output VQ3 of RVQs to extract the prosody features and speaker information in the pure acoustic features respectively, and strip the prosody features and speaker information from the pure acoustic features through the residual structure of RVQs to obtain pure background noise information; A detection module, which is used to input the background noise information into a detection model for speech detection to determine whether the original audio is synthesized audio.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the synthesized speech detection method according to any one of claims 1 to 6.

Images