Synthetic sound detection method and device, server and medium
The audio feature extraction model and pre-trained large language model detect the synthesized tone features in audio files, solving the problem of inefficiency in the existing technology, achieving efficient and accurate synthesized tone recognition, and improving user experience.
Patent Information
- Application Number
- CN202411357599.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-10
AI Technical Summary
The existing synthetic sound detection methods rely on manual adjustment of score thresholds and feature filtering, which are inefficient and cannot meet the identification needs of large numbers of audio files.
The audio feature extraction model is used to extract the target audio features that characterize the synthetic tone characteristics, and the pre-trained large language model is used for context detection and audio structure processing to determine whether there are synthetic tone in the audio file.
It improves the efficiency and accuracy of synthetic sound detection, avoids the random application of synthetic sound in prohibited use scenarios, and improves the user experience.
Smart Images
Figure CN119296579A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of synthetic sound detection, and in particular to a synthetic sound detection method, device, server and medium. Background Art
[0002] Synthetic sound refers to non-natural human voice or mechanical sound generated by synthesis technology. It is widely used in various business scenarios because of its specific frequency, amplitude, pitch and other characteristics. For example, adding synthetic sound to video files to explain the video content, adding synthetic sound to navigation applications for voice navigation, or adding synthetic sound to game applications to achieve character dialogue, etc.
[0003] In order to prevent synthesized sounds from being arbitrarily applied to business scenarios, it is often necessary to detect whether there is synthesized sound in the audio file to prevent the randomly applied synthesized sound from affecting the user experience in specific business scenarios. Therefore, how to efficiently and accurately detect whether there is synthesized sound in the audio file is a prerequisite for improving the user experience in specific business scenarios. Summary of the invention
[0004] In order to solve the above technical problems, the present disclosure provides a synthetic sound detection method, device, server and medium.
[0005] In a first aspect, the present disclosure provides a synthetic voice detection method, which is applied to a server, and the method includes:
[0006] Get the audio file to be detected;
[0007] Using an audio feature extraction model, extracting features from the audio file to be detected to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize characteristics of synthetic sound;
[0008] The pre-trained large language model is used to perform context detection and audio structure processing on the target audio features to determine whether there is synthetic sound in the audio file to be detected.
[0009] In a second aspect, the present disclosure provides a synthetic sound detection device, which is configured on a server, and the device includes:
[0010] An audio file acquisition module is used to acquire the audio file to be detected;
[0011] An audio feature acquisition module, used to extract features from the audio file to be detected using an audio feature extraction model, and obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize the characteristics of the synthetic sound;
[0012] The synthetic sound detection module is used to use a pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether there is a synthetic sound in the audio file to be detected.
[0013] In a third aspect, an embodiment of the present disclosure further provides a server, the server comprising:
[0014] one or more processors;
[0015] a storage device for storing one or more programs,
[0016] When one or more programs are executed by one or more processors, the one or more processors implement the method provided in the first aspect.
[0017] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the first aspect is implemented.
[0018] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0019] A synthetic sound detection method, device, server and medium of the disclosed embodiments obtain an audio file to be detected; use an audio feature extraction model to extract features from the audio file to be detected to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize the characteristics of the synthetic sound; use a pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether there is synthetic sound in the audio file to be detected. Thus, for audio files that need to be detected for synthetic sound, the audio feature extraction model and the pre-trained large language model are used successively to extract audio features that can characterize the characteristics of the synthetic sound from the audio file to be detected and to determine whether there is synthetic sound in the audio file to be detected, without relying on manual or other algorithms for detection, thereby improving the detection efficiency and accuracy of synthetic sound, and preventing audio files with synthetic sound from being arbitrarily applied to business scenarios that prohibit the use of synthetic sound, so as to improve the user experience in specific business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0022] Figure 1 A flowchart of a synthetic sound detection method provided by an embodiment of the present disclosure;
[0023] Figure 2 A schematic diagram of the process of S120 provided in an embodiment of the present disclosure;
[0024] Figure 3 A schematic diagram of the process of S130 provided in an embodiment of the present disclosure;
[0025] Figure 4 A logic diagram of a synthetic sound detection method provided by an embodiment of the present disclosure;
[0026] Figure 5 A schematic diagram of the structure of a synthetic sound detection device provided by an embodiment of the present disclosure;
[0027] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0028] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0030] In the related art, the method for identifying synthetic sound is generally as follows: the server first identifies audio features from the audio file, and then calculates the fusion score of the audio features based on each audio feature and its corresponding weight, and then compares the fusion score with a preset score threshold. If the fusion score exceeds the preset score threshold, it is determined that there is synthetic sound in the audio file.
[0031] However, this method requires manual regular sampling and continuous adjustment of the preset score threshold, as well as manual screening of audio features for calculating the fusion score. As a result, the synthetic sound detection method relies on large computing resources and low efficiency, and cannot meet the needs of synthetic sound recognition for a large number of audio files.
[0032] In order to improve the detection efficiency of synthetic sound, the following Figures 1 to 4 The synthetic sound detection method provided in the embodiment of the present disclosure is described. In the embodiment of the present disclosure, the synthetic sound detection method can be executed by a server.
[0033] Figure 1 A schematic flow chart of a synthetic sound detection method provided by an embodiment of the present disclosure is shown.
[0034] like Figure 1 As shown, the synthetic sound detection method may include the following steps.
[0035] S110: Obtain an audio file to be detected.
[0036] In business scenarios such as short video recommendation and distribution, navigation application operation, and game application operation, it is often necessary to detect whether the audio files of specific business scenarios contain synthetic sounds, so that the server can recommend and distribute short videos with synthetic sounds and natural human voices according to different priorities, verify whether the voice type of the audio file of the navigation application is the type set by the user, and verify whether the voice type of the audio file of the game application is allowed.
[0037] In the face of various synthetic sound detection requirements, the server obtains audio files from the business scenarios corresponding to the requirements as the audio files to be detected. For example, if the business scenario is a short video distribution scenario, the server first obtains the video to be recommended and distributed, then separates the audio data from the video, and then saves the audio data in the form of a file to obtain the audio file to be detected.
[0038] The audio file to be detected is a file containing audio data of a specific business scenario. Optionally, the audio file to be detected includes one or both of natural human voice and synthetic voice.
[0039] S120. Perform feature extraction on the audio file to be detected by using an audio feature extraction model to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize characteristics of the synthesized sound.
[0040] It is understandable that synthetic sound has unique audio characteristics compared to natural human voice. For example, synthetic sound has specific frequency, specific amplitude, specific intensity distribution, specific pitch, specific noise, etc. Therefore, the server can use a large number of audio files and synthetic sound feature labels of the audio files in advance to iteratively train an audio feature extraction model so that the audio feature extraction model has the ability to extract audio features of synthetic sound characteristics.
[0041] Based on this principle, in this embodiment, the server inputs the audio file to be detected into the audio feature extraction model, utilizes the ability of the audio feature extraction model to extract audio features with synthetic sound characteristics, extracts audio features characterizing the synthetic sound characteristics from the audio file to be detected, and outputs the audio features as target audio features.
[0042] Accordingly, the target audio features include, but are not limited to, one or more combinations of frequency features, amplitude features, intensity distribution features, pitch features, and noise characteristics.
[0043] Therefore, the audio feature extraction model can be used to directly extract audio features with synthetic sound characteristics from the audio file to be detected, without relying on manual or other algorithms to screen out the target audio features from a variety of audio features, thereby improving the feature extraction efficiency.
[0044] In order to improve the extraction accuracy of the target audio features, the audio file to be detected can also be subjected to noise suppression processing. Accordingly, S120 can be specifically implemented in the following manner: performing noise suppression processing on the audio file to be detected to obtain the audio file after the noise suppression processing; using the audio feature extraction model to perform feature extraction on the audio file after the noise suppression processing to obtain the target audio features of the audio file to be detected.
[0045] In some embodiments, a specific implementation method of performing noise suppression processing on an audio file to be detected and obtaining the audio file after the noise suppression processing includes: using a pre-trained noise suppression model to process the audio file to be detected to obtain the audio file after the noise suppression processing, wherein the noise suppression model is obtained by training a deep neural network through a first sample audio containing noise and a second sample audio containing no noise.
[0046] Among them, the noise suppression model can specifically be a deep neural network model.
[0047] In other embodiments, a specific implementation method of performing noise suppression processing on an audio file to be detected and obtaining the audio file after the noise suppression processing includes: in a noise-free environment, performing noise estimation on the audio file to be detected and determining a noise-free signal of the audio file to be detected; performing short-time Fourier transform on the original audio signal and the noise-free signal contained in the audio file to be detected to obtain spectrum data of the original audio signal and spectrum data of the noise-free signal; subtracting the spectrum data of the original audio signal from the spectrum data of the noise-free signal to obtain a spectrum data difference; and performing an inverse short-time Fourier transform on the spectrum data difference to obtain the audio file after the noise suppression processing.
[0048] The noise-free environment refers to a simulated completely quiet environment, so that a noise-free signal of the audio file to be detected is generated in this environment.
[0049] The original audio signal contained in the audio file to be detected refers to an audio signal containing noise.
[0050] Therefore, before using the audio feature extraction model to extract features from the audio file to be detected, noise suppression is performed on the audio file to be detected to prevent the noise in the audio file to be detected from affecting the audio extraction accuracy, thereby improving the recognition effect of the target audio features.
[0051] S130: Use the pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether there is synthetic sound in the audio file to be detected.
[0052] In this embodiment, the server can first generate an audio text corresponding to the target audio feature, and then call a pre-trained large language model to perform context detection and audio structure processing on the target audio feature in the audio text to determine whether there is synthetic sound in the audio file to be detected.
[0053] Among them, the pre-trained large language model is a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language.
[0054] Among them, context detection can be understood as using a pre-trained large language model to capture contextual features in the target audio features. For example, the position, part of speech, co-occurring words, syntactic structure, context and other features of words are captured from the target audio features as contextual features.
[0055] Among them, audio structure processing can be understood as using a pre-trained large language model to understand the features of different levels in the target audio features and the relationship between different levels, for example, understanding the features of the time domain, frequency domain, and acoustic features of the target audio features, as well as understanding the relationship between the time domain, frequency domain, and acoustic features.
[0056] Furthermore, the server can also use the audio text corresponding to the target audio features and the synthetic sound detection results to fine-tune the pre-trained large language model, so as to migrate the large language model to the synthetic sound detection scenario, thereby continuously improving the ability of the large language model to detect synthetic sounds from audio files.
[0057] In this way, the pre-trained large language model can be directly called to perform context detection and audio structure processing on the target audio features that characterize the characteristics of the synthetic sound, so as to mine the deep information of the target audio features, thereby quickly and accurately identifying whether there is synthetic sound in the audio file to be detected.
[0058] A synthetic sound detection method of an embodiment of the present disclosure obtains an audio file to be detected; uses an audio feature extraction model to extract features from the audio file to be detected to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize the characteristics of synthetic sounds; uses a pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether there is synthetic sound in the audio file to be detected. Thus, for audio files that need to be detected for synthetic sound, the audio feature extraction model and the pre-trained large language model are successively used to extract audio features that can characterize the characteristics of synthetic sounds from the audio file to be detected and to determine whether there is synthetic sound in the audio file to be detected, without relying on manual or other algorithms for detection, thereby improving the efficiency and accuracy of synthetic sound detection, avoiding audio files with synthetic sounds from being arbitrarily applied to business scenarios that prohibit the use of synthetic sounds, and improving the user experience in specific business scenarios.
[0059] In another embodiment of the present disclosure, step S120 is refined.
[0060] Figure 2 A schematic diagram of the process of S120 provided in an embodiment of the present disclosure is shown.
[0061] like Figure 2 As shown, S120 may include the following steps.
[0062] S210: Pre-process the audio file to be detected based on the pre-processing network in the audio feature extraction model to obtain pre-processing features of the audio file to be detected.
[0063] Among them, the pre-processing network is a network structure that can pre-process the audio file to be detected.
[0064] In some embodiments, the audio feature extraction model is a pre-trained audio feature extractor. Accordingly, the specific implementation method of S210 includes: based on the pre-emphasis processing layer in the pre-processing network, pre-emphasize the audio signal in the audio file to be detected to obtain the pre-emphasized audio signal; based on the framing layer in the pre-processing network, divide the pre-emphasized audio signal into multiple frames of overlapping audio signals; based on the windowing layer in the pre-processing network, perform windowing on the multiple frames of overlapping audio signals to obtain the pre-processing features of the audio file to be detected.
[0065] Specifically, first, the server uses a first-order high-pass filter to pre-emphasize the audio signal through the pre-emphasis processing layer in the pre-processing network to enhance the high-frequency signal and obtain a pre-emphasized audio signal; then, the server uses a framing layer in the pre-processing network to divide the pre-emphasized audio signal into multiple overlapping short-time frames through a sliding window of a specific duration as a multiple-frame overlapping audio signal; then, the server uses a windowing layer in the pre-processing network to perform windowing on the multiple-frame overlapping audio signal using a window function such as a Hamming window or a Henning window to obtain the pre-processing features of the audio file to be detected, so as to reduce the discontinuity of the frame edges and reduce spectrum leakage.
[0066] In some embodiments, the audio feature extraction model is a network model formed based on a representation learning method. Accordingly, the specific implementation method of S210 includes: based on the first normalization layer in the pre-processing network, normalizing the audio signal in the audio file to be detected to obtain the first normalized feature of the audio signal; based on the segmentation layer in the pre-processing network, segmenting the first normalized feature into multiple feature segments; based on the feature extraction layer in the pre-processing network, performing short-time Fourier transform or Mel frequency cepstral coefficient processing on the multiple feature segments to obtain the pre-processing feature of the audio file to be detected.
[0067] Specifically, first, the server normalizes the audio signal through the first normalization layer in the pre-processing network, so that the audio signal is in a standardized interval (such as [-1,1]), and obtains the first normalized feature of the audio signal; then, the server divides the first normalized feature into segments of fixed length through the segmentation layer in the pre-processing network, each segment represents a small time window, and obtains multiple feature segments; then, the server performs short-time Fourier transform (STFT) or Mel-frequency cepstral coefficient processing (MFCCs) on multiple feature segments through the feature extraction layer in the pre-processing network to obtain the pre-processing features of the audio file to be detected.
[0068] In some embodiments, the audio feature extraction model is another network model formed based on a representation learning method. Accordingly, the specific implementation method of S210 includes: based on the second normalization layer in the pre-processing network, normalizing the audio signal in the audio file to be detected to obtain the second normalized feature of the audio signal; based on the quantization layer in the pre-processing network, quantizing the second normalized feature of the audio signal to obtain the pre-processing feature of the audio file to be detected.
[0069] Specifically, first, the server normalizes the audio signal through the second normalization layer in the pre-processing network, so that the audio signal is in a standardized interval (such as [-1,1]), and obtains the second normalized feature of the audio signal; then, the server converts the continuous first normalized feature into a discrete value through the quantization layer in the pre-processing network, and obtains the pre-processing feature of the audio file to be detected.
[0070] Therefore, before using the audio feature extraction model to extract the target audio features, different methods are used to pre-process the audio files to be detected based on the pre-processing network, which is beneficial to improving the extraction accuracy of the target audio features.
[0071] S220: Based on the feature extraction network in the audio feature extraction model, extract the pre-processing features to obtain the initial audio features of the audio file to be detected.
[0072] Among them, the feature extraction network is a network structure that actually has the ability to extract audio features.
[0073] In some embodiments, the audio feature extraction model is a pre-trained audio feature extractor. Accordingly, the specific implementation method of S220 includes: based on the feature conversion layer in the feature extraction network, converting the pre-processing features from the time domain to the frequency domain; based on the filtering processing layer in the feature extraction network, obtaining the Mel cepstrum from the pre-processing features in the frequency domain; based on the logarithmic calculation layer in the feature extraction network, performing logarithmic calculation on the Mel cepstrum to obtain the Mel cepstrum logarithm; based on the linear transformation layer in the feature extraction network, performing linear transformation on the Mel cepstrum logarithm to determine the Mel-frequency cepstrum coefficient; based on the Mel-frequency cepstrum coefficient screening layer in the feature extraction network, screening the Mel-frequency cepstrum coefficient to obtain the initial audio features of the audio file to be detected.
[0074] Specifically, first, the server performs a fast Fourier transform (FFT) on the pre-processing features through the feature conversion layer in the feature extraction network, so as to convert the pre-processing features from the time domain to the frequency domain; then, the server uses a Mel frequency filter (Mel frequency filter) to filter the pre-processing features in the frequency domain through the filtering processing layer in the feature extraction network to simulate the sensitivity difference of the human auditory system to different frequencies, and obtain the Mel cepstrum; then, the server performs logarithmic calculation on the Mel cepstrum through the logarithmic calculation layer in the feature extraction network to highlight important frequency components and obtain the Mel cepstrum logarithm, thereby increasing the dynamic range and making the signal more stable; then, the server uses a discrete cosine transform method to perform a linear transformation on the Mel cepstrum logarithm through the linear transformation layer in the feature extraction network to obtain the Mel-frequency cepstrum coefficients; finally, the server uses the Mel-frequency cepstrum coefficient screening layer in the feature extraction network to screen a preset number (for example, the first 12-13) of Mel-frequency cepstrum coefficients as the initial audio features of the audio file to be detected.
[0075] In some embodiments, the audio feature extraction model is a network model formed based on a representation learning method. Accordingly, the specific implementation method of S220 includes: adding position features to pre-processing features based on the position encoding layer in the feature extraction network; performing multi-head self-attention processing on pre-processing features carrying position features based on the multi-head self-attention layer in the feature extraction network to obtain self-attention features corresponding to synthetic sounds at different time points; performing nonlinear transformation on the self-attention features based on the feedforward network in the feature extraction network to obtain the initial audio features of the audio file to be detected.
[0076] Specifically, first, the server adds position features to the pre-processing features through the position encoding layer in the feature extraction network to help the feature extraction network understand the order of each feature in the pre-processing features; then, the server captures information of different intensities from the pre-processing features carrying position features through the multi-head self-attention layer in the feature extraction network, thereby maintaining attention to the position of each pre-processing feature to help the model capture different frequencies, amplitudes, pitches, noises and other features, thereby obtaining self-attention features corresponding to synthetic sounds at different time points; then, the server performs nonlinear transformation on the self-attention features through the feedforward network in the feature extraction network to obtain the initial audio features of the audio file to be detected.
[0077] In some embodiments, the audio feature extraction model is another network model formed based on a representation learning method. Accordingly, the specific implementation method of S220 includes: based on the causal convolution layer, the residual connection layer and the gated activation layer in the feature extraction network, the pre-processing features are processed to obtain the initial audio features of the audio file to be detected.
[0078] Specifically, first, the server captures frequency, amplitude, pitch, noise and other features from the pre-processing features through the causal convolution layer in the feature extraction network, then adds the features through the residual connection layer, and controls which information is passed to the next layer through the gated activation layer, thereby obtaining the initial audio features of the audio file to be detected.
[0079] Therefore, when using the feature extraction network of the audio feature extraction model, different methods are adopted to extract target audio features representing the characteristics of the synthetic sound from the preprocessing features, thereby improving the flexibility and accuracy of the audio feature extraction.
[0080] S230: Based on the post-processing network in the audio feature extraction model, the initial audio features are post-processed to obtain target audio features of the audio file to be detected.
[0081] Among them, the post-processing network is a network structure that optimizes the initial audio features.
[0082] In some embodiments, the audio feature extraction model is a pre-trained audio feature extractor. Accordingly, the specific implementation method of S230 includes: based on the differential coefficient calculation layer in the post-processing network, performing differential coefficient calculation on the initial audio features at adjacent moments to obtain the target audio features of the audio file to be detected.
[0083] Specifically, the server calculates the differential coefficients between the initial audio features at adjacent moments through the differential coefficient calculation layer in the post-processing network to obtain the target audio features. These differential coefficients can capture information that changes over time, usually including first-order differences (such as velocity) and second-order differences (such as acceleration).
[0084] In some embodiments, the audio feature extraction model is a network model formed based on a representation learning method. Accordingly, the specific implementation method of S230 includes: processing the initial audio features based on the pooling layer in the post-processing network to obtain the pooling features of the initial audio features; processing the pooling features based on the fully connected layer in the post-processing network to obtain the target audio features of the audio file to be detected.
[0085] Specifically, the server first performs a pooling operation on the initial audio features through the pooling layer in the post-processing network to reduce the dimension and extract key features, and then integrates the pooled features through the fully connected layer in the post-processing network. The regularization layer can also be connected to overfit the integrated pooled features to obtain the target audio features of the audio file to be detected.
[0086] Therefore, before using the audio feature extraction model to extract the initial audio features, different methods are used to post-process the audio files to be detected based on the post-processing network, so as to optimize the initial audio features, which is conducive to improving the extraction accuracy of the target audio features.
[0087] In yet another embodiment of the present disclosure, step S130 is refined.
[0088] Figure 3 A schematic diagram of the process of S130 provided in an embodiment of the present disclosure is shown.
[0089] like Figure 3 As shown, S130 may include the following steps.
[0090] S310: Based on the context understanding network in the pre-trained large language model, position parsing and context extraction are performed on the target audio features to obtain context features in the target audio features.
[0091] Among them, the context understanding network refers to a network structure that has the function of understanding the location of target audio features and extracting context.
[0092] Among them, the specific implementation method of S310 includes but is not limited to the following methods: based on the self-attention mechanism in the context understanding network, parsing the position of the target audio feature; based on the bidirectional encoding layer in the context understanding network, performing context extraction on the target audio features at different positions to obtain the context features in the target audio features.
[0093] Specifically, first, the server converts the target audio features into audio text, and captures the positions of different target audio features in the audio text through the self-attention mechanism in the context understanding network, so as to understand the positional relationship between different target audio features, and captures the before and after contextual information of target audio features at different positions through the bidirectional encoding layer in the context understanding network, for example, capturing the position, part of speech, co-occurring words, syntactic structure, context and other features of words, thereby obtaining the contextual features in the target audio features.
[0094] Therefore, the self-attention mechanism and bidirectional encoding layer in the context understanding network are used to parse the position of the target audio features and extract the context information, so as to accurately determine the context features in the target audio features, which is conducive to improving the detection accuracy of the synthetic sound.
[0095] S320: Based on the audio structure processing network in the pre-trained large language model, extract hierarchical features and inter-feature relationships of the target audio features to obtain the audio structure in the target audio features.
[0096] Among them, the audio structure processing network refers to a network structure with complex structural functions for processing target audio features, specifically capturing features at different levels and the relationship between features from the target audio features.
[0097] Among them, the specific implementation method of S320 includes but is not limited to the following methods: based on the hierarchical capture layer in the audio structure processing network, the features of different levels in the target audio features are extracted to obtain the hierarchical features of the target audio features; based on the multi-head attention mechanism in the audio structure processing network, the relationship between features of different levels is extracted to obtain the audio structure in the target audio features.
[0098] Specifically, the server uses multiple Transformer models as hierarchical capture layers in the audio structure processing network to gradually process the complex language structure in the target audio features. Each Transformer model can capture features at different levels, for example, features at the time domain, frequency domain, and acoustic features, so as to better understand the complex patterns in the target audio features and obtain the hierarchical features in the target audio features. At the same time, the server applies a multi-head attention mechanism in the audio structure processing network to capture the relationship between features at different levels from different angles, for example, capturing the relationship between the time domain, frequency domain, and acoustic features. It can also better understand the complex patterns in the target audio features, for example, understand the frequency changes in the frequency domain and the pitch changes in the acoustic features, so as to obtain the audio structure of the target audio features.
[0099] Therefore, by utilizing multiple Transformer models and multi-head attention mechanisms in the audio structure processing network, we can extract hierarchical features and inter-feature relationships of the target audio features, thereby better understanding the complex patterns in the target audio features and improving the detection accuracy of synthetic sounds.
[0100] S330: Based on the classification decision network in the pre-trained large language model, the context features and the audio structure are classified and processed to determine whether there is a synthetic sound in the audio file to be detected.
[0101] The classification decision network refers to a classifier capable of determining whether synthetic sound exists.
[0102] Specifically, the server uses the classification decision network in the pre-trained large language model to make classification decisions on context features and audio structures to output a probability distribution, and then determines whether there is synthetic sound in the audio file to be detected from the label corresponding to the maximum probability in the probability distribution.
[0103] Therefore, by utilizing the pre-trained large language model's ability to understand contextual features, obtain audio structure, and make classification decisions, it is possible to accurately determine whether there is synthetic sound in the audio file to be tested.
[0104] Figure 4 A logical schematic diagram of a synthetic sound detection method provided by an embodiment of the present disclosure is shown.
[0105] like Figure 4 As shown, the synthetic sound detection method may include the following steps:
[0106] S410: Obtain the audio file to be detected uploaded by the user.
[0107] S420: Utilize an audio feature extraction model to extract target audio features from the audio file to be detected.
[0108] Specifically, based on the pre-processing network in the audio feature extraction model, the audio file to be detected is pre-processed to obtain the pre-processing features of the audio file to be detected; based on the feature extraction network in the audio feature extraction model, the pre-processing features are extracted to obtain the initial audio features of the audio file to be detected; based on the post-processing network in the audio feature extraction model, the initial audio features are post-processed to obtain the target audio features of the audio file to be detected.
[0109] S430: Use the pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether there is synthetic sound in the audio file to be detected.
[0110] Specifically, based on the context understanding network in the pre-trained large language model, the target audio features are positionally parsed and context extracted to obtain the context features in the target audio features; based on the audio structure processing network in the pre-trained large language model, the hierarchical features and the relationship between features of the target audio features are extracted to obtain the audio structure in the target audio features; based on the classification decision network in the pre-trained large language model, the context features and audio structures are classified and processed to determine whether there is synthetic sound in the audio file to be detected.
[0111] S440: Update the audio feature extraction model using the synthetic sound detection result of the audio file to be detected.
[0112] Specifically, the synthetic sound detection result of the audio file to be detected and the target audio feature of the audio file to be detected are taken as samples, and then return to S420 to extract the updated audio feature model.
[0113] S450: Store the target audio features in a sample library.
[0114] S460. Fine-tune the pre-trained large language model using target audio features extracted from the sample library.
[0115] Specifically, the pre-trained large language model is fine-tuned using the target audio features extracted from the sample library, and the process returns to execute S430, so that the large language model is migrated to the synthetic sound detection scenario, thereby continuously improving the ability of the large language model to detect synthetic sounds.
[0116] Figure 5 A schematic structural diagram of a synthetic sound detection device provided in an embodiment of the present disclosure is shown.
[0117] like Figure 5 As shown, the synthetic sound detection device 500 may include:
[0118] The audio file acquisition module 510 is used to acquire the audio file to be detected;
[0119] The audio feature acquisition module 520 is used to extract features of the audio file to be detected by using an audio feature extraction model to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features with synthetic sound characteristics;
[0120] The synthetic sound detection module 530 is used to use a pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether there is a synthetic sound in the audio file to be detected.
[0121] A synthetic sound detection device of an embodiment of the present disclosure obtains an audio file to be detected; uses an audio feature extraction model to perform feature extraction on the audio file to be detected to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize the characteristics of synthetic sounds; uses a pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether synthetic sounds exist in the audio file to be detected. Thus, for audio files that require synthetic sound detection, the audio feature extraction model and the pre-trained large language model are successively used to extract audio features that can characterize the characteristics of synthetic sounds from the audio file to be detected and to determine whether synthetic sounds exist in the audio file to be detected, without relying on manual or other algorithms for detection, thereby improving the efficiency and accuracy of synthetic sound detection, and preventing audio files with synthetic sounds from being arbitrarily applied to business scenarios that prohibit the use of synthetic sounds, so as to improve the user experience in specific business scenarios.
[0122] In some embodiments of the present disclosure, the synthetic sound detection module 530 includes:
[0123] A context feature acquisition unit, configured to perform position parsing and context extraction on the target audio feature based on the context understanding network in the pre-trained large language model, and acquire context features in the target audio feature;
[0124] An audio structure acquisition unit, configured to extract hierarchical features and inter-feature relationships from the target audio features based on the audio structure processing network in the pre-trained large language model, and acquire the audio structure in the target audio features;
[0125] A classification processing unit is used to classify the context features and the audio structure based on the classification decision network in the pre-trained large language model to determine whether the synthetic sound exists in the audio file to be detected.
[0126] In some embodiments of the present disclosure, the context feature acquisition unit is specifically used to:
[0127] Parsing the position of the target audio feature based on the self-attention mechanism in the context understanding network;
[0128] Based on the bidirectional encoding layer in the context understanding network, context extraction is performed on the target audio features at different positions to obtain context features in the target audio features.
[0129] In some embodiments of the present disclosure, the audio structure acquisition unit is specifically used to:
[0130] Based on the hierarchical capture layer in the audio structure processing network, features at different levels in the target audio feature are extracted to obtain hierarchical features of the target audio feature;
[0131] Based on the multi-head attention mechanism in the audio structure processing network, the relationship between features at different levels is extracted to obtain the audio structure in the target audio feature.
[0132] In some embodiments of the present disclosure, the audio feature acquisition module 520 includes:
[0133] A pre-processing unit, configured to pre-process the audio file to be detected based on the pre-processing network in the audio feature extraction model, and obtain the pre-processing features of the audio file to be detected;
[0134] A feature extraction unit, configured to extract features from the pre-processing features based on a feature extraction network in the audio feature extraction model, and obtain initial audio features of the audio file to be detected;
[0135] A post-processing unit is used to post-process the initial audio features based on the post-processing network in the audio feature extraction model to obtain the target audio features of the audio file to be detected.
[0136] In some embodiments of the present disclosure, the pre-processing unit is specifically used to:
[0137] Based on the pre-emphasis processing layer in the pre-processing network, pre-emphasis processing is performed on the audio signal in the audio file to be detected to obtain a pre-emphasized audio signal;
[0138] Based on the framing layer in the pre-processing network, the pre-emphasized audio signal is divided into multiple frames of overlapping audio signals;
[0139] Based on the windowing layer in the pre-processing network, windowing processing is performed on the multiple frames of overlapping audio signals to obtain the pre-processing features of the audio file to be detected.
[0140] In some embodiments of the present disclosure, the pre-processing unit is specifically used to:
[0141] Based on the first normalization layer in the pre-processing network, the audio signal in the to-be-detected audio file is normalized to obtain a first normalized feature of the audio signal;
[0142] Based on the segmentation layer in the pre-processing network, segment the first normalized feature into a plurality of feature segments;
[0143] Based on the feature extraction layer in the pre-processing network, short-time Fourier transform or Mel-frequency cepstral coefficient processing is performed on the multiple feature segments to obtain the pre-processing features of the audio file to be detected.
[0144] In some embodiments of the present disclosure, the feature extraction unit is specifically used to:
[0145] Based on the feature conversion layer in the feature extraction network, the pre-processing features are converted from the time domain to the frequency domain;
[0146] Based on the filtering processing layer in the feature extraction network, obtaining the Mel-frequency cepstrum from the pre-processing features in the frequency domain;
[0147] Based on the logarithmic calculation layer in the feature extraction network, performing logarithmic calculation on the Mel-cepstrum to obtain the Mel-cepstrum logarithm;
[0148] Based on the linear transformation layer in the feature extraction network, linearly transform the Mel-frequency cepstrum logarithm to determine the Mel-frequency cepstrum coefficients;
[0149] Based on the Mel-frequency cepstral coefficient screening layer in the feature extraction network, the Mel-frequency cepstral coefficients are screened to obtain the initial audio features of the audio file to be detected.
[0150] In some embodiments of the present disclosure, the feature extraction unit is specifically used to:
[0151] Adding position features to the pre-processing features based on the position encoding layer in the feature extraction network;
[0152] Based on the multi-head self-attention layer in the feature extraction network, multi-head self-attention processing is performed on the pre-processed features carrying the position features to obtain self-attention features corresponding to the synthesized sounds at different time points;
[0153] Based on the feedforward network in the feature extraction network, the self-attention feature is nonlinearly transformed to obtain the initial audio feature of the audio file to be detected.
[0154] In some embodiments of the present disclosure, the post-processing unit is specifically used to:
[0155] Based on the differential coefficient calculation layer in the post-processing network, differential coefficient calculation is performed on the initial audio features at adjacent moments to obtain the target audio features of the audio file to be detected.
[0156] In some embodiments of the present disclosure, the post-processing unit is specifically used to:
[0157] Based on the pooling layer in the post-processing network, the initial audio features are processed to obtain pooling features of the initial audio features;
[0158] Based on the fully connected layer in the post-processing network, the pooled features are processed to obtain target audio features of the audio file to be detected.
[0159] In some embodiments of the present disclosure, the audio feature acquisition module 520 includes:
[0160] A noise suppression unit, used to perform noise suppression processing on the audio file to be detected, and obtain the audio file after the noise suppression processing;
[0161] The target audio feature extraction unit is used to use the audio feature extraction model to perform feature extraction on the audio file after the noise suppression process to obtain the target audio feature of the audio file to be detected.
[0162] In some embodiments of the present disclosure, the noise suppression unit is specifically configured to:
[0163] The audio file to be detected is processed using a pre-trained noise suppression model to obtain the audio file after the noise suppression processing, wherein the noise suppression model is obtained by training a deep neural network using a first sample audio containing noise and a second sample audio containing no noise.
[0164] It should be noted that Figure 5 The synthetic speech detection apparatus 500 shown can perform Figures 1 to 4 The various steps in the method embodiment shown in the figure are implemented Figures 1 to 4 The various processes and effects in the method embodiment shown are not described in detail here.
[0165] Figure 6 A schematic diagram of the structure of a server provided by an embodiment of the present disclosure is shown.
[0166] like Figure 6As shown, the server may include a processor 601 and a memory 602 storing computer program instructions.
[0167] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0168] The memory 602 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 602 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 602 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 602 is a non-volatile solid-state memory. In a particular embodiment, the memory 602 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.
[0169] The processor 601 reads and executes the computer program instructions stored in the memory 602 to perform the steps of the synthetic sound detection method provided in the embodiment of the present disclosure.
[0170] In one example, the server may further include a transceiver 603 and a bus 604. Figure 6 As shown, the processor 601, the memory 602 and the transceiver 603 are connected via a bus 604 and communicate with each other.
[0171] The bus 604 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 604 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
[0172] The following is an embodiment of a computer-readable storage medium provided in an embodiment of the present disclosure. The computer-readable storage medium and the synthetic sound detection method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the computer-readable storage medium, please refer to the embodiment of the above-mentioned synthetic sound detection method.
[0173] This embodiment provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to perform a synthetic speech detection method. The method includes:
[0174] Get the audio file to be detected;
[0175] Using an audio feature extraction model, extracting features from the audio file to be detected to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize characteristics of synthetic sound;
[0176] The pre-trained large language model is used to perform context detection and audio structure processing on the target audio features to determine whether there is synthetic sound in the audio file to be detected.
[0177] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present disclosure are not limited to the above method operations, and can also execute related operations in the synthetic sound detection method provided by any embodiment of the present disclosure.
[0178] Through the above description of the implementation methods, the technical personnel in the relevant field can clearly understand that the present disclosure can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer cloud platform (which can be a personal computer, a server, or a network cloud platform, etc.) to execute the synthetic sound detection method provided by each embodiment of the present disclosure.
[0179] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.
Claims
1. A synthetic sound detection method, characterized in that: Applied to a server, the method comprises: Get the audio file to be detected; Using an audio feature extraction model, extracting features from the audio file to be detected to obtain target audio features of the audio file to be detected, wherein the target audio features are audio features that characterize characteristics of synthetic sound; The pre-trained large language model is used to perform context detection and audio structure processing on the target audio features to determine whether there is synthetic sound in the audio file to be detected.
2. The method according to claim 1, characterized in that The method of using the pre-trained large language model to perform context detection and audio structure processing on the target audio feature to determine whether there is a synthetic sound in the audio file to be detected includes: Based on the context understanding network in the pre-trained large language model, position parsing and context extraction are performed on the target audio feature to obtain context features in the target audio feature; Based on the audio structure processing network in the pre-trained large language model, extract hierarchical features and relationships between features of the target audio features to obtain the audio structure in the target audio features; Based on the classification decision network in the pre-trained large language model, the context features and the audio structure are classified and processed to determine whether the synthetic sound exists in the audio file to be detected.
3. The method according to claim 2, characterized in that The context understanding network based on the pre-trained large language model performs position parsing and context extraction on the target audio feature to obtain context features in the target audio feature, including: Parsing the position of the target audio feature based on the self-attention mechanism in the context understanding network; Based on the bidirectional encoding layer in the context understanding network, context extraction is performed on the target audio features at different positions to obtain context features in the target audio features.
4. The method according to claim 2, characterized in that: The audio structure processing network based on the pre-trained large language model extracts hierarchical features and inter-feature relationships of the target audio features to obtain the audio structure in the target audio features, including: Based on the hierarchical capture layer in the audio structure processing network, features at different levels in the target audio feature are extracted to obtain hierarchical features of the target audio feature; Based on the multi-head attention mechanism in the audio structure processing network, the relationship between features at different levels is extracted to obtain the audio structure in the target audio feature.
5. The method according to claim 1, characterized in that The step of extracting features of the audio file to be detected by using an audio feature extraction model to obtain target audio features of the audio file to be detected includes: Based on the pre-processing network in the audio feature extraction model, pre-processing the audio file to be detected to obtain the pre-processing features of the audio file to be detected; Based on the feature extraction network in the audio feature extraction model, feature extraction is performed on the pre-processing features to obtain initial audio features of the audio file to be detected; Based on the post-processing network in the audio feature extraction model, the initial audio features are post-processed to obtain the target audio features of the audio file to be detected.
6. The method according to claim 5, characterized in that The pre-processing of the audio file to be detected based on the pre-processing network in the audio feature extraction model to obtain the pre-processing features of the audio file to be detected includes: Based on the pre-emphasis processing layer in the pre-processing network, pre-emphasis processing is performed on the audio signal in the audio file to be detected to obtain a pre-emphasized audio signal; Based on the framing layer in the pre-processing network, the pre-emphasized audio signal is divided into multiple frames of overlapping audio signals; Based on the windowing layer in the pre-processing network, windowing processing is performed on the multiple frames of overlapping audio signals to obtain the pre-processing features of the audio file to be detected.
7. The method according to claim 5, characterized in that The pre-processing of the audio file to be detected based on the pre-processing network in the audio feature extraction model to obtain the pre-processing features of the audio file to be detected includes: Based on the first normalization layer in the pre-processing network, the audio signal in the to-be-detected audio file is normalized to obtain a first normalized feature of the audio signal; Based on the segmentation layer in the pre-processing network, segment the first normalized feature into a plurality of feature segments; Based on the feature extraction layer in the pre-processing network, short-time Fourier transform or Mel-frequency cepstral coefficient processing is performed on the multiple feature segments to obtain the pre-processing features of the audio file to be detected.
8. The method according to claim 5, characterized in that The step of extracting features from the pre-processing features based on the feature extraction network in the audio feature extraction model to obtain initial audio features of the audio file to be detected includes: Based on the feature conversion layer in the feature extraction network, the pre-processing features are converted from the time domain to the frequency domain; Based on the filtering processing layer in the feature extraction network, obtaining the Mel-frequency cepstrum from the pre-processing features in the frequency domain; Based on the logarithmic calculation layer in the feature extraction network, performing logarithmic calculation on the Mel-cepstrum to obtain the Mel-cepstrum logarithm; Based on the linear transformation layer in the feature extraction network, linearly transform the Mel-frequency cepstrum logarithm to determine the Mel-frequency cepstrum coefficients; Based on the Mel-frequency cepstral coefficient screening layer in the feature extraction network, the Mel-frequency cepstral coefficients are screened to obtain the initial audio features of the audio file to be detected.
9. The method according to claim 5, characterized in that The step of extracting features from the pre-processing features based on the feature extraction network in the audio feature extraction model to obtain initial audio features of the audio file to be detected includes: Adding position features to the pre-processing features based on the position encoding layer in the feature extraction network; Based on the multi-head self-attention layer in the feature extraction network, multi-head self-attention processing is performed on the pre-processed features carrying the position features to obtain self-attention features corresponding to the synthesized sounds at different time points; Based on the feedforward network in the feature extraction network, the self-attention feature is nonlinearly transformed to obtain the initial audio feature of the audio file to be detected.
10. The method according to claim 5, characterized in that The post-processing of the initial audio features based on the post-processing network in the audio feature extraction model to obtain the target audio features of the audio file to be detected includes: Based on the differential coefficient calculation layer in the post-processing network, differential coefficient calculation is performed on the initial audio features at adjacent moments to obtain the target audio features of the audio file to be detected.
11. The method according to claim 5, characterized in that The post-processing of the initial audio features based on the post-processing network in the audio feature extraction model to obtain the target audio features of the audio file to be detected includes: Based on the pooling layer in the post-processing network, the initial audio features are processed to obtain pooling features of the initial audio features; Based on the fully connected layer in the post-processing network, the pooled features are processed to obtain target audio features of the audio file to be detected.
12. The method according to any one of claims 1 to 11, characterized in that: The step of extracting features of the audio file to be detected by using an audio feature extraction model to obtain target audio features of the audio file to be detected includes: Performing noise suppression processing on the audio file to be detected to obtain the audio file after the noise suppression processing; The audio feature extraction model is used to extract features from the audio file after the noise suppression process to obtain target audio features of the audio file to be detected.
13. The method according to claim 12, characterized in that The performing noise suppression processing on the audio file to be detected to obtain the audio file after the noise suppression processing includes: The audio file to be detected is processed using a pre-trained noise suppression model to obtain the audio file after the noise suppression processing, wherein the noise suppression model is obtained by training a deep neural network using a first sample audio containing noise and a second sample audio containing no noise.
14. A synthetic sound detection device, characterized in that: Configured on a server, the device includes: An audio file acquisition module is used to acquire the audio file to be detected; An audio feature acquisition module, used to extract features from the audio file to be detected using an audio feature extraction model, and obtain target audio features of the audio file to be detected, wherein the target audio features are audio features with synthetic sound characteristics; The synthetic sound detection module is used to use a pre-trained large language model to perform context detection and audio structure processing on the target audio features to determine whether there is a synthetic sound in the audio file to be detected.
15. A server, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method described in any one of claims 1 to 13.
16. A computer-readable storage medium having a computer program stored thereon, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method according to any one of claims 1 to 13.