Method and computing device for fake audio discrimination based on multi-domain acoustic feature fusion with small sample learning

Abstract

A method and a computing device for fake audio discrimination based on small sample learning and multi-domain acoustic feature fusion are provided. The method comprises: obtaining audio to be discriminated and preprocessing the audio to be discriminated; extracting mel-frequency cepstral coefficients, mel-spectrogram and pitch from the preprocessed audio to be discriminated; fusing the mel-frequency cepstral coefficients, the mel-spectrogram and the pitch to obtain fused features; inputting the fused features into a pre-trained classification model to generate a probability distribution of the audio to be discriminated being real or fake; and determining whether the audio to be discriminated is real or fake according to the probability distribution. In the step of extracting the mel-frequency cepstral coefficients, dynamic range compression is included; in the step of extracting the mel-spectrogram, a high-resolution mel filter bank is used; and in the step of extracting the pitch, noise-resistant processing is included. The fake audio discrimination method of the present application effectively overcomes the problems of insufficient data and insufficient generalization by small sample learning and multi-domain acoustic features.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a small sample learning based audio discrimination method, in particular, a small sample learning based multi-domain acoustic feature fusion audio discrimination method for efficiently discriminating fake audio, especially suitable for application scenarios with scarce or difficult to label audio data. BACKGROUND

[0002] Fake audio synthesis technology generally refers to the use of various algorithms and deep learning models to synthesize or tamper with audio content to generate realistic fake audio technology. The corresponding fake audio discrimination algorithm is to distinguish it from the real audio that has not been synthesized or tampered with through certain technical means.

[0003] From the synthesis method of fake audio technology, the synthesis type can be roughly divided into three categories: speech synthesis (Text-to-Speech, TTS), voice cloning (Voice Cloning), and voice conversion (Voice Conversion). Among them, speech synthesis and voice cloning are the current mainstream fake methods. Speech synthesis converts text directly into natural and fluent speech, commonly used in virtual assistants and automated broadcast systems, while voice cloning can generate similar voices to the target person based on a small number of audio samples, so it has high simulation and imitation in synthesis technology. These two methods have been widely used in generating natural and humanized voices and are considered the core technical means of fake audio.

[0004] The fake audio discrimination method generally includes three types: traditional methods based on signal processing, models based on machine learning and deep learning, and methods based on multi-modal fusion. Traditional detection methods based on frequency domain features of audio (such as short-time energy, zero-crossing rate, and spectrum graph) are effective for simple audio forgery (such as splicing and cutting), but have limited effect on deep learning generated high simulation fake audio; deep learning models based on spectrum graph or waveform graph features for audio analysis have high dependence on training data and are prone to generalization problems on different data sets or new fake technologies; multi-modal methods that combine audio and other modalities (such as video and text) for detection, such as using video lip movement information to verify the authenticity of audio, but in practical applications, there may be data missing, poor synchronization between different modalities, and other problems, limiting the application. SUMMARY

[0005] In order to overcome the problems and shortcomings of the above-mentioned fake detection methods, the present disclosure proposes a small sample learning based multi-domain acoustic feature fusion fake audio discrimination method. Thus, the deficiencies in existing fake audio discrimination methods are effectively made up.

[0006] The embodiment of the present disclosure provides a small sample learning based multi-domain acoustic feature fusion fake audio discrimination method, which comprises the following steps: acquiring audio to be discriminated, and preprocessing the audio to be discriminated; extracting mel frequency cepstral coefficients, mel spectrograms and pitches from the preprocessed audio to be discriminated; fusing the mel frequency cepstral coefficients, the mel spectrograms and the pitches to obtain fused features; inputting the fused features into a pre-trained classification model to generate a probability distribution of the audio to be discriminated being real or fake; and determining whether the audio to be discriminated is real or fake according to the probability distribution.

[0007] In the embodiment of the present disclosure, the step of extracting mel frequency cepstral coefficients, mel spectrograms and pitches comprises: performing short-time Fourier transform on each frame of the audio to be discriminated to obtain a spectrum; performing dynamic range compression on the spectrum to obtain a compressed spectrum, applying a first mel filter bank to the compressed spectrum to obtain a first mel spectrum, and performing discrete cosine transform on the first mel spectrum to obtain mel frequency cepstral coefficients; applying a second mel filter bank to the spectrum to obtain a second mel spectrum, and outputting the second mel spectrum as a mel spectrogram, wherein the number of filters of the second mel filter bank is greater than the number of filters of the first mel filter bank, and the number of filters of the second mel filter bank in the high frequency band is greater than the number of filters of the second mel filter bank in the low frequency band; and calculating the fundamental frequency of the audio to be discriminated by a self-correlation algorithm to extract the pitch.

[0008] Optionally, the step of performing dynamic range compression on the spectrum can adopt the following formula:

[0009]

[0010] wherein, is the spectrum, f represents the frequency, and C is the compression coefficient.

[0011] Optionally, the number of filters of the first mel filter bank can be 40, and the number of filters of the second mel filter bank can be 96.

[0012] Optionally, the step of calculating the fundamental frequency of the audio to be discriminated by the self-correlation algorithm to extract the pitch can comprise: calculating the autocorrelation function of each frame of the audio to be discriminated, wherein the autocorrelation function is represented as , is the autocorrelation function, and for a total of N sampling points, is the amplitude of the nth sampling point of the audio to be discriminated, is a delay, N is a positive integer, n is in the range of 0 to N-1, the noise-robust processing is performed by filtering out high-frequency noise of the autocorrelation function using a low-pass filter and identifying a main peak of the autocorrelation function using a peak detection algorithm, and the pitch is calculated according to a position of the main peak of the autocorrelation function, wherein the pitch is represented as , is the pitch, is the position of the main peak of the autocorrelation function.

[0013] Optionally, the step of fusing the mel-frequency cepstral coefficient, the mel-spectrogram and the pitch to obtain the fused feature can comprise: fusing the mel-frequency cepstral coefficient, the mel-spectrogram and the pitch to generate multi-scale fused features; and weighting the multi-scale fused features using a multi-head attention mechanism to obtain the fused feature.

[0014] Optionally, the step of fusing the mel-frequency cepstral coefficient, the mel-spectrogram and the pitch to generate multi-scale fused features can comprise: performing layer-by-layer feature extraction on each of the mel-frequency cepstral coefficient, the mel-spectrogram and the pitch using a convolutional neural network to obtain first, second and third layer features with gradually reduced resolutions; and performing multi-level feature fusion using a 1x1 convolution operation and an up-sampling operation to obtain first and second level fused features, wherein, , , is the first layer feature, is the second layer feature, is the third layer feature, represents the up-sampling, is the first level fused feature, is the second level fused feature; and performing lateral connection on the first, second level fused features and the third layer feature to obtain the multi-scale fused features, wherein the multi-scale fused features are represented as .

[0015] Optionally, the weight for the multi-head attention mechanism can be represented as wherein, is the multi-scale fused feature, is a learnable parameter for generating a scoring vector of the i-th attention head to measure the importance of the multi-scale fused feature , and the fused feature can be represented as wherein, is a weight corresponding to each of the mel-frequency cepstral coefficient, the mel-spectrogram and the pitch, The multi-scale fusion feature represents each of a mel-frequency cepstral coefficient, a mel-spectrogram, and a pitch.

[0016] Optionally, the pre-trained classification model is an EfficientNet model, and the EfficientNet model comprises: an adaptive convolution layer, configured to dynamically adjust a convolution kernel parameter to generate an adaptive feature according to the fusion feature; a hierarchical convolution architecture, comprising a plurality of convolution layers arranged in series, the plurality of convolution layers performing convolution operations on the input feature layer by layer, the input feature of a first layer of the plurality of convolution layers being the adaptive feature, and an output feature of each layer of the plurality of convolution layers being used as an input feature of a next layer; and an output layer, comprising a classification head, configured to obtain a probability distribution of the to-be-identified audio being real or fake based on the feature output by each layer of the hierarchical convolution architecture, wherein the convolution kernel parameter is obtained by the following formula: wherein, is the convolution kernel parameter, is the fusion feature, is the adaptive feature, is the convolution kernel weight dynamically adjusted according to the adaptive feature.

[0017] Optionally, the method for identifying fake audio further comprises the following steps of training the EfficientNet model based on a meta-learning algorithm: loading the pre-trained EfficientNet model; obtaining training data comprising real audio and fake audio; dividing the training data into a plurality of small sample tasks, each small sample task comprising a support set for model training and a query set for model evaluation; for each small sample task, copying the parameters of the EfficientNet model to obtain task-specific parameters, updating the task-specific parameters by gradient descent based on a task loss calculated by the support set; and calculating a query loss of each small sample task using the query set of each small sample task and the updated task-specific parameters, aggregating the query losses of all small sample tasks to obtain a meta-learning loss, and updating the parameters of the EfficientNet model based on the meta-learning loss by gradient descent.

[0018] Optionally, the task loss is calculated by a cross-entropy loss.

[0019] Optionally, the method for identifying fake audio further comprises the following steps of fine-tuning the EfficientNet model after the meta-learning training is completed: optimizing the EfficientNet model using a joint loss, wherein the joint loss comprises a classification loss and a contrastive learning loss, the classification loss is calculated according to the sample proportion of real audio and fake audio in the training data, and the contrastive learning loss is calculated by constructing positive sample pairs and negative sample pairs from the training data.

[0020] Optionally, the classification loss can be represented as:

[0021]

[0022]

[0023] wherein, denotes a classification loss, denotes a loss of real audio, denotes a loss of fake audio, and denotes a weight coefficient, and are respectively a sample number of real audio and a sample number of fake audio.

[0024] Optionally, the contrastive learning loss can be expressed as:

[0025]

[0026] wherein, denotes a contrastive learning loss, T is a temperature hyperparameter, and N denotes a sample number of training data, and are respectively a similarity of a positive sample pair and a similarity of a negative sample pair, denotes a similarity of the kth negative sample pair in N training data, and the similarity of the feature vectors is calculated using a cosine similarity:

[0027]

[0028] denotes a feature vector of the ith sample, denotes a feature vector of a sample forming a positive sample pair or a negative sample pair with a function s(·) denotes a similarity.

[0029] Optionally, the joint loss can be expressed by the following formula:

[0030]

[0031] denotes a joint loss, is a balance coefficient.

[0032] Optionally, obtaining the training data including real audio and fake audio further comprises performing a data enhancement operation on an audio signal of the training data, including at least one of the following operations: performing time domain stretching or compression on the audio signal; randomly adding a tiny time domain offset in the audio signal; obtaining a mel spectrogram based on the audio signal, and randomly masking part of the frequency area on the mel spectrogram and / or adding random frequency domain noise; obtaining a mel frequency cepstral coefficient based on the audio signal, and randomly perturbing the mel frequency cepstral coefficient; and randomly jittering the pitch of the audio signal.

[0033] Embodiments of the present disclosure provide a method for discriminating fake audio based on multi-domain acoustic feature fusion based on small sample learning, which comprises: dividing the same audio to be discriminated into multiple audio segments; obtaining the probability distribution of each of the multiple audio segments being real or fake based on the method for discriminating fake audio as described above; and determining whether the audio to be discriminated is real or fake according to all the probability distributions.

[0034] Embodiments of the present disclosure provide a computer readable storage medium storing a computer program, when the computer program is executed by a processor, a method for discriminating fake audio based on multi-domain acoustic feature fusion based on small sample learning as described above is implemented.

[0035] Embodiments of the present disclosure provide a computing device, which comprises: a processor; and a memory storing a computer program, when the computer program is executed by the processor, a method for discriminating fake audio based on multi-domain acoustic feature fusion based on small sample learning as described above is implemented.

[0036] Because there are many types of audio forgery techniques, and the generated fake audio has high simulation and imitation, in reality, if you want to collect a large amount of various types of fake data, the cost is high. According to one or more embodiments of an embodiment of the present disclosure, the method and computing device for discriminating fake audio based on multi-domain acoustic feature fusion based on small sample learning of the present disclosure aim to start from the essential difference between real audio and fake audio (i.e., real audio has unforgeable characteristics in spectral smoothness, fluent prosody, and natural details), based on small samples, from the perspective of multiple domains, design representative multi-dimensional acoustic features, use attention mechanism for feature fusion, and use subsequent deep learning model for training to improve the robustness of the model when facing various forgery methods, so as to have generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of a method for discriminating fake audio based on multi-domain acoustic feature fusion based on small sample learning according to an embodiment of the present disclosure;

[0038] Figure 2 is a flowchart of extracting mel-frequency cepstral coefficients, mel-spectrogram, and pitch according to an embodiment of the present disclosure;

[0039] Figure 3 is a schematic diagram of fusing mel-frequency cepstral coefficients, mel-spectrogram, and pitch to obtain fusion features according to an embodiment of the present disclosure;

[0040] Figure 4is a block diagram of a pre-trained EfficientNet model according to an embodiment of the present disclosure;

[0041] Figure 5 is a schematic diagram of training an EfficientNet model according to an embodiment of the present disclosure;

[0042] Figure 6 is a flowchart of a method of fake audio discrimination based on multi-domain acoustic feature fusion according to another embodiment of the present disclosure; and

[0043] Figure 7 is a block diagram of a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The following DETAILED DESCRIPTION provides implementation insights to assist in gaining a comprehensive understanding of the methods, devices, and / or systems described herein. After considering this disclosure, those skilled in the art will be able to employ the methods, devices, and / or systems described herein without undue experimentation. For example, the order of operations described herein is merely an example and is not limited to those set forth herein, but can be altered as is appreciated where the operations must take place in a different order than described, as will be appreciated after considering this disclosure. Further, the description of features known to those of skill in the art can be omitted in order to more clearly and concisely describe embodiments of the present disclosure.

[0045] The features described herein can be implemented in different ways depending upon the particular application, the explicit teaching provided herein, and / or the implicit understanding of those skilled in the relevant art(s). Various modifications and changes can be made thereto by those skilled in the art without departing from the scope of the present disclosure, which is set forth in the following claims.

[0046] As used herein, the term “and / or” includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0047] Although terms such as “first,” “second,” and “third” can be used herein to describe various elements, components, regions, layers, or sections, these elements, components, regions, layers, or sections should not be limited by these terms. Instead, these terms are only used to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, the first element, the first component, the first region, the first layer, or the first section referred to in the examples described herein can also be referred to as the second element, the second component, the second region, the second layer, or the second section without departing from the teachings of the examples.

[0048] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including" and "has" are not intended to exclude many embodiments of the disclosure that do not so include, e.g., embodied in a written document application, but that do include further components in addition, or in the alternative, to a specific one disclosed. Numerous specific embodiments of the disclosure have been set forth herein, described in the context of the examples and will be readily apparent to those skilled in the art, from this disclosure, given the general principles of the disclosure. It is intended that the application should not be limited to the examples described but carry out all aspects of the application nonetheless. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the present disclosure contained within the scope of the claims.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs when read in light of the entire disclosure. Terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0050] Further, in the description of the examples, detailed descriptions of well-known related structures or functions are omitted when it is deemed that such detailed description will cause ambiguous interpretation of the present disclosure.

[0051] Figure 1 is a flowchart of a method of fake audio discrimination based on multi-domain acoustic feature fusion according to an embodiment of the present disclosure. Figure 2 is a flowchart of extracting mel-frequency cepstral coefficients, mel-spectrograms, and pitch according to an embodiment of the present disclosure. Figure 3 is a schematic diagram of fusing mel-frequency cepstral coefficients, mel-spectrograms, and pitch to obtain fused features according to an embodiment of the present disclosure. Figure 4 is a block diagram of a pre-trained EfficientNet model according to an embodiment of the present disclosure.

[0052] Few-shot learning (FSL) is a machine learning method that aims to train a model with a small number of labeled samples, such as tens or hundreds of data, so that it can learn and reason effectively in the case of data scarcity. In the present disclosure, the few-shot learning method is embodied in two aspects: feature transfer and contrastive learning. Few-shot learning often relies on a model that has been pre-trained on a large dataset, and applies the knowledge learned from the large dataset to the small sample task through transfer learning. By fine-tuning the model, it can adapt to new tasks or data. On the other hand, the contrastive learning method maximizes the difference between different class samples and minimizes the difference between the same class samples, helping the model to learn more robust feature representations. Contrastive learning is particularly suitable for few-shot learning, as it can learn enough feature information from a small amount of labeled data.

[0053] Referring to Figure 1 In step S100, the audio to be identified is obtained and preprocessed.

[0054] In an embodiment, the uniform sampling rate of the input audio is set to 16000 Hz, and other parameter settings for sampling include at least one of the following settings: 96 Mel frequency bands (MEL_BINS = 96) to achieve high frequency resolution; frame step length of 160 sampling points (HOP_LENGTH = 160, equivalent to 10 milliseconds) to ensure time resolution; maximum width of 400 steps (MAX_WIDTH = 400) to ensure consistent input size. If the audio length is insufficient, zero padding can be performed, and if it exceeds, it can be cropped for subsequent feature extraction and model processing.

[0055] Referring to Figure 1 and Figure 2 In step S200, the mel frequency cepstral coefficients, the mel spectrogram, and the pitch are extracted from the preprocessed audio to be identified.

[0056] Mel frequency cepstral coefficients (MFCC) can simulate the characteristics of human ear frequency perception and are widely used in speech recognition and audio analysis. Fake audio often has distortion in the low frequency band (such as the fundamental frequency and harmonics), but traditional MFCC calculation may not effectively capture these distortions, especially in cases where the dynamic range is large.

[0057] Therefore, in an embodiment of the present disclosure, referring to Figure 2 In the step of extracting mel frequency cepstral coefficients, a short-time Fourier transform is performed on each frame of the audio to be identified to obtain a frequency spectrum; dynamic range compression is performed on the frequency spectrum to obtain a compressed frequency spectrum, a first mel filter bank is applied to the compressed frequency spectrum to obtain a first mel spectrum, and a discrete cosine transform is performed on the first mel spectrum to obtain the mel frequency cepstral coefficients.

[0058] As an example, the more specific algorithm steps for extracting mel frequency cepstral coefficients are as follows:

[0059] S201a, pre-emphasis processing is performed on the input audio to be identified to enhance high frequency components;

[0060] S201b, frame the audio to be identified, and window each frame of signal (such as Hamming window);

[0061] S201c, perform fast Fourier transform (FFT) on each frame of signal to obtain a frequency spectrum;

[0062] S201d, perform dynamic range compression on the frequency spectrum, the formula is as follows:

[0063]

[0064] wherein, is the frequency spectrum, f represents the frequency, and C is the compression coefficient;

[0065] S201e, passing the compressed frequency spectrum through a first mel filter bank to obtain a first mel spectrum; and

[0066] S201f, performing a discrete cosine transform (DCT) on the first mel spectrum to obtain MFCC coefficients.

[0067] Dynamic range compression is used to reduce the dynamic range of a signal, making the difference between weak and strong signals more apparent. The introduction of dynamic range compression in MFCC calculation can enhance the detail information of the low frequency band, thereby more effectively capturing the distortion of the fake audio.

[0068] The mel spectrum is a spectrum obtained by converting the mel scale, which can provide detailed frequency distribution information. Fake audio often has spectral abnormalities in the high frequency band (such as noise, formants), but the resolution of the mel filter bank may not be sufficient to capture these abnormalities.

[0069] To this end, in an embodiment of the present disclosure, with reference to Figure 2 step S202 in the step of extracting the mel spectrum, a short-time Fourier transform is performed on each frame of the audio to be identified to obtain a frequency spectrum; a second mel filter bank is applied to the frequency spectrum to obtain a second mel spectrum, and the second mel spectrum is output as a mel spectrum, wherein the number of filters of the second mel filter bank is greater than the number of filters of the first mel filter bank, and the number of filters of the second mel filter bank in the high frequency band is greater than the number of filters of the second mel filter bank in the low frequency band.

[0070] As an example, the more specific algorithm steps for extracting the mel spectrum are as follows:

[0071] S202a, pre-emphasis processing is performed on the input audio signal to be identified to enhance the high frequency components;

[0072] S202b, the audio to be identified is framed, and each frame of signal is windowed (such as a Hamming window);

[0073] S202c, a fast Fourier transform (FFT) is performed on each frame of signal to obtain a frequency spectrum;

[0074] S202d, a high-resolution second mel filter bank is designed, for example, the number of filters is increased and / or the filter bandwidth is adjusted so that it is more dense in the high frequency band;

[0075] S202e, a second mel filter bank is applied to the frequency spectrum to obtain a second mel spectrum, obtaining a high-resolution second mel spectrum.

[0076] As an example, the number of filters of the first mel filter bank is 40, and the number of filters of the second mel filter bank is 96.

[0077] By way of example only, a mel filter bank is typically a set of triangular filters distributed on a mel frequency scale, with frequencies The range of mel scale is typically 0-8000 Hz (depending on the sampling rate). The mel scale is defined as follows:

[0078]

[0079] A mel filter bank extracts a mel spectrogram of an audio signal by weighting the audio spectrum with a plurality of adjacent triangular filters.

[0080] The bandwidth of a filter refers to the frequency range covered by the filter, i.e., the bandwidth of a triangular filter is the difference between the left frequency boundary and the right frequency boundary of the filter. For example, assuming that the center frequency of a filter is 2000 Hz, the left boundary frequency is 1900 Hz, and the right boundary frequency is 2100 Hz, then the bandwidth is 2100 Hz-1900 Hz = 200 Hz. The density of filters can be changed by increasing the number of filters or adjusting the bandwidth of filters. For example, in the range of 1000 Hz, if the originally distributed 10 filters are changed to 20 filters, then the density increases from 1 / 100 Hz to 2 / 100 Hz. In embodiments of the present disclosure, the high frequency band can be made more dense by adjusting the number of filters and / or adjusting the bandwidth of filters. For example, 8 filters are arranged in the frequency range of 0-1000 Hz, with an average bandwidth of 125 Hz, 28 filters are arranged in the frequency range of 1000-4000 Hz, with an average bandwidth of 107 Hz, and 60 filters are arranged in the frequency range of 4000-8000 Hz, with an average bandwidth of 67 Hz.

[0081] A high-resolution mel filter bank improves the resolution of the mel spectrogram by increasing the number of filters and adjusting the bandwidth of the filters. The mel spectrogram obtained by the high-resolution mel filter bank can more finely capture the spectral abnormalities of the high frequency band, thereby more effectively revealing the distortion of the forged audio.

[0082] Pitch is a feature that reflects the change of the fundamental frequency of audio, and is particularly sensitive to unnaturalness in forged audio. Traditional pitch calculation methods are easily disturbed in a noisy environment, resulting in inaccurate calculation results.

[0083] Therefore, in embodiments of the present disclosure, reference is made to Figure 2In step S203, the pitch of the audio to be identified is extracted by calculating the fundamental frequency of the audio to be identified by using an autocorrelation algorithm.

[0084] As an example, the more specific algorithm steps of extracting the pitch are as follows:

[0085] S203a, pre-emphasizing the input audio signal to be identified to enhance the high-frequency components.

[0086] S203b, framing the audio signal and windowing each frame of the signal (such as a Hamming window).

[0087] S203c, calculating the autocorrelation function of each frame of the signal, as follows:

[0088]

[0089] wherein, for a total of N sampling points, is the amplitude of the audio to be identified at the nth sampling point, is the delay, N is a positive integer, and n ranges from 0 to N-1.

[0090] S203d, performing noise-resistant processing by filtering out the high-frequency noise of the autocorrelation function using a low-pass filter and identifying the main peak of the autocorrelation function using a peak detection algorithm;

[0091] S203e, calculating the pitch according to the position of the main peak of the autocorrelation function, as follows:

[0092]

[0093] wherein, is the position of the main peak of the autocorrelation function.

[0094] The noise-resistant algorithm according to the embodiments of the present disclosure can accurately calculate the pitch in a noisy environment. The noise-resistant algorithm introduced in the pitch calculation can improve the robustness of the pitch feature, thereby more effectively capturing the abnormal fundamental frequency of the fake audio.

[0095] The present disclosure combines the mel-frequency cepstral coefficient, the mel-spectrum diagram and the pitch feature, wherein the mel-frequency cepstral coefficient and the mel-spectrum diagram belong to the frequency domain feature, and the pitch feature belongs to the time domain feature. The mel-frequency cepstral coefficient can capture the distortion of the fake audio in the low frequency band, the mel-spectrum diagram can reveal the spectral anomaly in the high frequency band, and the pitch feature can reflect the unnatural fluctuation on the fundamental frequency. The three types of features have a synergistic effect, and a single feature or a combination of two features cannot comprehensively cover the abnormalities of the fake audio. Only the combination of the three can achieve the best identification effect. In addition, by referring to the mel-frequency cepstral coefficient, the mel-spectrum diagram and the pitch feature, the present disclosure can effectively identify the fake audio in the noisy environment. Figure 2The optimized feature extraction step makes the extracted mel-frequency cepstral coefficients, mel-spectrogram and pitch features more representative.

[0096] With reference to Figure 1 and Figure 3 In step S300, the mel-frequency cepstral coefficients, mel-spectrogram and pitch are fused to obtain fused features.

[0097] As an example, with reference to Figure 3 More specific steps to obtain the fused features include:

[0098] In step S301, the mel-frequency cepstral coefficients, mel-spectrogram and pitch are fused to generate multi-scale fused features.

[0099] In an embodiment of the present disclosure, multi-scale extraction of features can be performed first, and then a feature pyramid network structure (FPN structure) is used to fuse features of different scales.

[0100] First, in step S301a, multi-scale extraction of features is performed. For each of the mel-frequency cepstral coefficients, mel-spectrogram and pitch, layer-by-layer feature extraction is performed by a convolutional neural network (CNN) to obtain first layer features (low layer high resolution), second layer features (middle layer) and third layer features (high layer low resolution) with gradually decreasing resolution, thereby performing feature extraction of different scales on the mel-frequency cepstral coefficients, mel-spectrogram and pitch. The first layer features (high resolution) contain detailed time-frequency information, but have more noise. The third layer features (low resolution) contain global speech patterns and are more robust. To achieve multi-scale extraction of features, multi-level feature extraction can be performed using convolutions of different scales (such as 2D CNN).

[0101] Then, in step S301b, the FPN structure is used to fuse features of different scales.

[0102] Bottom-Up: Layer-by-layer feature extraction is performed using the CNN in step S301a.

[0103] Top-Down: Multi-level feature fusion is performed using a 1x1 convolution operation and an up-sampling operation to obtain first level fused features and second level fused features.

[0104]

[0105]

[0106] wherein, is the first layer feature, is the second layer feature, is a third-level feature, denotes upsampling, is a first-level fused feature, is a second-level fused feature.

[0107] Lateral Connection: the first-level fused feature, the second-level fused feature and the third-level feature are connected laterally to obtain a multi-scale fused feature, wherein the multi-scale fused feature is denoted as .

[0108] In step S302, the multi-scale fused feature is weighted by using a multi-head attention mechanism (MHA) to obtain a fused feature.

[0109] In an embodiment of the present disclosure, the weight for the multi-head attention mechanism is denoted as:

[0110]

[0111] wherein, is the multi-scale fused feature, is a learnable parameter for generating a scoring vector of the i-th attention head to measure the importance of the multi-scale fused feature .

[0112] The weight of each feature domain is dynamically learned by the multi-head attention mechanism, thereby enhancing the representation ability of the key feature. The fused feature obtained by using the multi-head attention mechanism is denoted as:

[0113]

[0114] wherein, is a weight corresponding to each of the mel-frequency cepstral coefficient, the mel-spectrum diagram and the pitch, denotes the multi-scale fused feature of each of the mel-frequency cepstral coefficient, the mel-spectrum diagram and the pitch.

[0115] In the feature fusion process, introducing the FPN structure for fusion can enhance the perception ability of the model to different scale features, thereby improving the discrimination ability to the fake audio. By using the multi-head attention mechanism, the low-layer local detail features and the high-layer global information can be adaptively given appropriate weights, thereby improving the overall effect of the model and helping the model to more effectively utilize the multi-domain acoustic features (MFCC coefficients, Mel spectrum diagram, pitch).

[0116] Referring to Figure 1 and Figure 4In step S400, the fusion feature is input into the pre-trained classification model to generate a probability distribution of the to-be-authenticated audio being real or fake.

[0117] In an embodiment, the pre-trained classification model is an EfficientNet model. The EfficientNet model is a family of convolutional neural network models that simultaneously scale network depth, width, and resolution. Compared with traditional convolutional networks, the core idea of the EfficientNet is to maintain the synchronous expansion of the other two dimensions while moderately expanding in a single dimension (depth, width, or image resolution) through a compound scaling strategy, thereby achieving higher accuracy and better inference speed without significantly increasing the number of parameters. In practical applications, the EfficientNet has flexibility and universality, and its excellent performance and smaller model size provide a better deep learning solution for resource-constrained scenarios. By further combining transfer learning and fine-tuning strategies, the EfficientNet model can maintain strong generalization ability in a small amount of data scenario.

[0118] In an embodiment of the present disclosure, the EfficientNet model includes: an adaptive convolution layer that dynamically adjusts convolution kernel parameters to generate adaptive features; a hierarchical convolution architecture including a plurality of convolution layers arranged in series, the plurality of convolution layers performing convolution operations on input features layer by layer, the input feature of the first layer of the plurality of convolution layers being the adaptive feature, and the output feature of each layer of the plurality of convolution layers being used as the input feature of the next layer; and an output layer including a classification head that obtains a probability distribution of the to-be-authenticated audio being real or fake based on the features output by each layer of the hierarchical convolution architecture.

[0119] In this embodiment, the convolution kernel parameters are obtained by the following formula:

[0120]

[0121] wherein, is the convolution kernel parameter, is the fusion feature, is the convolution kernel weight dynamically adjusted according to the fusion feature .

[0122] In the embodiment, the classification head of the output layer includes a fully connected layer and a Softmax layer, which are used to complete the conversion from features to probability distribution. The fully connected layer maps the output features of the hierarchical convolutional architecture into a feature vector, and the Softmax layer converts the feature vector into a probability distribution of whether the audio to be identified is real or fake. For example, the model prediction result is y = [0.2, 0.8], which means that the model judges that the probability of the audio being "real" is 20%, and the probability of the audio being "fake" is 80%.

[0123] In the embodiment of the present disclosure, the method further includes the step of training the EfficientNet model, which will be described in detail later with reference to Figure 5

[0124] Referring to Figure 1 In step S500, it is judged according to the probability distribution whether the audio to be identified is real or fake.

[0125] In the embodiment, after obtaining the probability distribution of "real" and "fake" given by the classification model, a threshold value can be set in advance according to the task requirement: if the "fake" probability given by the classification model is higher than the threshold value, the audio is determined to be a fake audio, otherwise it is considered to be a real audio. In practical application, the threshold value can be determined by multiple experiments on the validation set, so as to take into account the false negative rate and the false positive rate, so as to realize more accurate and robust distinction between fake audio and real audio.

[0126] Figure 5 FIG. 1 is a schematic diagram of training an EfficientNet model according to an embodiment of the present disclosure.

[0127] In the embodiment of the present disclosure, the method for identifying fake audio further includes the step of training the EfficientNet model based on a meta-learning algorithm:

[0128] In step S501, a pre-trained EfficientNet model is loaded.

[0129] In step S502, training data including real audio and fake audio is obtained.

[0130] ​In the implementation process, in order to obtain the audio authenticity training sample, real audio and fake audio need to be collected at the same time. The real audio can be directly recorded in the real speech scene in the natural environment, for example, representative segments are selected from the speech material with clear pronunciation, no obvious noise or echo; or in the experimental conditions, the natural voice of the subject is collected by a high-fidelity recording device to ensure that the collected real audio has a high signal-to-noise ratio and diversified speech characteristics. In contrast, the fake audio can be synthesized or tampered with in various ways, for example, a speech synthesis (Text-to-Speech, TTS), voice conversion or voice cloning system is used to generate simulated audio similar to the target voice color. For speech synthesis, a text is usually input first, and then a synthesized speech corresponding to the synthesized model is output; and voice conversion and voice cloning will use a small number of recording samples to simulate or imitate the source audio in voice color, rhythm or speed, and finally obtain fake audio material. Through the above methods, a sample set covering various fake means can be constructed under limited data conditions, and the positive and negative sample pairs required for training and testing of the subsequent fake audio identification model are provided. As an example, there are three categories of speech synthesis, voice conversion and voice cloning, each of which prepares 200-400 samples.

[0131] In the embodiments of the present disclosure, in order to enrich the training data, a data enhancement operation can also be performed on the audio signal of the training data, including at least one of the following operations: time domain stretching or compression is performed on the audio signal; a small time domain offset is randomly added in the audio signal; a mel spectrogram is obtained based on the audio signal, and part of the frequency area on the mel spectrogram is randomly masked and / or random frequency domain noise is added; the mel frequency cepstral coefficient is obtained based on the audio signal, and the mel frequency cepstral coefficient is randomly disturbed; and the pitch of the audio signal is randomly jittered.

[0132] Data enhancement can generate more diversified training samples and alleviate the problem of insufficient data in small sample learning. By simulating different types of fake audio, data enhancement can improve the robustness and generalization ability of the model.

[0133] In step S503, the training data is divided into a plurality of small sample tasks, and each small sample task includes a support set for model training and a query set for model evaluation.

[0134] In step S504, the inner loop of meta-learning training is performed. For each small sample task, the parameters of the EfficientNet model are copied to obtain task-specific parameters, and the task-specific parameters are updated by gradient descent based on the task loss calculated by the support set. In an embodiment, the task loss is calculated by cross-entropy loss.

[0135]

[0136] wherein, represents the task loss, represents the cross-entropy loss function, represents the prediction probability distribution generated by the model for the input sample, and represents the output result of the EfficientNet model based on the input data under the condition of the task-specific parameter. represents the true class label of the input sample. Usually one-hot encoding is adopted. For example, the label is "fake audio" and the corresponding vector is [0, 1]. The cross-entropy loss is to measure the difference between the distribution predicted by the model and the true label .

[0137] The update of the task-specific parameter can be represented as:

[0138]

[0139] wherein, is the learning rate of the inner loop. is the updated task-specific parameter.

[0140] That is, no matter how many training tasks (meta-tasks) there are, the task-specific parameters of their models are assigned a common initial value at the beginning of the inner loop. Then, for a small number of training samples (support set) of the task, a series of gradient descent updates (such as one step to several steps) are performed on the model to obtain the parameters adapted to the task. After the inner loop fine-tuning, the model has produced a set of adapted parameters that can better distinguish between real and fake audio in the task.

[0141] In step S505, the outer loop of meta-learning training is performed. The query loss of each few-shot task is calculated using the query set of each few-shot task and the updated task-specific parameter, the query loss of all few-shot tasks is aggregated to obtain the meta-learning loss, and the parameters of the EfficientNet model are updated by gradient descent based on the meta-learning loss.

[0142] In an embodiment, the meta-learning loss can be represented as:

[0143]

[0144] wherein, represents the meta-learning loss, represents the query loss based on the query set under the condition of the updated task-specific parameter .

[0145] The update of the model parameters can be expressed as:

[0146]

[0147] wherein, is the learning rate of the outer loop.

[0148] Meta-learning enables the model to quickly adapt to new types of fake audio and improve the generalization ability under small sample conditions. Through the training of multiple small sample tasks, the model can learn more general feature representations, thereby enhancing the discrimination ability of fake audio.

[0149] In an embodiment of the present disclosure, the fake audio discrimination method further comprises: after the meta-learning training is completed, performing step S506 to fine-tune the EfficientNet model.

[0150] In step S506, the EfficientNet model is optimized using a joint loss, wherein the joint loss includes a classification loss and a contrastive learning loss. The classification loss is calculated according to the sample proportion of real audio and fake audio in the training data, and the contrastive learning loss is calculated by constructing positive sample pairs and negative sample pairs from the training data. The positive sample pairs include real-real sample pairs and fake-fake sample pairs, and the negative sample pairs include real-fake sample pairs.

[0151] The classification loss is expressed as:

[0152]

[0153]

[0154] wherein, represents the classification loss, represents the loss of real audio, represents the loss of fake audio, and represents a weight coefficient, and are the number of real audio samples and the number of fake audio samples, respectively.

[0155] The contrastive learning loss is expressed as:

[0156]

[0157] wherein, represents the contrastive learning loss, T is a temperature hyperparameter, and N represents the number of samples in the training data, and are the similarity of positive sample pairs and the similarity of negative sample pairs, respectively, represents the similarity of the kth negative sample pair in N training data, and the similarity of the feature vector is calculated using cosine similarity:

[0158]

[0159] wherein, represents the feature vector of the ith sample, represents the feature vector of the sample that forms a positive sample pair or a negative sample pair, and the function s(·) represents the similarity.

[0160] The temperature parameter T can control the distribution smoothness of softmax in contrast learning. A smaller T makes the softmax more sensitive to the difference in similarity (“sharp” distribution), while a larger T makes the distribution more “smooth”. The contrastive learning loss can improve the discriminative ability of the model under limited sample conditions. In contrast learning, a positive sample pair represents a sample pair that should be close in the feature space. For example, different enhanced versions of the same audio segment, samples of the same class (such as real-real sample pairs and fake-fake sample pairs), etc. A negative sample pair represents a sample pair that should be distinguished in the feature space (for example, real-fake sample pairs). The similarity of the positive sample pair is maximized so that they are close in the feature space. The similarity of the negative sample pair is minimized so that they are far apart in the feature space. Contrastive learning can make samples of the same class (e.g., real audio) gather together in the feature space. At the same time, samples of different classes (e.g., real audio and fake audio) are separated in the feature space, forming better feature discrimination ability. Even with a small amount of data, the contrastive learning loss can help the model learn the essential features of the data.

[0161] The joint loss is represented by the following formula:

[0162]

[0163] represents the joint loss, is a balance coefficient.

[0164] ​The joint loss fuses the weighted classification loss and the contrastive learning loss, so that the model improves the classification accuracy and optimizes the distribution structure of the feature space. The classification loss effectively alleviates the class imbalance problem by assigning a higher loss weight to the fake class samples, thereby improving the model's sensitivity to fake audio. The contrastive learning loss guides the model to form a compact intra-class and separated inter-class embedding structure in the feature space, thereby enhancing the model's ability to distinguish subtle tampered features. The two work together to improve the model's discriminant performance on the training set, significantly enhance its generalization ability and robustness under the condition of few samples, and especially help to improve the model's recognition effect on boundary samples and ambiguous fake audio.

[0165] Figure 6 is a flowchart of a small sample learning-based multi-domain acoustic feature fusion fake audio discrimination method according to another embodiment of the disclosure.

[0166] If the same audio is segmented or sampled in multiple segments, the "fake" probability of each segment can be weighted and averaged or maximized to further improve the stability of the judgment, and when the average or maximum probability exceeds the threshold, the entire audio can be determined as a fake audio.

[0167] As another example, the fake audio discrimination method includes:

[0168] In step S601, the same audio to be discriminated is divided into multiple audio segments.

[0169] In step S602, the probability distribution of each segment being real or fake is obtained based on the multiple audio segments respectively by Figures 1 to 5 the small sample learning-based multi-domain acoustic feature fusion fake audio discrimination method.

[0170] As an example, the average value of the probability distribution may be represented as:

[0171]

[0172] where N is the number of segments, is the probability of the i-th segment.

[0173] In step S603, the audio to be discriminated is determined to be real or fake according to all the probability distributions (e.g., the average value of the probability distribution).

[0174] In summary, according to the method for discriminating fake audio based on multi-domain acoustic feature fusion of small sample learning provided by one or more embodiments of the present disclosure, the essential differences between real audio and fake audio in terms of spectral smoothness, fluent prosody, and natural details can be fully tapped in the case of limited audio features. By adaptively fusing multi-dimensional acoustic features and dynamically weighting each feature with the help of an attention mechanism, combined with fine-tuning of a deep learning model, the model has stronger robustness and generalization when identifying different types of fake audio. That is, the present disclosure can maintain high discrimination accuracy in the case of limited data, and can adapt to various fake means, effectively improving the practicality and reliability of the audio discrimination task.

[0175] In addition, the effectiveness of the method for discriminating fake audio based on multi-domain acoustic feature fusion of small sample learning provided by the embodiments of the present disclosure has not been verified, and the following will be verified by multiple data sets.

[0176] Data set introduction:

[0177] ST-CMDS is a Chinese speech data set, containing more than 100,000 speech files, about 100 hours of speech data. The data content is mainly based on daily online voice chat and intelligent voice control statements, with 855 different speakers, male and female voices, suitable for use in various scenarios. Here, 10,000 audio data were randomly selected for testing.

[0178] aidatatang_200zh is a real Chinese Mandarin speech data set, containing 200 hours of high-quality audio data, widely used in speech recognition, speech synthesis and other fields. The total audio duration is 200 hours, with 600 speakers from different regions, genders and ages, covering quiet indoor, noisy street and other scenarios, including news, conversations, stories and other types. The sampling rate is 16 kHz, and 16-bit PCM encoding is used. Here, 10,000 audio data were randomly selected for testing.

[0179] thchs30 is an open source Chinese speech data set. A total of 30 hours. thchs30 also provides 217 Chinese phonemes with tones as labels and a dictionary corresponding to Chinese characters. According to the dictionary, each sentence can generate its phoneme annotation. Here, 5,000 audio data were randomly selected for testing.

[0180] CVoiceFake is a data set for speech forgery detection, aiming to support content privacy protection for audio deep forgery detection research. The data set contains real speech and fake speech samples generated by various advanced and classic speech cloning techniques. It contains five common languages (English, Chinese, German, French and Italian). Various advanced speech generation techniques are used, including:

[0181] Griffin-Lim: Reconstructs waveforms from spectrograms through an iterative phase estimation method, as a traditional baseline.

[0182] Multi-band MelGAN: Divides the spectrum into subbands for faster and more stable multilingual vocoder training.

[0183] Parallel WaveGAN: A non-autoregressive vocoder model that generates high-fidelity audio quickly.

[0184] Style MelGAN: Captures subtle prosodic and stylistic variations in speech, suitable for voice synthesis that requires high expressiveness.

[0185] WORLD: A statistical parametric speech synthesis system that provides fine control over spectral and prosodic features.

[0186] Only the Chinese dataset (CVoiceFake_small / zh-CN / ****) is used here. And 10% of the samples are randomly selected from the entire dataset.

[0187] Self-generated dataset, where:

[0188] OpenTTS is an open-source text-to-speech (TTS) synthesis tool that supports multiple speech synthesis techniques, including SoVITS and traditional TTS models.

[0189] SoVITS is a technology that combines voice conversion (VC) and text-to-speech (TTS). It can generate high-quality personalized speech with a small amount of target speaker voice data. SoVITS is commonly used for customized speech synthesis and can generate speech that closely resembles the target speaker's voice.

[0190] TTS is a technology that converts text into speech. Common TTS models include Tacotron, FastSpeech, WaveNet, etc. These models generate natural and fluent speech through deep learning, supporting multiple languages and voice styles.

[0191] In the verification process, the dataset named OpenTTS represents the fake data synthesized using both technologies (SoVITS + TTS).

[0192] The XunFeiYuYin dataset is a fake dataset generated using existing speech generation platforms.

[0193] The verification test results are shown in the table below.

[0194]

[0195] From the table data, it can be seen that the combination of multi-domain acoustic features of MFCC coefficients, Mel spectrogram and pitch can achieve high recognition accuracy of fake audio, and excellent performance is shown in multiple real and fake data sets. For example, on real speech data sets such as ST-CMDS, aidatatang_200zh and thchs30, the basic multi-domain feature extraction method has achieved an accuracy of 98.92%, 96.53% and 97.78% respectively. On fake data sets such as opentts, sovits and tts, it also achieved 92.06%, 99.94% and 97.88% respectively. On the other hand, after further optimization of the feature extraction method, for example, according to the dynamic range compression introduced to the MFCC coefficients, the high-resolution filter bank used for the Mel spectrogram, and the noise-resistant processing of the pitch feature, the overall recognition performance is further improved. The accuracy of the optimized method on the corresponding data sets is improved to 99.5% (ST-CMDS), 99.81% (aidatatang_200zh), 99.96% (thchs30), 100% (opentts), 99.99% (sovits) and 99.97% (tts). The maximum improvement is 9.59%. In summary, the optimized multi-domain feature extraction strategy shows higher robustness and discriminability in the discrimination task of real and fake audio, further verifying the effectiveness of the scheme in fake audio detection.

[0196] Figure 7 is a block diagram of a computing device according to an embodiment of the disclosure.

[0197] Referring to Figure 7 The computing device 700 according to an embodiment of the disclosure can include a processor 710 and a memory 720. The processor 710 can include, but is not limited to, a central processing unit (CPU), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), a system on chip (SoC), a microprocessor, an application specific integrated circuit (ASIC), etc. The memory 720 can store a computer program to be executed by the processor 710. The memory 720 includes a high-speed random access memory and / or a non-volatile computer readable storage medium. When the processor 710 executes the computer program stored in the memory 720, the fake audio discrimination method based on small sample learning and multi-domain acoustic feature fusion as described above can be implemented.

[0198] Examples of computer-readable storage media include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random- access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid-state drive (SSD), card-type memory such as a multimedia card, secure digital (SD) card, or extreme digital (XD) card, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store computer programs and any associated data, data files, and data structures in a non-transitory manner and provide the computer programs and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer programs. In one example, the computer programs and any associated data, data files, and data structures are distributed over a networked computer system such that the computer programs and any associated data, data files, and data structures are stored, accessed, and executed by one or more processors or computers in a distributed manner.

[0199] While some embodiments of the present disclosure have been presented and described, it is to be understood that the application of modifications could be made by one skilled in the art, without departing from the principles and spirit of the present disclosure, the scope of which is to be defined by the following claims and their equivalents.

Claims

1. A method for discriminating fake audio based on small sample learning and multi-domain acoustic feature fusion, characterized in that, The counterfeit audio identification method comprises the following steps: Obtaining the audio to be identified, and preprocessing the audio to be identified; Extracting the mel-frequency cepstral coefficient, mel-spectrogram and pitch from the preprocessed audio to be identified; Fusing the mel-frequency cepstral coefficient, mel-spectrogram and pitch to obtain fused features; Inputting the fused features into a pre-trained classification model to generate a probability distribution of the audio to be identified being real or fake; and Determining whether the audio to be identified is real or fake according to the probability distribution, Wherein, the step of extracting the mel-frequency cepstral coefficient, mel-spectrogram and pitch comprises: Performing short-time Fourier transform on each frame of the audio to be identified to obtain a spectrum; Performing dynamic range compression on the spectrum to obtain a compressed spectrum, applying a first mel filter bank to the compressed spectrum to obtain a first mel spectrum, and performing discrete cosine transform on the first mel spectrum to obtain the mel-frequency cepstral coefficient; Applying a second mel filter bank to the spectrum to obtain a second mel spectrum, and outputting the second mel spectrum as the mel-spectrogram, wherein the number of filters of the second mel filter bank is greater than that of the first mel filter bank, and the number of filters of the second mel filter bank in the high frequency band is greater than that in the low frequency band; and Calculating the fundamental frequency of the audio to be identified by a self-correlation algorithm to extract the pitch, wherein the step of performing dynamic range compression on the spectrum employs the following formula: wherein, is the spectrum, f denotes the frequency, and C is a compression factor, The step of calculating the fundamental frequency of the audio to be identified by using a self-correlation algorithm to extract the pitch includes: calculating a self-correlation function of each frame of the audio to be identified, wherein the self-correlation function is represented as , is the self-correlation function, for a total of N sampling points, is an amplitude of the audio to be identified at the nth sampling point, is a delay, N is a positive integer, and n ranges from 0 to N-1; performing noise reduction processing by filtering high-frequency noise of the self-correlation function using a low-pass filter and identifying a main peak of the self-correlation function using a peak detection algorithm; and calculating the pitch according to a position of the main peak of the self-correlation function, wherein the pitch is represented as , is the pitch, is the position of the main peak of the self-correlation function, and The pre-trained classification model is an EfficientNet model, and the EfficientNet model comprises: an adaptive convolution layer, which dynamically adjusts a convolution kernel parameter according to the fusion feature to generate an adaptive feature; a hierarchical convolution architecture, which comprises a plurality of convolution layers arranged in series, the plurality of convolution layers perform convolution operations on input features layer by layer, the input feature of a first layer of the plurality of convolution layers is the adaptive feature, and the output feature of each layer of the plurality of convolution layers is used as the input feature of a next layer; and an output layer, which comprises a classification head, the classification head obtains a probability distribution of the to-be-identified audio being real or fake based on the feature output by each layer of the hierarchical convolution architecture, wherein the convolution kernel parameter is obtained by the following formula: wherein, is the convolution kernel parameter, is the fusion feature, is the adaptive feature, is a convolution kernel weight dynamically adjusted according to the fusion feature.

2. The method of claim 1, wherein, The number of filters of the first mel filter bank is 40, and the number of filters of the second mel filter bank is 96.

3. The method of claim 1, wherein, The step of fusing the mel-frequency cepstral coefficient, mel-spectrogram and pitch to obtain fused features comprises: Fusing the mel-frequency cepstral coefficient, mel-spectrogram and pitch to generate multi-scale fused features; and Using a multi-head attention mechanism to weight the multi-scale fused features to obtain the fused features.

4. The method of claim 3, wherein, The step of fusing the mel-frequency cepstral coefficient, mel-spectrogram and pitch to generate multi-scale fused features comprises: Performing layer-by-layer feature extraction on each of the mel-frequency cepstral coefficient, mel-spectrogram and pitch by a convolutional neural network to obtain first, second and third layer features with gradually decreasing resolutions; The 1x1 convolution operation and the up-sampling operation are adopted to perform multi-level feature fusion to obtain a first-level fused feature and a second-level fused feature, wherein, , , is a first layer feature, is a second layer feature, is a third layer feature, denotes up-sampling, is a first-level fused feature, is a second-level fused feature; performing a lateral connection on the first-level fusion feature, the second-level fusion feature, and the third-level feature to obtain the multi-scale fusion feature, wherein the multi-scale fusion feature is represented as .​ 5. The method of claim 4, wherein, weights for the multi-head attention mechanism is represented as , wherein, is the multi-scale fusion feature, is a learnable parameter, used to generate a scoring vector of the i-th attention head, to measure the importance of the multi-scale fusion feature , and the fusion feature is represented as wherein, is a weight corresponding to each of the mel-frequency cepstral coefficients, the mel-spectrogram, and the pitch, denotes a multi-scale fusion feature of each of the mel-frequency cepstral coefficients, the mel-spectrogram, and the pitch.

6. The method of claim 1, wherein, The counterfeit audio identification method further comprises the step of training the EfficientNet model based on a meta-learning algorithm: Loading a pre-trained EfficientNet model; Obtaining training data comprising real audio and fake audio; Dividing the training data into a plurality of small sample tasks, each small sample task comprising a support set for model training and a query set for model evaluation; For each small sample task, copying the parameters of the EfficientNet model to obtain task-specific parameters, and updating the task-specific parameters by gradient descent based on the task loss calculated by the support set; and The query loss of each few-shot task is calculated using the query set and the updated task-specific parameters of the few-shot task, the query losses of all few-shot tasks are aggregated to obtain a meta-learning loss, and the parameters of the EfficientNet model are updated by gradient descent based on the meta-learning loss.

7. The method of claim 6, wherein, The task loss is calculated by cross-entropy loss.

8. The method of claim 6, wherein, The method further comprises the step of fine-tuning the EfficientNet model after the meta-learning training is completed. The EfficientNet model is optimized using a joint loss, The joint loss includes a classification loss and a contrastive learning loss, the classification loss is calculated according to the sample proportion of the real audio and the fake audio in the training data, and the contrastive learning loss is calculated by constructing positive sample pairs and negative sample pairs from the training data.

9. The method of claim 8, wherein, The classification loss is represented as: wherein, denotes a classification loss, denotes a loss for the real audio, denotes a loss for the fake audio, and denotes a weight coefficient, and are the number of samples of the real audio and the number of samples of the fake audio, respectively.

10. The method of claim 9, wherein, The contrastive learning loss is represented as: wherein, represents a contrastive learning loss, T is a temperature hyperparameter, N represents a number of samples of the training data, and are a similarity of the positive sample pair and a similarity of the negative sample pair, respectively, represents a similarity of the kth negative sample pair in N training data, and the similarity of the feature vectors is calculated using a cosine similarity: denotes a feature vector of the i-th sample, denotes a feature vector of the i-th sample, denotes a feature vector of the i-th sample, and the function s(·) denotes a similarity.

11. The method of claim 10, wherein, The joint loss is represented by the following formula: represents the joint loss, is a balancing coefficient.

12. The method of claim 6, wherein, The training data including real audio and fake audio further includes performing a data enhancement operation on the audio signal of the training data, including at least one of the following operations: Stretching or compressing the audio signal in the time domain; Randomly adding a tiny time domain offset to the audio signal; Obtaining a mel-spectrogram based on the audio signal, and randomly masking part of the frequency region on the mel-spectrogram and / or adding random frequency domain noise; Obtaining a mel-frequency cepstral coefficient based on the audio signal, and randomly perturbing the mel-frequency cepstral coefficient; and Randomly jittering the pitch of the audio signal.

13. A method for discriminating fake audio based on small sample learning multi-domain acoustic feature fusion, characterized in that, The method for identifying fake audio comprises: Dividing the same audio to be identified into multiple audio segments; Obtaining the probability distribution of each segment being real or fake based on the multiple audio segments by the method for identifying fake audio based on small sample learning and multi-domain acoustic feature fusion according to claim 1; and Determining whether the audio to be identified is real or fake according to all probability distributions.

14. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method for identifying fake audio based on small sample learning and multi-domain acoustic feature fusion according to any one of claims 1 to 13 is realized.

15. A computing device, comprising: The computing device comprises: a processor; and a memory storing a computer program, when the computer program is executed by the processor, the method for identifying fake audio based on small sample learning and multi-domain acoustic feature fusion according to any one of claims 1 to 13 is realized.

Images