Audio depth forgery detection method and device, equipment, storage medium and product

Abstract

The invention discloses an audio deep forgery detection method and device, equipment, a storage medium and a product, and the method comprises the steps: carrying out the detection of to-be-processed audio data through a pre-trained audio deep forgery detection model, and carrying out the detection of the to-be-processed audio data in the detection process of the audio deep forgery detection model, extracting audio frequency features and audio time features of the audio data based on a time-frequency double-branch structure, and enhancing breathing sound clues in the audio time features through a breathing sound feature linear modulation mechanism; performing cross attention fusion on the audio time features and the audio frequency features, and predicting the fused features to obtain a forged audio detection result; according to the method, the breathing sound feature linear modulation mechanism and the cross attention fusion mechanism are utilized, breathing sound clue enhancement and fusion of the breathing sound clues and the time-frequency features can be achieved, natural voice fine clues such as breathing sound are effectively recognized, and therefore the accuracy of audio deep forgery detection is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and product for detecting deepfake audio. Background Technology

[0002] Existing speech forgery detection technologies mainly employ multimodal feature extraction and deep learning models. This involves extracting multimodal features from synthesized speech, constructing a deep learning model to identify these features, and ultimately determining whether the speech is forged.

[0003] However, existing speech forgery detection technologies often overlook subtle non-linguistic cues such as breathing sounds, making it difficult to accurately capture the special artifacts of synthesized speech, thus affecting the accuracy of speech forgery detection. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide an audio deep forgery detection method, apparatus, device, storage medium, and product. By fusing breathing sound cues with time-frequency features, it effectively identifies subtle cues in natural speech, such as breathing sounds, thereby improving the accuracy of audio deep forgery detection.

[0005] In a first aspect, embodiments of the present invention provide an audio depth forgery detection method, comprising: Acquire the audio data to be processed; The audio data is detected by a pre-trained deep learning audio forgery detection model to obtain forged audio detection results; In the detection process of the audio deepfake detection model, audio frequency features and audio time features are extracted from the audio data based on a time-frequency dual-branch structure, and the breathing sound cues in the audio time features are enhanced through a breathing sound feature linear modulation mechanism; the audio time features and the audio frequency features are fused by cross-attention, and the fused features are predicted to obtain the fake audio detection result.

[0006] As an improvement to the above scheme, the audio deepfake detection model includes: a temporal branch network, a frequency branch network, a cross-attention fusion module, and a classification module; The temporal branching network is used to extract audio temporal features from the audio data and enhance the breathing cue in the audio temporal features through a breathing sound feature linear modulation mechanism to obtain the final audio temporal features. The frequency branching network is used to extract frequency features from the audio data based on a discriminative frequency information mining mechanism to obtain audio frequency features. The cross-attention fusion module is used to perform multi-head cross-attention fusion on the audio time features and the audio frequency features to obtain fused features; The classification module is used to predict the fused features to obtain the fake audio detection result.

[0007] As an improvement to the above scheme, the audio deepfake detection model uses a hidden discriminant loss function to update the model parameters; the hidden discriminant loss function includes one or more combinations of supervised contrast loss, center loss, and adversarial loss.

[0008] As an improvement to the above scheme, the hidden discriminant loss function is obtained by weighted summation of the supervised contrast loss term, the center loss term, and the adversarial loss term.

[0009] As an improvement to the above scheme, the temporal branch network includes: a self-supervised cross-lingual speech processing module, a sensitive layer selection module, a breath sound detection module, and a breath sound feature linear modulation module; the self-supervised cross-lingual speech processing module includes a convolutional neural network layer and multiple Transformer layers; The convolutional neural network layer is used to extract waveform temporal features from the audio data to obtain a speech waveform feature vector. Multiple Transformer layers are used to perform contextual association on the speech waveform feature vectors to obtain multiple speech feature representations; The sensitive layer selection module is used to aggregate the speech feature representations output by multiple Transformer layers according to the importance weight of each Transformer layer to obtain aggregated features; The breathing sound detection module is used to detect breathing sounds in the audio data according to a set time granularity to obtain breathing sound features; The respiratory sound feature linear modulation module is used to selectively amplify the respiratory sound cues in the aggregated features based on the respiratory sound feature linear modulation mechanism to generate the audio time feature.

[0010] As an improvement to the above scheme, the frequency branch network includes: a frequency information mining module; The frequency information mining module is used to pre-weight the audio data to enhance the high-frequency components in the audio data, and to perform bandpass convolution based on the Sinc function on the enhanced audio data to generate audio frequency features aligned with the audio time feature dimension.

[0011] As an improvement to the above scheme, the process by which the sensitivity layer selection module aggregates the speech feature representations output by multiple Transformer layers includes: An adaptive average pooling operation is performed on each of the aforementioned speech feature representations on the time axis to obtain the merged features; The merged features are input into a fully connected layer, and the importance weights of each Transformer layer are generated by the sigmoid activation function. Based on the importance weights of each Transformer layer, the speech feature representations output by each Transformer layer are weighted and summed to obtain aggregated features.

[0012] As an improvement to the above scheme, the process by which the respiratory sound feature linear modulation module selectively amplifies the respiratory sound cues in the aggregated features includes: Each breath sound mask in the breath sound features is converted using a two-layer multilayer perceptron to generate a first gating coefficient; The first gating coefficient is amplified to obtain the second gating coefficient; The second gating coefficient is element-wise multiplied with the aggregated feature to obtain the audio time feature.

[0013] Secondly, embodiments of the present invention provide an audio depth forgery detection device, comprising: The audio data acquisition module is used to acquire the audio data to be processed. The fake audio detection module is used to detect the audio data using a pre-trained deep audio fake detection model to obtain fake audio detection results; In the detection process of the audio deepfake detection model, audio frequency features and audio time features are extracted from the audio data based on a time-frequency dual-branch structure, and the breathing sound cues in the audio time features are enhanced through a breathing sound feature linear modulation mechanism; the audio time features and the audio frequency features are fused by cross-attention, and the fused features are predicted to obtain the fake audio detection result.

[0014] Thirdly, embodiments of the present invention provide an audio deepfake detection device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the audio deepfake detection method as described in any one of the first aspects.

[0015] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the audio depth spoofing detection method as described in any one of the first aspects.

[0016] Fifthly, embodiments of the present invention provide a computer program product, including a computer program or instructions that, when executed by a processor, implement the audio depth spoofing detection method as described in any one of the first aspects.

[0017] Compared to existing technologies, this invention provides an audio deepfake detection method, apparatus, device, storage medium, and product. It uses a pre-trained audio deepfake detection model to detect audio data. During the detection process, the audio data undergoes audio frequency and time feature extraction based on a time-frequency dual-branch structure. A respiratory sound feature linear modulation mechanism enhances respiratory sound cues within the audio time features. Cross-attention fusion is then performed on the audio time and frequency features, and the fused features are predicted to obtain the forged audio detection result. This invention utilizes a respiratory sound feature linear modulation mechanism and a cross-attention fusion mechanism to enhance respiratory sound cues and fuse them with time-frequency features, effectively identifying subtle natural speech cues such as breathing sounds, thereby improving the accuracy of audio deepfake detection. Attached Figure Description

[0018] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an audio depth forgery detection method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the network architecture of the audio deepfake detection model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the network architecture of the sensitive layer selection module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the network architecture of the respiratory sound feature linear modulation module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the network architecture of the cross-attention fusion module provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the network architecture of the classification module provided in an embodiment of the present invention; Figure 7 This is a structural block diagram of an audio depth forgery detection device provided in an embodiment of the present invention; Figure 8 This is a structural block diagram of an audio depth forgery detection device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It is understood that the various numerical designations used in the embodiments of this invention are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0022] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element. The term "a plurality or several" refers to two or more.

[0023] See Figure 1 , Figure 1 This is a flowchart of an audio deepfake detection method provided by an embodiment of the present invention. The audio deepfake detection method specifically includes: S11: Obtain the audio data to be processed; S12: The audio data is detected using a pre-trained deep audio forgery detection model to obtain forged audio detection results; In the detection process of the audio deepfake detection model, audio frequency features and audio time features are extracted from the audio data based on a time-frequency dual-branch structure, and the breathing sound cues in the audio time features are enhanced through a breathing sound feature linear modulation mechanism; the audio time features and the audio frequency features are fused by cross-attention, and the fused features are predicted to obtain the fake audio detection result.

[0024] It should be noted that the audio deepfake detection method described in this embodiment of the invention can be executed by terminal devices such as servers, computers, tablets, and mobile phones. First, the audio data to be processed is acquired, including but not limited to real-time voice streams (such as real-time voice during a call, real-time voice during remote account opening and transfer verification in the financial field, real-time interactive voice during live streaming, etc.) and voice files (such as voice messages in social media, call recordings, voice messages interacting with voice assistants, etc.).

[0025] The collected audio data is input into a pre-trained deep audio forgery detection model. This deep audio forgery detection model first extracts audio frequency features and audio time features from the audio data based on a time-frequency dual-branch structure. Specifically, it extracts audio time features (also known as time-domain features) from the audio data in the time branch and enhances the breath sound features in the audio time features through a linear modulation mechanism for breath sound features. It also extracts audio frequency features (also known as frequency-domain features) from the audio data in the frequency branch. Then, the audio time features and audio frequency features extracted from the two branches are fused by cross-attention, and the fused features are predicted to obtain the forgery audio detection result of whether the audio data is forged speech.

[0026] This invention utilizes a linear modulation mechanism for respiratory sound features to enhance the respiratory sound lines in the audio time features of the extracted audio data on the time branch; it also utilizes a cross-attention fusion mechanism to perform cross-attention fusion between the audio time features that enhance respiratory sound cues and the audio frequency features, thereby enhancing respiratory sound cues and fusing respiratory sound cues with time-frequency features, thus effectively identifying subtle natural speech cues such as breathing sounds and improving the accuracy of audio deep forgery detection.

[0027] In one optional embodiment, the audio deepfake detection model includes: a temporal branch network, a frequency branch network, a cross-attention fusion module, and a classification module; The temporal branching network is used to extract audio temporal features from the audio data and enhance the breathing cue in the audio temporal features through a breathing sound feature linear modulation mechanism to obtain the final audio temporal features. The frequency branching network is used to extract frequency features from the audio data based on a discriminative frequency information mining mechanism to obtain audio frequency features. The cross-attention fusion module is used to perform multi-head cross-attention fusion on the audio time features and the audio frequency features to obtain fused features; The classification module is used to predict the fused features to obtain the fake audio detection result.

[0028] like Figure 2 As shown, the audio deepfake detection model includes: a time-frequency dual-branch structure (including a time branch network and a frequency branch network), a cross-attention fusion module, and a classification module (also known as a classifier); wherein, the time branch network includes: a self-supervised cross-language speech processing (XLS-R) module, a sensitive layer selection (SLS) module, a breath sound detection module, and a breath feature-wise linear modulation (BreathFiLM) module; the frequency branch network includes: a discriminative frequency information mining (DFIM) module; On the temporal branch network of the model, the XLS-R module is used as the front-end encoder for feature extraction to extract audio temporal features from the audio data. By designing the BreathFiLM module, a linear modulation mechanism for breath sound features is introduced to enhance the breath sound cues in the audio temporal features, thus obtaining the final audio temporal features. In the frequency branch network of the model, the DFIM module is used as the front-end encoder for feature extraction to extract frequency features from the audio data and obtain audio frequency features. In the cross-attention fusion module of the model, the audio time features output by the time branch network and the audio frequency features output by the frequency branch network are fused by multi-head cross-attention to obtain fused features; In the model's classification module, the fusion features output by the cross-attention fusion module are classified and predicted to determine whether the audio data is fake speech, thus obtaining the fake audio detection result.

[0029] It should be noted that the training process of the audio deepfake detection model is existing technology and will not be elaborated here. For example, model training can be achieved by combining rule-based strategies with self-training paradigms without the need for manual annotation of labels, thus achieving efficient training.

[0030] This invention employs a time-frequency dual-branch structure, fusing time-domain features extracted by XLS-R and frequency-domain features extracted by DFIM. Dynamic fusion is achieved through a multi-head cross-attention mechanism, enabling multimodal feature extraction of audio data. This fully leverages the complementary time-frequency information of the audio data, improving the accuracy of deep forgery detection. Furthermore, by introducing a linear modulation mechanism for breath sound features, frame-level breathing cue enhancement is achieved, strengthening the time-domain features of breath sound-related regions. This effectively captures subtle artifacts caused by abnormal breathing in synthesized speech, further improving the accuracy of deep forgery detection.

[0031] Specifically, the self-supervised cross-language speech processing module includes a Convolutional Neural Network (CNN) layer and multiple Transformer layers; The convolutional neural network layer is used to extract waveform temporal features from the audio data to obtain a speech waveform feature vector. Multiple Transformer layers are used to perform contextual association on the speech waveform feature vectors to obtain multiple speech feature representations; The sensitivity layer selection module is used to aggregate the speech feature representations output by multiple Transformer layers according to the importance weight of each Transformer layer to obtain aggregated features. The breathing sound detection module is used to detect breathing sounds in the audio data according to a set time granularity to obtain breathing sound features; The respiratory sound feature linear modulation module is used to selectively amplify the respiratory sound cues in the aggregated features based on the respiratory sound feature linear modulation mechanism to generate the audio time features.

[0032] For example, the XLS-R model is built on the wav2vec 2.0 framework, combining CNN layers and Transformer layers. First, the CNN layer performs convolution and downsampling operations on the original audio data (also known as the original audio waveform) to obtain a series of speech waveform feature vectors. Subsequently, multiple Transformer layers perform more in-depth context-aware analysis on the series of speech waveform feature vectors output by the CNN layer, thereby generating richer speech feature representations and effectively extracting the speech waveform feature representations of the audio data.

[0033] In the SLS module, adaptive importance weights are assigned based on the relative importance of each Transformer layer. Learnable aggregation is performed on the speech feature representations output by all Transformer layers to obtain aggregated features in the temporal domain. Through this aggregation process in the SLS module, the model can emphasize the representations with the most information while reducing the weights of representations with lower relevance to audio depth forgery detection, thus improving the model's detection accuracy.

[0034] In the model's respiratory sound detection module, respiratory sounds are detected in the audio data according to a set time granularity, the respiratory regions in the audio data are identified, and respiratory sound features are obtained. For example, the breath sound detection module uses a Conformer (Convolution-augmented Transformer) architecture, which can achieve fine temporal resolution and frame-level speech recognition. The Conformer layer is used to capture the local spectral features and global long-term dependencies of breath sounds in the audio data, and outputs a discriminative feature representation (such as a binary mask) to achieve high-precision recognition of breath sounds. Specifically, the audio data is first normalized to audio segments of fixed length with a set time granularity (such as 4 seconds); audio data exceeding this length is truncated, while shorter audio data is lengthened by content copying. The normalized audio segments are then divided into T non-overlapping frames, which are then input into a pre-trained breath detection module to generate a binary mask (also called a breath sound mask), which can be expressed by the formula: (1); in, This represents the function of the breathing detection module; H represents the input speech segment. This represents the binary mask of frame t. This indicates that a breathing sound was detected in frame t. This indicates that no breathing sounds were detected. By integrating the binary masks of all audio segments of the audio data, the breathing sound characteristics of the audio data can be obtained.

[0035] In the linear modulation module of the model's breath sound features, the breath sound features output by the breath sound detection module are combined with the breath sound features. Based on the BreathFiLM mechanism, the breath sound cues in the aggregated features output by the SLS module are selectively amplified to generate the final audio time features.

[0036] The embodiments of the present invention can achieve frame-level breathing cue extraction. Combined with the BreathFiLM mechanism, it can effectively enhance the model's sensitivity to key weak artifact regions (such as breathing sound regions), making up for the problem that existing methods ignore breathing cues and cannot effectively capture the subtle artifact features caused by breathing abnormalities in synthesized speech.

[0037] Specifically, the process by which the sensitivity layer selection module aggregates the speech feature representations output by multiple Transformer layers includes: An adaptive average pooling operation is performed on each of the aforementioned speech feature representations on the time axis to obtain the merged features; The merged features are input into a fully connected layer, and the importance weights of each Transformer layer are generated by the sigmoid activation function. Based on the importance weights of each Transformer layer, the speech feature representations output by each Transformer layer are weighted and summed to obtain aggregated features.

[0038] like Figure 3 As shown, the SLS module mainly includes a global average pooling (Avgpool) layer, a fully connected (FC) layer, and an activation function (Sigmoid) layer. Assume the output of the XLS-R module front-end is represented as... ,in, Let represent the hidden output of the i-th Transformer layer, L be the total number of Transformer layers, T be the time length, and D be the feature dimension. To compute inter-layer dependencies, the SLS module first applies adaptive global average pooling on the time axis, as shown below: (2); The merged features will then be used. Input a fully connected (FC) layer and pass it through a sigmoid activation function. Generate standardized hierarchical importance weights: (3); in, It contains learnable importance weights for each Transformer layer. This represents the importance weight of the i-th Transformer layer.

[0039] Based on the importance weights of the levels calculated above The aggregated features are obtained by weighted summation of the hidden outputs of all Transformer layers. The specific meaning is as follows: (4).

[0040] This invention introduces a Sensitive Layer Selection (SLS) mechanism to adaptively weight and aggregate the hidden outputs of each Transformer layer in the self-supervised cross-language speech processing module XLS-R, thereby automatically selecting the most discriminative temporal information and flexibly adapting to the feature requirements of different speech data.

[0041] Specifically, the process by which the respiratory sound feature linear modulation module selectively amplifies the respiratory sound cues in the aggregated features includes: Each breath sound mask in the breath sound features is converted using a two-layer multilayer perceptron to generate a first gating coefficient; The first gating coefficient is amplified to obtain the second gating coefficient; The second gating coefficient is element-wise multiplied with the aggregated feature to obtain the audio time feature.

[0042] This invention introduces a characteristic linear modulation mechanism for breath sounds by designing a BreathFiLM module in the model. This mechanism selectively amplifies the temporal feature representation when breath sounds are present in the audio data, enabling the model to optimize its focus on subtle breathing cues without distorting speech-related information. Specifically, BreathFiLM is applied to the temporal feature representation output by the SLS module (i.e., the aforementioned aggregated features) to generate finer-grained features for the audio deepfake detection task.

[0043] like Figure 4 As shown, the respiratory sound feature linear modulation module includes a two-layer multilayer perceptron (MLP), which comprises a linear layer, a ReLU layer, and a sigmoid activation function layer. In the respiratory sound feature linear modulation module, to achieve finer dynamic feature adjustment based on the presence of respiratory sounds, a binary mask is used for each frame output by the aforementioned respiratory sound detection module. It will be processed through a two-layer, multi-layer perceptron to generate gating coefficients. (That is, the first gating coefficient mentioned above), specifically represented as follows: (5); in, These represent the weight matrices of the first and second linear layers, respectively, and ReLU() represents the ReLU activation function; represents the sigmoid activation function; breath sound features represent the input breath sound features; H represents the output dimension of the first linear layer; and D represents the output dimension of the second linear layer.

[0044] A two-layer multilayer perceptron converts a binary mask into smooth gating coefficients. This allows for proportional scaling of features. To ensure that features relevant to the respiratory region are highlighted while preserving information from non-respiratory segments, a linear modulation mechanism is introduced. This mechanism selectively adds modulated features to the original features rather than completely replacing them. Based on these principles, the final gating coefficients are generated. (That is, the second gating coefficient mentioned above), specifically represented as follows: (6); Finally, the gating coefficients are multiplied by elements. The aggregation features obtained above Multiply to obtain the audio time features The specific representation is as follows: (7); in, This represents element-wise multiplication.

[0045] This invention utilizes a breathing detection model to identify breathing regions in audio, and introduces the BreathFiLM mechanism to selectively amplify features present in breathing regions, making the model more sensitive to breathing cues. This effectively captures subtle artifact features caused by breathing abnormalities in synthesized speech, thereby improving the discrimination performance of downstream classification modules.

[0046] Specifically, the frequency information mining module is used to pre-weight the audio data to enhance the high-frequency components in the audio data, and to perform a bandpass convolution operation based on the Sinc function on the enhanced audio data to generate audio frequency features aligned with the audio time feature dimension.

[0047] Considering that frequency domain features can also be used for audio deepfake detection, this embodiment of the invention uses the spectral features of audio data as supplementary information to the temporal domain features to improve detection performance. The DFIM module mainly includes a pre-weighted operation layer, a sinc convolution (SincConv) layer, a max pooling operation layer, a batch normalization and SELU (Scaled Exponential Linear Unit) activation layer, and a linear projection layer.

[0048] As can be understood, SincConv is a sinusoidal convolutional kernel convolutional layer (also known as a sinc function-based convolutional layer). Its core principle is to replace the randomly initialized convolutional kernels of traditional convolutional layers with a mathematically defined sinc function. Instead of learning unconstrained filter weights, SincConv parameterizes each filter as a bandpass filter defined by its low and high cutoff frequencies. SincConv enables neural networks to learn frequency domain filters in an end-to-end manner. Therefore, sinusoidal convolution-based SincConv (i.e., the DFIM module) can directly extract discriminative frequency information from the raw audio waveform.

[0049] In this embodiment of the invention, the DFIM module begins with a pre-weighting operation, which enhances the high-frequency components of the input audio data, as specifically described below: (8); in, Represents the raw audio data. This indicates the signal obtained after the preweighting operation. Indicates signal length. This represents the preweighting coefficients. After the preweighting operation, DFIM applies SincConv, which uses a parameterized bandpass filter. Each bandpass filter... Defined as: (9); in, For time indexing, and These are the lower cutoff frequency and upper cutoff frequency of the bandpass filter, respectively. This represents the Hamming window. The normalized sinc function is expressed as: (10); Here, z represents the input to the sinc function.

[0050] To provide sufficient spectral resolution, the DFIM model uses 128 bandpass filters uniformly distributed across the frequency range. The output of the SincConv layer is a high-dimensional feature map capable of capturing frequency-selective representations. After taking the absolute value of the frequency feature representation output by the SincConv layer, the DFIM module employs an adaptive max-pooling operation to reduce the temporal information content of the frequency feature representation. This step preserves discriminative information while generating a more compact time-frequency representation. The features output after max-pooling are then downsampled and processed by batch normalization and a SELU activation layer, followed by linear projection, ultimately generating the audio frequency features. The audio frequency characteristics In terms of dimensionality and audio temporal features Embedding maintains alignment, thereby enabling effective integration in classification tasks.

[0051] The embodiments of the present invention adopt a time-frequency dual-branch structure. On the time branch, audio time features (i.e., time domain features) are extracted by XLS-R, and on the frequency branch, audio frequency features (i.e., frequency domain features) are extracted by SincConv+DFIM. Dynamic fusion is performed through a multi-head cross-attention mechanism, which can realize multimodal feature extraction of audio data, fully explore the complementary information of time and frequency, and improve detection performance.

[0052] For example, in the model's cross-attention module, a multi-head cross-attention mechanism can be used to establish two different feature sequences (such as the audio frequency features mentioned above). Audio temporal characteristics The connection between the two integrates them into a fusion feature. This architecture enables the model to dynamically match and fuse information from the time and frequency domains, thereby enhancing its ability to detect common artifacts in deepfake speech.

[0053] like Figure 5 As shown, given and They are projected into different subspaces using a trainable linear transformation, as shown below: (11); in, , , This is a learnable parameter matrix. Q, K, and V represent the query vector, key vector, and value vector, respectively, with Q based on... pass The projection is obtained, and K is based on pass V is obtained by projection, based on pass Obtained by projection. These represent the dimensions of the parameter matrices corresponding to the query vector, key vector, and value vector, respectively.

[0054] The multi-head cross-attention mechanism uses scaled dot product attention to calculate the relevance score, as shown below: (12); Where, the superscript of K This indicates the transpose operation. It is the dimension of the parameter matrix corresponding to the key vector.

[0055] To enhance the model's ability to capture diverse dependencies, a multi-head cross-attention mechanism divides the input features into m heads and performs parallel attention computation on each head: (13); in The specific calculation is shown in formula (12). Indicates the output projection matrix. The concatenated output is projected back into the original feature space to ensure that the output dimension is consistent with the input. For example, using... 8. Parameter settings.

[0056] like Figure 6 As shown, the classification module includes two bidirectional long short-term memory (BiLSTM) layers, a mean pooling layer, a fully connected layer (FC), and the fused features output by the cross-attention module. The input is fed into the classification module, where BiLSTM is used to capture the fused features. The system uses contextual reasoning to predict and obtain the final result of the fake audio detection.

[0057] The embodiments of the present invention employ a multi-head cross-attention mechanism to fuse time-frequency features, enabling the model to effectively integrate complementary discriminative cues between frequency domain and time domain features, significantly improving the collaborative expression capability of information from different modalities, overcoming the limitations of relying solely on single features in the time or frequency domain, thereby significantly enhancing the deepfake detection capability.

[0058] In one optional embodiment, the audio deepfake detection model uses a hidden discriminant loss function to update model parameters; the hidden discriminant loss function includes one or more combinations of supervised contrast loss, center loss, and adversarial loss.

[0059] In this embodiment of the invention, during model training, in order to help the classification module better distinguish deepfake data from real samples, the fused features obtained after cross-attention fusion are regarded as hidden representations, and a hidden discriminant loss function that promotes intra-class compactness and inter-class separation is applied.

[0060] The supervised contrastive loss term improves classification performance by guiding real-class samples to converge towards their true class in the feature space, while pushing deepfake samples (i.e., fake audio samples) away from their true class. Let... and Let these represent the sample sets for the real category and the deepfake category, respectively. This represents the fusion feature of the real-class sample u (i.e., the non-fake audio sample). Furthermore, for the set of samples of the real class... For each sample, multiple augmented samples are generated by adding scaled Gaussian noise. Specifically, the noise sampling uses a standard normal distribution. To enhance the fusion features of real-class samples, the calculation formula is as follows: ,in, Indicates augmented samples, This is a scaling factor for random sampling, used to control the noise intensity. This enhancement strategy effectively improves the diversity of real-class samples during contrastive learning, thereby enhancing the model's robustness to perturbations. The supervised contrastive loss term is defined as follows: (14); in, Represents the embedding vector and Cosine similarity between them Similarly.

[0061] Center Loss Term: While supervised contrastive loss can distinguish different categories in the feature space, its effect is relatively coarse. To further improve the compactness of intra-class features, this embodiment of the invention introduces a center loss term to reduce distance. The function definition of the center loss term between each true class sample and a learnable class center is as follows: (15); in, These represent the sample centers of the true classes. During training, the sample centers of the true classes... The update is performed using a momentum-based strategy, and the specific process is as follows: First, calculate the mean of the fused features of the true class samples in the batch. Then update the center using the following formula: (16); in, It is the momentum coefficient.

[0062] The center loss encourages true class samples to form tighter clusters in the feature space, effectively reducing intra-group variation and enabling the model to learn more compact and well-structured feature representations. This not only improves the robustness of the model's classification but also complements the coarse-grained separation mechanism forced by supervised contrastive loss.

[0063] Adversarial Loss Term: Since deepfake samples employ multiple generation methods, it is difficult to force them to form tight clusters in the feature space. Therefore, this embodiment of the invention uses adversarial loss to increase the distance between all deepfake samples and the centers of the real classes. This design not only strengthens the separation effect of classes in the feature space but also improves the model's ability to distinguish between real class samples and deepfake samples. Considering the diversity and complexity of deepfake samples, this embodiment of the invention employs a targeted enhancement strategy for optimization. Specifically, in each batch, a hybrid sample generation method is used to generate new deepfake samples, and these deepfake samples are optimized to maintain an appropriate distance from the centers of the real classes. This enriches the distribution of deepfake data, reduces overfitting to specific synthesis methods, and encourages the model to learn more robust decision boundaries. The function definition of the adversarial loss term is as follows: (17); in, This represents the fusion features of deepfake sample k. This indicates that the fused features of deepfake samples ki and kj are obtained after processing (such as normalization). . This represents the number of samples in the sample set for the deepfake category.

[0064] This invention introduces a hidden discriminant loss function composed of supervised contrastive loss, center loss, and contrastive loss. This function enhances inter-class separability and intra-class compactness from the perspective of feature space structure, which can fundamentally improve the model's generalization ability to various forgery methods and complex scenarios. This further increases the difference between deepfake and real class samples in the feature space and improves the model's discriminant performance.

[0065] Specifically, the hidden discriminant loss function is obtained by weighted summation of the supervised contrast loss term, the center loss term, and the adversarial loss term.

[0066] For example, the hidden discriminant loss function described above can be obtained by weighting the supervised contrast loss term, the center loss term, and the contrast loss term. The definition of the hidden discriminant loss function is as follows: (18); in, , This represents a hyperparameter used to balance the contributions of various loss terms.

[0067] This invention introduces a hidden discriminant loss function composed of supervised contrastive loss, center loss, and contrastive loss. This function enhances inter-class separability and intra-class compactness from the perspective of feature space structure, which can fundamentally improve the model's generalization ability to various forgery methods and complex scenarios. This further increases the difference between deepfake and real class samples in the feature space and improves the model's discriminant performance.

[0068] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: A time-frequency dual-branch architecture is adopted, extracting information from both time and frequency branches. The time branch utilizes a pre-trained XLS-R model with SLS and BreathFiLM modules. By introducing the SLS mechanism, adaptive weighted aggregation of the Transformer representations of each layer in the self-supervised XLS-R model is performed, automatically selecting the most discriminative temporal information. The BreathFiLM module introduces a frame-level breathing cue enhancement mechanism, using a breathing detection model to identify breathing regions in the audio and selectively enhancing the temporal features of relevant regions through feature-level linear modulation, effectively capturing subtle artifacts caused by breathing abnormalities in synthesized speech. XLS-R, as the basic feature encoder, has been pre-trained on 436,000 hours of multilingual unannotated audio, possessing stronger cross-lingual generalization capabilities and able to capture subtle features such as phase artifacts and prosodic abnormalities in synthesized audio. The frequency branch uses the DFIM module to extract discriminative frequency information from the original audio waveform. The features of these two branches are adaptively aligned and fused together through a cross-attention mechanism, fully exploiting the complementary time-frequency information. The fused features are then fed into the classification module for prediction, thereby improving the accuracy of deepfake audio detection.

[0069] The classification module employs BiLSTM to capture the temporal context of speech, combined with a differential learning rate strategy and a hidden discriminant loss consisting of supervised contrastive loss, center loss, and contrastive loss. This optimizes inference efficiency while maintaining model performance, addressing to some extent the issues of large model size and resource-intensive inference, making it adaptable to low-resource or real-time scenarios. Simultaneously, based on the hidden discriminant loss composed of supervised contrastive loss, center loss, and contrastive loss, it improves the separability of classes in the feature space by bringing similar samples closer together, compressing the distribution of real classes, and increasing the distance between fake samples and the centers of real classes. This alleviates the problems of sensitivity to training sample distribution, insufficient generalization ability, and overfitting with small samples during model training, significantly enhancing the separability of real and fake samples in the feature space and the model's generalization ability. Furthermore, during model training, a joint optimization strategy can be used to fine-tune the self-supervised front-end XLS-R and the classification module, combined with audio data enhancement (RawBoost, i.e., breath sound feature enhancement), to improve model robustness.

[0070] See Figure 7 , Figure 7 This is a structural block diagram of an audio depth forgery detection device provided in an embodiment of the present invention. The audio depth forgery detection device includes: Audio data acquisition module 11 is used to acquire audio data to be processed; The fake audio detection module 12 is used to detect the audio data using a pre-trained audio deep fake detection model to obtain fake audio detection results; In the detection process of the audio deepfake detection model, audio frequency features and audio time features are extracted from the audio data based on a time-frequency dual-branch structure, and the breathing sound cues in the audio time features are enhanced through a breathing sound feature linear modulation mechanism; the audio time features and the audio frequency features are fused by cross-attention, and the fused features are predicted to obtain the fake audio detection result.

[0071] In one optional embodiment, the audio deepfake detection model includes: a temporal branch network, a frequency branch network, a cross-attention fusion module, and a classification module; The temporal branching network is used to extract audio temporal features from the audio data and enhance the breathing cue in the audio temporal features through a breathing sound feature linear modulation mechanism to obtain the final audio temporal features. The frequency branching network is used to extract frequency features from the audio data based on a discriminative frequency information mining mechanism to obtain audio frequency features. The cross-attention fusion module is used to perform multi-head cross-attention fusion on the audio time features and the audio frequency features to obtain fused features; The classification module is used to predict the fused features to obtain the fake audio detection result.

[0072] In one optional embodiment, the audio deepfake detection model uses a hidden discriminant loss function to update model parameters; the hidden discriminant loss function includes one or more combinations of supervised contrast loss, center loss, and adversarial loss.

[0073] In one alternative embodiment, the hidden discriminant loss function is obtained by weighted summation of the supervised contrast loss term, the center loss term, and the adversarial loss term.

[0074] As an improvement to the above scheme, the temporal branch network includes: a self-supervised cross-lingual speech processing module, a sensitive layer selection module, a breath sound detection module, and a breath sound feature linear modulation module; the self-supervised cross-lingual speech processing module includes a convolutional neural network layer and multiple Transformer layers; The convolutional neural network layer is used to extract waveform temporal features from the audio data to obtain a speech waveform feature vector. Multiple Transformer layers are used to perform contextual association on the speech waveform feature vectors to obtain multiple speech feature representations; The sensitivity layer selection module is used to aggregate the speech feature representations output by multiple Transformer layers according to the importance weight of each Transformer layer to obtain aggregated features. The breathing sound detection module is used to detect breathing sounds in the audio data according to a set time granularity to obtain breathing sound features; The respiratory sound feature linear modulation module is used to selectively amplify the respiratory sound cues in the aggregated features based on the respiratory sound feature linear modulation mechanism to generate the audio time features.

[0075] In one optional embodiment, the frequency branch network includes: a frequency information mining module; The frequency information mining module is used to pre-weight the audio data to enhance the high-frequency components in the audio data, and to perform bandpass convolution based on the Sinc function on the enhanced audio data to generate audio frequency features aligned with the audio time feature dimension.

[0076] In one optional embodiment, the process by which the sensitivity layer selection module aggregates the speech feature representations output by multiple Transformer layers includes: An adaptive average pooling operation is performed on each of the aforementioned speech feature representations on the time axis to obtain the merged features; The merged features are input into a fully connected layer, and the importance weights of each Transformer layer are generated by the sigmoid activation function. Based on the importance weights of each Transformer layer, the speech feature representations output by each Transformer layer are weighted and summed to obtain aggregated features.

[0077] In one optional embodiment, the process by which the respiratory sound feature linear modulation module selectively amplifies the respiratory sound cues in the aggregated features includes: Each breath sound mask in the breath sound features is converted using a two-layer multilayer perceptron to generate a first gating coefficient; The first gating coefficient is amplified to obtain the second gating coefficient; The second gating coefficient is element-wise multiplied with the aggregated feature to obtain the audio time feature.

[0078] It should be noted that the working process of each module in the audio deepfake detection device described in the embodiments of the present invention can refer to the working process of the audio deepfake detection method described in the above embodiments, and the technical effect achieved is the same as that of the audio deepfake detection method described in the above embodiments, so it will not be repeated here.

[0079] See Figure 8 , Figure 8 This is a structural block diagram of an audio deepfake detection device provided in an embodiment of the present invention. The audio deepfake detection device includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the various audio deepfake detection method embodiments described above, such as steps S11 to S12.

[0080] For example, the computer program may be divided into one or more modules or units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules or units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the audio depth forgery detection device.

[0081] The audio deepfake detection device may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of an audio deepfake detection device and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the audio deepfake detection device may also include input / output devices, network access devices, buses, etc.

[0082] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the audio deepfake detection device, connecting all parts of the device via various interfaces and lines.

[0083] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the audio deepfake detection device by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0084] If the modules or units integrated into the audio deepfake detection device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0085] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0086] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting deepfake audio, characterized in that, include: Acquire the audio data to be processed; The audio data is detected by a pre-trained deep learning audio forgery detection model to obtain forged audio detection results; In the detection process of the audio deepfake detection model, audio frequency features and audio time features are extracted from the audio data based on a time-frequency dual-branch structure, and the breathing sound cues in the audio time features are enhanced through a breathing sound feature linear modulation mechanism; the audio time features and the audio frequency features are fused by cross-attention, and the fused features are predicted to obtain the fake audio detection result.

2. The audio depth forgery detection method as described in claim 1, characterized in that, The audio deepfake detection model includes: a temporal branch network, a frequency branch network, a cross-attention fusion module, and a classification module; The temporal branching network is used to extract audio temporal features from the audio data and enhance the breathing cue in the audio temporal features through a breathing sound feature linear modulation mechanism to obtain the final audio temporal features. The frequency branching network is used to extract frequency features from the audio data based on a discriminative frequency information mining mechanism to obtain audio frequency features. The cross-attention fusion module is used to perform multi-head cross-attention fusion on the audio time features and the audio frequency features to obtain fused features; The classification module is used to predict the fused features to obtain the fake audio detection result.

3. The audio depth forgery detection method as described in claim 1 or 2, characterized in that, The audio deepfake detection model uses a hidden discriminant loss function to update model parameters; the hidden discriminant loss function includes one or more combinations of supervised contrast loss, center loss, and adversarial loss.

4. The audio depth forgery detection method as described in claim 3, characterized in that, The hidden discriminant loss function is obtained by weighted summation of the supervised contrast loss term, the center loss term, and the adversarial loss term.

5. The audio depth forgery detection method as described in claim 2, characterized in that, The temporal branching network includes: a self-supervised cross-language speech processing module, a sensitivity layer selection module, a breath sound detection module, and a breath sound feature linear modulation module; the self-supervised cross-language speech processing module includes a convolutional neural network layer and multiple Transformer layers; The convolutional neural network layer is used to extract waveform temporal features from the audio data to obtain a speech waveform feature vector. Multiple Transformer layers are used to perform contextual association on the speech waveform feature vectors to obtain multiple speech feature representations; The sensitivity layer selection module is used to aggregate the speech feature representations output by multiple Transformer layers according to the importance weight of each Transformer layer to obtain aggregated features. The breathing sound detection module is used to detect breathing sounds in the audio data according to a set time granularity to obtain breathing sound features; The respiratory sound feature linear modulation module is used to selectively amplify the respiratory sound cues in the aggregated features based on the respiratory sound feature linear modulation mechanism to generate the audio time features.

6. The audio deepfake detection method as described in claim 2, characterized in that, The frequency branch network includes: a frequency information mining module; The frequency information mining module is used to pre-weight the audio data to enhance the high-frequency components in the audio data, and to perform bandpass convolution based on the Sinc function on the enhanced audio data to generate audio frequency features aligned with the audio time feature dimension.

7. The audio depth forgery detection method as described in claim 5, characterized in that, The process by which the sensitivity layer selection module aggregates the speech feature representations output by multiple Transformer layers includes: An adaptive average pooling operation is performed on each of the aforementioned speech feature representations on the time axis to obtain the merged features; The merged features are input into a fully connected layer, and the importance weights of each Transformer layer are generated by the sigmoid activation function. Based on the importance weights of each Transformer layer, the speech feature representations output by each Transformer layer are weighted and summed to obtain aggregated features.

8. The audio deepfake detection method as described in claim 5, characterized in that, The process by which the linear modulation module of respiratory sound features selectively amplifies respiratory sound cues in the aggregated features includes: Each breath sound mask in the breath sound features is converted using a two-layer multilayer perceptron to generate a first gating coefficient; The first gating coefficient is amplified to obtain the second gating coefficient; The second gating coefficient is element-wise multiplied with the aggregated feature to obtain the audio time feature.

9. An audio depth forgery detection device, characterized in that, include: The audio data acquisition module is used to acquire the audio data to be processed. The fake audio detection module is used to detect the audio data using a pre-trained deep audio fake detection model to obtain fake audio detection results; In the detection process of the audio deepfake detection model, audio frequency features and audio time features are extracted from the audio data based on a time-frequency dual-branch structure, and the breathing sound cues in the audio time features are enhanced through a breathing sound feature linear modulation mechanism; the audio time features and the audio frequency features are fused by cross-attention, and the fused features are predicted to obtain the fake audio detection result.

10. An audio deepfake detection device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the audio depth spoofing detection method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the audio depth forgery detection method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the audio depth forgery detection method according to any one of claims 1 to 8.

Images