A speech deep spoofing detection method based on gated temporal sequence encoding and joint time-frequency self-attention mechanism
Patent Information
- Application Number
- CN202611048998.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-15
- Publication Date
- 2026-09-15
Smart Images
Figure CN122761900A_ABST
Abstract
Description
Technical Field
[0001] Belonging to the field of speech signal processing, this paper relates to a speech deep forgery detection method. Specifically, it proposes a multi-feature fusion speech deep forgery detection model based on gated short-sequence coding and joint time-frequency self-attention mechanism. Background Technology
[0002] With the development of technologies such as speech synthesis, speech conversion, and speech cloning, deepfake speech has continuously improved in terms of naturalness, similarity, and concealment. This has led to a greater demand for authenticity verification in voice security scenarios such as identity authentication, remote business processing, judicial evidence collection, public safety, and telecommunications fraud prevention. Deepfake speech detection methods typically involve preprocessing the input speech signal and extracting acoustic features to obtain feature representations that characterize the speech content, speaker characteristics, or acoustic variation patterns. Then, a classifier or deep neural network is used to model the speech features to determine whether the input speech is genuine or fake.
[0003] In existing deepfake speech detection methods, early approaches typically relied on manually designed acoustic features combined with shallow classifiers for detection. For example, they used spectral features, cepstral features, or other statistical features to describe abnormal changes in the speech signal. With the development of deep learning technology, existing methods have gradually adopted convolutional neural networks, recurrent neural networks, self-supervised speech representation models, attention mechanisms, or graph attention networks to automatically learn the temporal, frequency, and time-frequency features of speech, thereby enhancing the ability to represent abnormal traces in forged speech. These methods can improve the accuracy of deepfake speech detection to a certain extent and are adaptable to different types of synthetic or transmutation attacks.
[0004] However, in actual detection, anomalous traces in deepfake speech often do not manifest as obvious overall distortion, but rather as phase distortions, energy discontinuities, or local acoustic changes distributed over a short time range. Existing detection methods based on convolutional downsampling or global feature aggregation tend to smooth short-term abrupt changes when extracting high-level speech representations, weakening or masking local fine-grained artifacts during feature propagation, resulting in insufficient model perception of short-term anomalous traces.
[0005] Meanwhile, forgery traces in speech signals are usually manifested simultaneously in temporal variations and frequency distributions, with a certain degree of coupling dependency between the two. While some existing methods can extract time-domain or frequency-domain features separately, they are insufficient in modeling the long-range correlation between the time and frequency dimensions, making it difficult to fully capture cross-dimensional discriminative information. When forged speech exhibits high naturalness in local segments, or when its anomalous features are dispersed across different time locations and frequency regions, relying solely on local convolutional receptive fields or single-dimensional attention modeling can easily lead to insufficient utilization of global time-frequency structural information.
[0006] Furthermore, while some existing detection methods based on graph attention or multi-branch fusion can model the relationships between features of different nodes or branches, they lack a closer collaborative expression between local short-term anomaly features and global time-frequency dependent features. If local artifact features are not effectively enhanced in the front-end extraction stage, subsequent graph attention fusion or classification processes will struggle to fully utilize this fine-grained discriminative information; if the global time-frequency coupling relationship is not explicitly modeled, local enhancement features will also be difficult to effectively coordinate with the overall acoustic spectrum topology.
[0007] In summary, existing technologies suffer from several shortcomings, including insufficient perception of short-term fine-grained artifacts in deep speech forgery, inadequate modeling of long-range coupling relationships between the time and frequency dimensions, and insufficient collaborative representation of local anomaly features and global time-frequency structure features. Summary of the Invention
[0008] To address the shortcomings of existing technologies, such as insufficient perception of short-term fine-grained artifacts in deep speech forgery, inadequate modeling of long-range coupling relationships between the time and frequency dimensions, and insufficient collaborative representation of local anomaly features and global time-frequency structure features, the technical solution provided by this invention is as follows: A speech deep forgery detection method based on gated short-sequence coding and joint time-frequency self-attention mechanism includes: The steps include: acquiring the speech signal to be detected, extracting features from the speech signal to be detected, and obtaining a speech feature representation; The step of performing correlation modeling on the speech feature representation within a local time range to obtain local anomaly enhancement features; The step of generating a time-frequency feature representation with a distribution relationship in time and frequency dimensions based on the local anomaly enhancement features; The steps include performing global dependency modeling on the time dimension features and frequency dimension features in the time-frequency feature representation, and fusing the modeling results to obtain the time-frequency coupled features; The step of fusing and judging the local anomaly enhancement features with the time-frequency coupling features, and outputting the authenticity determination result of the speech signal to be detected.
[0009] Furthermore, in a preferred embodiment, when extracting features from the speech signal to be detected, the speech signal to be detected is first subjected to format unification processing to obtain a standardized speech signal, and then the standardized speech signal is subjected to context feature extraction and nonlinear compression processing to obtain the speech feature representation.
[0010] Furthermore, in a preferred embodiment, when performing correlation modeling on the speech feature representation within a local time range, multiple local time segments are divided along the time dimension of the speech feature representation, and the features within the corresponding local time segments are weighted and aggregated according to the correlation between different time positions within each local time segment to obtain the local anomaly enhancement features.
[0011] Furthermore, in a preferred embodiment, the local anomaly enhancement feature is formed by channel calibration of the weighted aggregated local time segment features. The channel calibration generates channel weights based on the weighted aggregated local time segment features and adjusts the feature responses of different channels according to the channel weights.
[0012] Furthermore, in a preferred embodiment, when generating the time-frequency feature representation based on the local anomaly enhancement features, a two-dimensional feature mapping is performed on the local anomaly enhancement features to convert the local anomaly enhancement features into a feature representation that simultaneously contains time dimension distribution information and frequency dimension distribution information.
[0013] Furthermore, in a preferred embodiment, when performing global dependency modeling on the time dimension features and the frequency dimension features, the dependency relationships between different time positions in the time dimension features and the dependency relationships between different frequency positions in the frequency dimension features are calculated respectively to obtain time-dependent features and frequency-dependent features.
[0014] A speech deep forgery detection device based on gated short-sequence coding and joint time-frequency self-attention mechanism, comprising: A module for acquiring the speech signal to be detected, extracting features from the speech signal to be detected, and obtaining a speech feature representation; A module for modeling the correlation of the speech feature representation within a local time range is obtained to obtain the local anomaly enhancement feature. A module that generates a time-frequency feature representation with a distribution relationship in both time and frequency dimensions based on the local anomaly enhancement features; Global dependency modeling is performed on the time dimension features and frequency dimension features in the time-frequency feature representation, and the modeling results are fused to obtain the module of time-frequency coupled features; A module that fuses and distinguishes the local anomaly enhancement features with the time-frequency coupling features, and outputs the authenticity determination result of the detected speech signal.
[0015] A computer storage medium for storing a computer program, which, when read by the computer, is executed by the computer using the method described thereon.
[0016] A computer, including a processor and a storage medium, executes the method when the processor reads a computer program stored in the storage medium.
[0017] A computer program product, which, as a computer program, implements the method when the computer program is executed.
[0018] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows: The pre-trained self-supervised model extracts high-dimensional context-related representations from the original speech signal, enabling the input speech to no longer rely solely on manually designed spectral or statistical features for representation. Instead, it preserves speech content, articulation variations, and contextual acoustic relationships in a higher dimension. This feature representation provides a more complete source of basic features for subsequent local artifact enhancement and time-frequency dependency modeling, reducing the problem that traditional hand-crafted features struggle to cover complex nonlinear forgery traces.
[0019] The KAN layer reconstructs and compresses the high-dimensional speech representation through a nonlinear mapping based on B-spline functions, compressing the 1024-dimensional context-dependent representation into a more compact discriminative speech embedding representation. This process is not a simple linear dimensionality reduction, but rather a nonlinear reparameterization of the feature space during dimensionality compression. This reduces the interference of redundant high-dimensional features on subsequent discrimination and preserves more valuable acoustic variation information for distinguishing real speech from fake speech.
[0020] Non-overlapping sliding local window slicing divides speech embeddings into local segments within shorter timeframes, allowing the model to focus on analyzing local variations in the speech signal within a short time window, rather than directly relying on smooth features formed by large-scale convolutions or global aggregation. Since anomalous traces such as phase distortion and energy discontinuities in deepfake speech are often distributed within short timeframes, this local slicing method can increase the probability of preserving short-term fine-grained artifacts during the feature extraction stage.
[0021] The multi-head self-attention mechanism within the window computes the interdependencies between acoustic features within each short-term local segment, enabling the correlation of anomalous changes at different time points within the same local window. Compared to local scanning using only convolutional kernels, multi-head self-attention can observe short-term acoustic changes from multiple subspaces, providing a more comprehensive representation of local phase anomalies, energy mutations, and subtle discontinuities, thereby enhancing the model's ability to perceive minute forgery traces.
[0022] The channel squeezing and activation mechanism learns channel weights through global average pooling and two-layer one-dimensional convolution, enabling different feature channels to be dynamically recalibrated before entering subsequent networks. For critical channels containing forgery vulnerabilities, this mechanism can increase their weight in feature propagation; for channels with weak or redundant relationships to authenticity discrimination, it can relatively reduce their influence, thereby preventing local minor artifacts from being overwhelmed by background acoustic information during subsequent deep feature extraction.
[0023] Learnable scalar factors perform residual aggregation between the output of the gated short-time encoder and the original speech embedding, ensuring that local enhancement features do not completely replace the original speech representation. This aggregation method highlights short-term local distortion information while preserving the overall topological contour of the original spectrogram, avoiding overemphasizing local anomalies and destroying the overall speech structure information, thus maintaining a balance between local artifact enhancement and global speech representation.
[0024] A pre-activated residual block, consisting of batch normalization, asymmetric 2D convolution, and the SeLU activation function, further maps the enhanced speech embedding into a four-dimensional tensor with a spatial topological structure. The asymmetric 2D convolution can adapt to the variations in speech features across different dimensions, while the residual structure facilitates the transfer of deep features, reduces the problem of effective information decay during deep network training, and provides a more stable high-level feature foundation for subsequent time-frequency attention modeling and graph attention fusion.
[0025] The parallel temporal and frequency branches perform scaled dot product self-attention calculations on temporal and frequency domain dependencies in deep features, respectively, so that the model no longer relies solely on local convolutional receptive fields to extract information from neighboring regions. The temporal branch can focus on the continuity of changes in speech segments over long periods of time, while the frequency branch can focus on anomalous distribution relationships between different frequency regions, thus addressing the shortcomings of traditional local feature extraction methods in modeling long-range dependencies.
[0026] The gated fusion mechanism utilizes learnable parameters and a sigmoid function to weight and combine the time-domain global description vector and the frequency-domain global description vector, enabling adaptive fusion of discriminative information in both the time and frequency dimensions based on the input speech features. This mechanism avoids the problem that simple concatenation or fixed-weight fusion is ill-suited to different types of forgery attacks, allowing the model to dynamically adjust the contribution ratio of time-domain and frequency-domain information in different speech samples, thereby more fully capturing cross-dimensional time-frequency coupled features.
[0027] After the decoupled temporal and frequency domain subgraphs are input into the graph attention layer and graph pooling layer, they can express the complex relationships between different time regions, different frequency regions, and different discriminant nodes by modeling node relationships. Compared with simply using vector concatenation or ordinary fully connected classification, graph attention fusion can further highlight the node relationships that contribute more to the true / false distinction, and reduce the interference of redundant nodes on the classification results through graph pooling.
[0028] The four-branch heterogeneous graph attention fusion layer models the relationships between high-order speech discrimination embeddings from different branches, enabling local short-term artifact features, temporal global dependency features, frequency global dependency features, and cross-dimensional time-frequency coupling features to form a collaborative representation before classification. This fusion approach alleviates the problem of incomplete feature representation in a single branch, allowing the model to simultaneously consider both local fine-grained anomalies and overall time-frequency structural anomalies.
[0029] The graph maximization and graph averaging operations preserve the salient response information and overall distribution information of node features, respectively. This allows the features fused by graph attention to reflect both a few key forgery vulnerabilities and the statistical trends of overall speech features. By combining these two types of graph-level representations, the model can obtain more stable discrimination criteria when faced with forged speech with weak local anomalies or scattered anomaly distributions.
[0030] The multi-branch high-order speech discrimination embedding vector is concatenated with the joint time-frequency self-attention branch output and then fed into the classifier. This allows the classification stage to simultaneously utilize local enhancement information, graph relation fusion information, and global time-frequency coupling information. This process avoids the problem of weak discrimination criteria caused by relying solely on a single path output, enabling the differences between real and fake speech to participate in the final discrimination from multiple feature levels.
[0031] Additive angular margin loss creates clearer inter-class separations between high-level features of real and fake speech in angular space. Compared to ordinary classification loss, this loss promotes greater aggregation of features of the same category and greater separation of features of different categories, thereby improving the classifier's ability to distinguish boundary samples and highly natural fake speech. Attached Figure Description
[0032] Figure 1This is a diagram of the overall architecture of GSTF-Net; Figure 2 This is a schematic diagram of the GSTE gated short-time encoder structure. Figure 3 This is a schematic diagram of the Joint Time-Frequency Self-Attention Mechanism (JTFSA) module. Detailed Implementation
[0033] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically: Implementation Method 1: This implementation method provides a speech deep forgery detection method based on gated short-sequence coding and a joint time-frequency self-attention mechanism, including: The steps include: acquiring the speech signal to be detected, extracting features from the speech signal to be detected, and obtaining a speech feature representation; The step of performing correlation modeling on the speech feature representation within a local time range to obtain local anomaly enhancement features; The step of generating a time-frequency feature representation with a distribution relationship in time and frequency dimensions based on the local anomaly enhancement features; The steps include performing global dependency modeling on the time dimension features and frequency dimension features in the time-frequency feature representation, and fusing the modeling results to obtain the time-frequency coupled features; The step of fusing and judging the local anomaly enhancement features with the time-frequency coupling features, and outputting the authenticity determination result of the speech signal to be detected.
[0034] When extracting features from the speech signal to be detected, the speech signal to be detected is first processed to unify the format to obtain a standardized speech signal, and then the standardized speech signal is processed to extract context features and perform nonlinear compression to obtain the speech feature representation.
[0035] When modeling the correlation of the speech feature representation within a local time range, multiple local time segments are divided along the time dimension of the speech feature representation. Based on the correlation between different time positions within each local time segment, the features within the corresponding local time segment are weighted and aggregated to obtain the local anomaly enhancement features.
[0036] The local anomaly enhancement feature is formed by weighted aggregation of local time segment features and channel calibration. The channel calibration generates channel weights based on the weighted aggregation of local time segment features and adjusts the feature responses of different channels according to the channel weights.
[0037] When generating the time-frequency feature representation based on the local anomaly enhancement features, a two-dimensional feature mapping is performed on the local anomaly enhancement features to convert the local anomaly enhancement features into a feature representation that simultaneously contains time dimension distribution information and frequency dimension distribution information.
[0038] When performing global dependency modeling on the time dimension features and the frequency dimension features, the dependency relationships between different time positions in the time dimension features and the dependency relationships between different frequency positions in the frequency dimension features are calculated respectively to obtain time dependency features and frequency dependency features.
[0039] A speech deep forgery detection device based on gated short-sequence coding and joint time-frequency self-attention mechanism, comprising: A module for acquiring the speech signal to be detected, extracting features from the speech signal to be detected, and obtaining a speech feature representation; A module for modeling the correlation of the speech feature representation within a local time range is obtained to obtain the local anomaly enhancement feature. A module that generates a time-frequency feature representation with a distribution relationship in both time and frequency dimensions based on the local anomaly enhancement features; Global dependency modeling is performed on the time dimension features and frequency dimension features in the time-frequency feature representation, and the modeling results are fused to obtain the module of time-frequency coupled features; A module that fuses and distinguishes the local anomaly enhancement features with the time-frequency coupling features, and outputs the authenticity determination result of the detected speech signal.
[0040] A computer storage medium for storing a computer program, which, when read by the computer, is executed by the computer using the method described thereon.
[0041] A computer, including a processor and a storage medium, executes the method when the processor reads a computer program stored in the storage medium.
[0042] A computer program product, which, as a computer program, implements the method when the computer program is executed.
[0043] Implementation Method Two: This implementation method is a further detailed description of the technical solution provided in Implementation Method One, specifically: This embodiment provides a speech deep forgery detection method based on local artifact enhancement and time-frequency coupling modeling, used to output a speech authenticity determination result based on the input speech signal. The method includes the following steps.
[0044] The input raw speech signal undergoes preprocessing to obtain a standardized speech signal. The raw speech signal can originate from the speech file to be detected, speech acquisition devices, or speech data to be verified in a speech authentication system. During preprocessing, the sampling rate of the raw speech signal is unified, amplitude is normalized, and duration is regularized, converting speech signals from different sources, of different lengths, or with different amplitude ranges into a standardized speech signal of a uniform format. This standardized speech signal serves as the input for subsequent speech feature extraction.
[0045] A standardized speech signal is input into a pre-trained self-supervised speech feature extraction network to extract a high-dimensional context-dependent representation of the speech signal. The pre-trained self-supervised speech feature extraction network encodes the standardized speech signal based on the contextual changes in the speech waveform, forming a high-dimensional speech feature containing temporal contextual information. This high-dimensional speech feature serves as the input for subsequent nonlinear reconstruction processing.
[0046] High-dimensional speech features are nonlinearly reconstructed and their dimensions compressed to obtain a discriminative speech embedding representation. The nonlinear reconstruction network compresses redundant dimensions in the high-dimensional speech features based on learnable nonlinear mappings, while retaining acoustic variation information relevant to speech authenticity determination, thus converting the high-dimensional speech features into a low-dimensional speech embedding representation. This speech embedding representation serves as the basic input for local artifact enhancement processing.
[0047] Local temporal window features are constructed based on speech embedding representations. Specifically, non-overlapping local windows are partitioned along the temporal dimension of the speech embedding representation, dividing the continuous speech embedding representation into multiple short-term local feature segments. Each short-term local feature segment corresponds to acoustic variation information within a local time range and serves as input for short-term self-attention modeling.
[0048] Local self-attention modeling is performed on each short-time local feature segment to obtain local temporal enhancement features. Within each short-time local feature segment, the correlation between different time positions is calculated. Based on the correlation results, the acoustic features within the local time range are weighted and aggregated, enabling phase distortion, energy discontinuities, or short-time anomalous changes in the local segments to be highlighted in the local features. These local temporal enhancement features serve as input to the channel gating calibration process.
[0049] Channel gating calibration is performed on local temporal enhancement features to obtain gated local enhancement features. Channel gating calibration first provides a global description of the local temporal enhancement features along the channel dimension, then generates weights for each channel through learnable channel mapping relationships, and reweights different channels in the local temporal enhancement features according to these weights. After reweighting, locally anomalous channels related to true / false discrimination are retained or enhanced, while redundant channels with weak discrimination relationships are suppressed, forming gated local enhancement features.
[0050] The gated local enhancement features are residually aggregated with the speech embedding representation to obtain the locally enhanced speech representation. During residual aggregation, the induction strength of the gated local enhancement features is adjusted by a learnable scalar factor. The adjusted gated local enhancement features are then added to the speech embedding representation, so that local short-term anomalies and the original overall speech representation are preserved simultaneously. The locally enhanced speech representation serves as the input for deep spatial feature extraction.
[0051] Deep spatial feature extraction is performed on the locally enhanced speech representation to obtain a time-frequency spatial feature tensor. The deep spatial feature extraction network performs two-dimensional feature modeling along the time and feature dimensions of the speech representation, transforming the one-dimensional speech embedding sequence into a time-frequency spatial feature tensor with spatial topological relationships. The time-frequency spatial feature tensor contains the locally enhanced speech features and their spatial distribution relationships in the time and frequency directions, and serves as the input for global time-frequency coupling modeling.
[0052] Based on the time-frequency spatial feature tensor, time-dimensional feature sequences and frequency-dimensional feature sequences are constructed respectively. The time-dimensional feature sequence uses the time position in the time-frequency spatial feature tensor as the modeling object, and the frequency-dimensional feature sequence uses the frequency position in the time-frequency spatial feature tensor as the modeling object. The time-dimensional feature sequence and the frequency-dimensional feature sequence serve as the inputs to the time branch and the frequency branch, respectively.
[0053] Global temporal self-attention modeling is performed on the time-dimensional feature sequence to obtain time-global dependency features; global frequency self-attention modeling is performed on the frequency-dimensional feature sequence to obtain frequency-global dependency features. The time branch establishes long-range temporal dependencies based on the correlation between different time positions, and the frequency branch establishes long-range frequency dependencies based on the correlation between different frequency positions, so that the global temporal variation information and global frequency distribution information in the time-frequency spatial feature tensor are extracted respectively.
[0054] Gated fusion of temporal and frequency-global dependency features yields joint time-frequency coupled features. Gated fusion determines the contribution ratio of temporal and frequency-global dependency features to the current input speech based on learnable fusion weights, and then weights and combines them according to this contribution ratio, forming a joint representation of long-range dependencies between the time and frequency dimensions. The joint time-frequency coupled features serve as one input for subsequent fusion and discrimination.
[0055] Based on the time-frequency spatial feature tensor, time-domain and frequency-domain subgraphs are constructed, and graph attention fusion is performed on the time-domain and frequency-domain subgraphs to obtain graph fusion discriminative features. Nodes in the time-domain subgraph correspond to feature regions in the time dimension, and nodes in the frequency-domain subgraph correspond to feature regions in the frequency dimension. Graph attention fusion updates node representations based on the correlation strength between nodes, and graph pooling is used to retain node information that contributes significantly to the true / false judgment, forming graph fusion discriminative features that include node relationship information.
[0056] The graph fusion discriminative features are concatenated with the joint time-frequency coupling features to obtain a high-order speech discriminative embedding vector. During concatenation, the node relationship features obtained from graph attention fusion are combined with the cross-dimensional global features obtained from joint time-frequency self-attention, allowing local artifact enhancement information, time-domain dependency information, frequency-domain dependency information, and cross-dimensional time-frequency coupling information to all enter the classification stage. The high-order speech discriminative embedding vector serves as the input to the classifier.
[0057] A high-order speech discrimination embedding vector is input into the classifier, which outputs the speech authenticity determination result. The classifier calculates a classification score based on the high-order speech discrimination embedding vector to determine whether the input speech belongs to real or fake speech, and then determines the authenticity category of the detected speech based on the classification score. When training the classifier, a classification loss that enhances inter-class margins can be used to create clearer class boundaries between real and fake speech features in the high-level feature space.
[0058] In one optional implementation, during the preprocessing of the original speech signal, the input speech is uniformly resampled to a preset sampling rate, the speech waveform is pre-emphasized, and speech segments are trimmed or padded to a preset length before amplitude normalization. This method can reduce the impact of different sampling rates, durations, and amplitude ranges on subsequent detection processes.
[0059] In one alternative implementation, the pre-trained self-supervised speech feature extraction network employs an encoding model capable of extracting context-dependent speech representations from the original waveform, while the nonlinear reconstruction network uses a feature compression structure based on a learnable one-dimensional function mapping. When compressing feature dimensions, the nonlinear reconstruction network does not directly compress using a fixed linear projection method, but rather reconstructs acoustic information from different dimensions through learnable nonlinear mapping relationships.
[0060] In one optional implementation, local time windows are continuously divided along the time dimension according to a preset window length, and the windows do not overlap. Within each local time window, a multi-head self-attention mechanism is used to calculate the correlation between time positions from multiple feature subspaces, and the attention results of multiple subspaces are fused to obtain the local temporal enhancement features of the corresponding window.
[0061] In one alternative implementation, channel gating calibration employs a channel squeezing and excitation structure. This structure first performs one-dimensional global average pooling on the local temporal enhancement features to obtain channel description vectors, then generates channel weights through a channel mapping network containing nonlinear activations, and finally uses the channel weights to adjust the local temporal enhancement features channel by channel.
[0062] In one optional implementation, the deep spatial feature extraction network includes multiple residual feature extraction units. Each residual feature extraction unit includes normalization processing, two-dimensional convolution processing, and nonlinear activation processing. The two-dimensional convolution processing can employ asymmetric convolution kernels to adapt to different variations in the time and frequency directions, respectively. Multiple residual feature extraction units are connected sequentially, enabling the local enhanced speech representation to be transformed layer by layer into a high-level time-frequency spatial feature tensor.
[0063] In one alternative implementation, the joint time-frequency coupling modeling employs parallel time and frequency branches. The time branch expands the time-frequency spatial feature tensor according to its time position and performs self-attention computation, while the frequency branch expands the time-frequency spatial feature tensor according to its frequency position and performs self-attention computation. Subsequently, the outputs of the time branch and the frequency branch are weighted and combined using a gating fusion method.
[0064] In one optional implementation, graph attention fusion includes multiple heterogeneous fusion branches. Each fusion branch receives features from a time-domain subgraph, a frequency-domain subgraph, or a combination of both, and updates the node representation based on the node attention weights. The updated node representations are then subjected to max pooling and average pooling to obtain graph-level representations in different statistical forms, and these graph-level representations are combined into graph fusion discriminative features.
[0065] In one alternative implementation, the classifier employs a multilayer perceptron architecture and introduces random deactivation processing in the high-level features. During training, the classifier is optimized using additive angular margin loss to create a larger angular gap between the real and fake speech categories. During detection, the classifier outputs the final speech authenticity determination result based on the scores for the real speech category and the fake speech category.
[0066] Implementation Method 3, in conjunction with Appendix Figure 1-3 This embodiment describes the technical solution provided above in further detail through specific examples. Specifically: Step 1: Input Data and Feature Extraction Input the raw speech signal and perform the following operations in parallel: Preprocessing and wav2vec2.0 Feature Extraction: The original speech signal is first resampled (16000Hz), pre-emphasized (coefficient 0.97), clipped to a fixed length (64600 sampling points, approximately 4 seconds), and the waveform amplitude is normalized. The processed signal is then input into the pre-trained encoder wav2vec2.0, which maps the one-dimensional time-domain signal to a 1024-dimensional high-dimensional context-dependent speech representation. Nonlinear Reconstruction and Compression (KAN Layer): A Kolmogorov-Arnold Network (KAN) layer is introduced to process the high-dimensional features. This layer introduces a nonlinear mapping based on B-spline functions, replacing traditional linear projection with a learnable one-dimensional function mapping. This compresses the feature dimension from 1024 to 128 while reparameterizing the feature space, resulting in a more compact and discriminative speech embedding representation. .
[0067] Step 2: Feature Enhancement and Local Modeling Enhance the ability to model local small artifacts using gated short-time encoders (GSTE): First, the speech sequence is sliced into non-overlapping sliding local windows of size 16. Within each window, a 4-head self-attention mechanism is used to calculate the interdependencies of acoustic features over a short period, capturing anomalous features such as phase distortion and energy discontinuities. Channel calibration and residual aggregation: Adaptive one-dimensional average pooling is used for feature "squeezing," and channel weights are learned through a two-layer one-dimensional convolutional network containing GELU and Sigmoid activation functions. The generated channel control vectors dynamically recalibrate the features, enhancing important forgery vulnerability channels. Finally, a learnable scalar factor is used... With speech embedding representation By performing residual addition, the original overall topological outline is preserved while accurately highlighting temporal distortions.
[0068] Step 3: Global time-frequency coupling and graph attention fusion processing: This stage is responsible for capturing long-range dependencies across dimensions and making the final determination: The features are input into a six-layer two-dimensional residual backbone network to extract deep spatial representations, generating a four-dimensional tensor with spatial topology. The module constructs two parallel scale-scaling dot product self-attention branches for time and frequency, respectively modeling global dependencies in the time and frequency domains. A gated fusion mechanism is then introduced, utilizing learnable parameters. By combining the sigmoid function with the time-domain and frequency-domain global description vectors in a weighted manner, long-range time-frequency coupling features across dimensions can be captured.
[0069] Heterogeneous graph attention fusion and discriminative classification: Decoupled temporal and frequency domain subgraphs are input into the graph attention layer and graph pooling layer. Complex relationships between nodes are modeled through a four-branch heterogeneous graph attention fusion layer. Finally, the high-order speech discriminative embedding vectors of each branch are concatenated and fed into a multilayer perceptron classifier. Additive angle margin loss is used to complete the final detection of "real speech" or "fake speech".
[0070] In one embodiment: The operating system used in this embodiment is Ubuntu 22.04. The hardware computing platform is equipped with a 100-core Intel Xeon Platinum 8470Q CPU and 368GB of RAM, along with four GPUs each with 48GB of dedicated video memory. The programming language is Python 3.10, the deep learning framework is PyTorch 2.1.0, and the acceleration environment is CUDA 12.1 and cuDNN. Training uses the AdamW optimizer, with an initial learning rate of... The weight decays to The total batch size is 512, and the number of training rounds is 100.
[0071] Step 1: Data Preparation The original audio was resampled (16kHz) and pre-emphasized (0.97). A uniform fixed-length crop was performed, with the standard detection frame length set to 64,600 sampling points (approximately 4 seconds), and the waveform amplitude was scaled to [-1, 1].
[0072] Step 2: Extract features from the audio using a feature extraction network. Step 2.1 Front-end Feature Extraction and Dimensionality Reduction Dimensionality: 1024-dimensional context-related features are extracted using pre-trained wav2vec2.0, and then compressed to 128 dimensions through a KAN layer.
[0073] Step 22: Local and Two-Dimensional Space Modeling Short-term dependencies are modeled using a gated short-time encoder with a window size of 16, and two-dimensional features are extracted through a convolutional backbone network containing 6 residual blocks. Each residual block consists of BatchNorm, asymmetric two-dimensional convolution, and SeLU activation mechanism.
[0074] Steps 2 and 3: Gated Short-Time Coding First, the speech sequence is sliced into non-overlapping sliding local windows of size 16. Within each window, a 4-head self-attention mechanism is used to calculate the interdependencies of acoustic features over a short period, capturing anomalous features such as phase distortion and energy discontinuities. Adaptive one-dimensional average pooling is used for feature "squeezing," and a two-layer one-dimensional convolutional network containing the GELU activation function is used to learn channel weight scalars. The generated channel control vectors dynamically recalibrate the features, enhancing important forgery vulnerability channels. Finally, a learnable scalar factor is used... With speech embedding representation The residuals are summed and then fed into the convolutional backbone network.
[0075] Step 3: Model Training and Evaluation Training strategy: The additive angle margin loss (AAM-Softmax) is used as the loss function to enhance the inter-class diversity of features on the hypersphere. A OneCycleLR learning rate scheduling strategy is employed to dynamically optimize the convergence process, and Dropout is applied to high-level features to alleviate overfitting.
[0076] Evaluation metrics: The training, development, and evaluation sets were defined using the official ASVspoof2019 Logical Access (LA) dataset. Evaluation metrics included equal error rate (EER) and minimum concatenation detection cost function (mint-DCF).
[0077] Step 4: Comparison of Experimental Results and Ablation Experiment Our method performs excellently on the ASVspoof2019LA evaluation set (which includes 13 attacks from A07 to A19).
[0078]
[0079] Example 1: Complete Model On the ASVspoof2019LA dataset, our method achieves an EER of 0.72% and a t-DCF of 0.0202. Compared to the baseline model AASIST, the EER is reduced by 13.3% and the t-DCF by 26.5%. In particular, under the A16 attack, the EER significantly decreases from 0.72% to 0.12%, representing a relative improvement of 83.3%.
[0080] Example 2: Removal of the gated short-time encoder The difference between this embodiment and Embodiment 1 is that the gated short-time encoder was removed. The results show that the EER increased to 0.77% and the t-DCF increased to 0.0212. This indicates that the lack of local fine-grained modeling weakens the model's ability to detect minute forgery traces.
[0081] Example 3: Removal of the Joint Time-Frequency Self-Attention Module The difference between this embodiment and Embodiment 1 is that the joint time-frequency self-attention module was removed. The results show that the EER deteriorated to 0.80%, while the t-DCF increased to 0.0270. This verifies the core role of this module in cross-dimensional global modeling, and its integration of long-range time-frequency dependencies is crucial.
[0082]
[0083] As shown in Table 2: ① The joint time-frequency self-attention module plays a core role in performance improvement and can integrate cross-dimensional discriminative information more effectively than heterogeneous graph attention; ② The gated short-time encoder has a stable gain in capturing local small artifacts; ③ The fusion of the two can effectively take into account both local fine-grained features and global dependency information, significantly improving detection robustness.
[0084] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for detecting deep speech forgery based on gated short-sequence coding and joint time-frequency self-attention mechanism, characterized in that, include: The steps include: acquiring the speech signal to be detected, extracting features from the speech signal to be detected, and obtaining a speech feature representation; The step of performing correlation modeling on the speech feature representation within a local time range to obtain local anomaly enhancement features; The step of generating a time-frequency feature representation with a distribution relationship in time and frequency dimensions based on the local anomaly enhancement features; The steps include performing global dependency modeling on the time dimension features and frequency dimension features in the time-frequency feature representation, and fusing the modeling results to obtain the time-frequency coupled features; The step of fusing and judging the local anomaly enhancement features with the time-frequency coupling features, and outputting the authenticity determination result of the speech signal to be detected.
2. The speech deep forgery detection method based on gated short-time coding and joint time-frequency self-attention mechanism according to claim 1, characterized in that, When extracting features from the speech signal to be detected, the speech signal to be detected is first processed to unify the format to obtain a standardized speech signal, and then the standardized speech signal is processed to extract context features and perform nonlinear compression to obtain the speech feature representation.
3. The speech deep forgery detection method based on gated short-time coding and joint time-frequency self-attention mechanism according to claim 1, characterized in that, When modeling the correlation of the speech feature representation within a local time range, multiple local time segments are divided along the time dimension of the speech feature representation. Based on the correlation between different time positions within each local time segment, the features within the corresponding local time segment are weighted and aggregated to obtain the local anomaly enhancement features.
4. The speech deep forgery detection method based on gated short-time coding and joint time-frequency self-attention mechanism according to claim 1, characterized in that, The local anomaly enhancement feature is formed by weighted aggregation of local time segment features and channel calibration. The channel calibration generates channel weights based on the weighted aggregation of local time segment features and adjusts the feature responses of different channels according to the channel weights.
5. The speech deep forgery detection method based on gated short-time coding and joint time-frequency self-attention mechanism according to claim 1, characterized in that, When generating the time-frequency feature representation based on the local anomaly enhancement features, a two-dimensional feature mapping is performed on the local anomaly enhancement features to convert the local anomaly enhancement features into a feature representation that simultaneously contains time dimension distribution information and frequency dimension distribution information.
6. The speech deep forgery detection method based on gated short-time coding and joint time-frequency self-attention mechanism according to claim 1, characterized in that, When performing global dependency modeling on the time dimension features and the frequency dimension features, the dependency relationships between different time positions in the time dimension features and the dependency relationships between different frequency positions in the frequency dimension features are calculated respectively to obtain time dependency features and frequency dependency features.
7. A speech deep forgery detection device based on gated short-time coding and joint time-frequency self-attention mechanism, characterized in that, include: A module for acquiring the speech signal to be detected, extracting features from the speech signal to be detected, and obtaining a speech feature representation; A module for modeling the correlation of the speech feature representation within a local time range is obtained to obtain local anomaly enhancement features; A module that generates a time-frequency feature representation with a distribution relationship in both time and frequency dimensions based on the local anomaly enhancement features; Global dependency modeling is performed on the time dimension features and frequency dimension features in the time-frequency feature representation, and the modeling results are fused to obtain the module of time-frequency coupled features; A module that fuses and distinguishes the local anomaly enhancement features with the time-frequency coupling features, and outputs the authenticity determination result of the speech signal to be detected.
8. A computer storage medium for storing computer programs, characterized in that, When the computer program is read by the computer, the computer executes the method of claim 1.
9. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.
10. A computer program product, as a computer program, is characterized by: When the computer program is executed, it implements the method of claim 1.