Speech anti-spoofing detection method and device based on enhanced timing and channel modeling
By employing an enhanced temporal and channel modeling approach, this method utilizes a pre-trained self-supervised speech model to extract multi-layer hidden state features. Combined with multi-head self-attention and gating mechanisms, it addresses the issues of insufficient temporal modeling and inadequate feature utilization in existing speech anti-spoofing detection technologies, thereby improving the detection performance and robustness of high-quality forged speech.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 电视电声研究所(中国电子科技集团公司第三研究所)
- Filing Date
- 2026-03-24
- Publication Date
- 2026-06-09
AI Technical Summary
Existing speech anti-spoofing detection methods have insufficient generalization performance in high-quality forged speech detection, inadequate temporal modeling and feature utilization, and are sensitive to noise, making it difficult to effectively capture various attack patterns.
We employ an enhanced temporal and channel modeling approach, extracting hidden state features from multiple network layers using a pre-trained self-supervised speech model. These features are then divided along the channel dimension and processed in a nested manner. By combining multi-head self-attention and gating mechanisms, we dynamically fuse historical memory with current output for fine-grained feature enhancement. Finally, we use a cross-layer attention mechanism for adaptive weighted fusion to generate the final discrimination result.
It improves the robustness of detecting unknown synthetic speech attacks, enhances the ability to represent the features of speech signals, and improves the generalization performance of the model in different scenarios.
Smart Images
Figure CN122177162A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of voice anti-spoofing technology, and in particular to a voice anti-spoofing detection method and apparatus based on enhanced temporal and channel modeling. Background Technology
[0002] With the rapid development of speech synthesis and voice conversion technologies, high-fidelity forged speech poses a serious threat to the security of systems such as Automatic Speaker Verification (ASV), intelligent customer service, and financial authentication. To address this challenge, academia and industry have proposed various speech anti-spoofing detection methods.
[0003] Currently, with the development of data quality and large-scale model technology, the realism of forged speech is becoming increasingly high, reaching a level that can "deceive the real thing" in many scenarios. However, the performance of speech anti-spoofing methods is often limited by practical factors such as limited training data and mismatch between the use case and the training case, resulting in poor generalization performance in real-world applications.
[0004] Recently, pre-trained large-scale speech models based on self-supervised learning (SSL) have achieved good results in various downstream speech tasks (such as speech recognition and speaker recognition). Typical SSL models such as wav2vec 2.0, HuberT, and WavLM have also been used in speech spoofing detection tasks, significantly improving the accuracy of speech anti-spoofing models compared with traditional discriminative features such as Mel-Frequency Cepstral Coefficient (MFCC) and Linear Frequency Cepstral Coefficient (LFCC). However, most related methods only utilize the last hidden state of the SSL model for classification, ignoring the rich acoustic details and local anomaly cues contained in the intermediate layers. Research shows that low-level features are sensitive to waveform distortion, while high-level features are sensitive to semantic contradictions. A single layer is insufficient to comprehensively capture various attack patterns, and the model's generalization performance is still insufficient in scenarios where the test data background differs significantly from the training data background.
[0005] To further enhance feature representation capabilities, recent studies have proposed nested fusion structures such as Res2Net, which partition features along the channel dimension and introduce multi-branch cumulative connections to enhance the diversity of local representations. However, this structure relies on convolutional operations, is limited by the local receptive field, cannot effectively model long-distance temporal dependencies, and lacks an adaptive fusion mechanism for abstract information at different levels. Summary of the Invention
[0006] The technical solution adopted in this invention is to effectively solve the problems of insufficient temporal modeling, inadequate feature utilization, and noise sensitivity in existing technologies, thereby improving the detection performance of high-quality synthesized speech. In view of this, this invention provides a speech anti-spoofing detection method and apparatus based on enhanced temporal and channel modeling.
[0007] The present invention proposes a voice anti-spoofing detection method based on enhanced temporal and channel modeling, comprising: Step 1: Input the speech signal into the pre-trained self-supervised speech model and obtain the hidden state features of K intermediate and output layers in the self-supervised speech model; Step 2: Perform nested feature enhancement processing on each layer of state features to obtain the sentence-level representation vector corresponding to that layer; Step 3: Input the sentence-level representation vectors of all layers, calculate the weights of each layer, and generate the final fused embedding vector; Step 4: Input the final fused embedding vector into the classifier and output the discrimination result of whether the corresponding speech is real speech or synthesized speech.
[0008] In one implementation, the step of performing nested feature enhancement processing on each layer of state features to obtain the sentence-level representation vector corresponding to that layer further includes: State features of this layer That is, by T frame, D The time series matrix, composed of 3D features, is divided along the channel dimension into N consecutive subgroups Each subgroup has a dimension of ; Initialize a memory variable It is a zero matrix; For the i Subgroups Perform the following operations: Construct the current input ; Will Input a time-series-channel co-modeling core unit To obtain intermediate output = , wherein This includes a multi-head self-attention mechanism, used to simultaneously model inter-frame dependencies in the temporal dimension and feature interactions in the channel dimension; Calculate a gating vector ,in , This represents a global average pooling operation along the time dimension. For the Sigmoid function; Generate updated memory state ,in This represents element-wise multiplication; Output all branches Enhanced features stitched together to form a complete channel dimension ; right Perform temporal pooling to obtain the sentence-level representation vector corresponding to this layer. .
[0009] In one implementation, the The module further includes layer normalization and feedforward neural network sublayers, which constitute the Transformer block structure. The time neighborhood W satisfies 5≤W≤20, corresponding to the duration of 0.1~0.4 seconds in the speech signal, in order to reduce computational complexity and improve inference efficiency.
[0010] In one implementation, the multi-head self-attention mechanism introduces relative position encoding when calculating attention weights to preserve relative order information in the time series and enhance sensitivity to local forgery cues, including speech start and end boundaries and pause anomalies.
[0011] In one implementation, the number of network layers K ≥ 3, and network layers located in the middle or above of the self-supervised speech model are selected.
[0012] In one implementation, the gate vector During the calculation process, The weight matrix is learnable, and the gating mechanism dynamically adjusts the historical memory. Compared with the current output The proportion of contribution.
[0013] In one implementation, the temporal pooling is attention pooling, i.e., learning a set of query vectors. ,right Perform weighted aggregation to obtain ,in For based on and The calculated attention weights.
[0014] Another aspect of the present invention provides a voice anti-spoofing detection device based on enhanced temporal and channel modeling, comprising: The acquisition module is used to input the speech signal into the pre-trained self-supervised speech model and acquire the hidden state features of K intermediate and output layers in the self-supervised speech model. The enhancement module is used to perform nested feature enhancement processing on the state features of each layer to obtain the sentence-level representation vector corresponding to that layer. The fusion module is used to take the sentence-level representation vectors of all layers as input, calculate the weights of each layer, and generate the final fused embedding vector. The discrimination module is used to input the final fused embedded vector into the classifier and output a discrimination result that determines whether the corresponding speech is real speech or synthesized speech.
[0015] Another aspect of the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the voice anti-spoofing detection method based on enhanced timing and channel modeling as described in any of the preceding claims.
[0016] Another aspect of the present invention provides a computer storage medium storing a computer program that is executed to implement the voice anti-spoofing detection method based on enhanced timing and channel modeling as described in any of the preceding claims.
[0017] By adopting the above technical solution, the present invention has at least the following advantages: The method provided by this invention first extracts the hidden states of multiple network layers from a pre-trained self-supervised speech model. Each layer's features are then divided into several subgroups along the channel dimension and sequentially input into an enhanced temporal-channel modeling module through a nested processing structure. In this module, a cumulative input approach combined with a multi-head self-attention mechanism is used to model temporal inter-frame dependencies and channel interactions. A gating mechanism is introduced to dynamically fuse historical memory and current output, achieving fine-grained feature enhancement. Each layer undergoes temporal pooling to obtain sentence-level representations, which are then adaptively weighted and fused through a cross-layer attention mechanism to generate the final embedding vector, which is then input into a classifier to distinguish between real and fake speech. This invention fully leverages the complementarity of features in multi-layer self-supervised learning models, improving robustness against unknown synthetic speech attacks. Attached Figure Description
[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the speech anti-spoofing detection method based on enhanced temporal and channel modeling according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall architecture of the voice anti-spoofing detection method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a nested time-channel modeling process for a single-layer feature according to an embodiment of the present invention; Figure 4This is a schematic diagram of the internal structure of the timing-channel joint modeling core unit according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the cross-layer attention fusion and sentence-level representation generation process according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a voice anti-spoofing detection system according to an embodiment of the present invention. Detailed Implementation
[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.
[0020] While exemplary embodiments of the invention are shown in the accompanying drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey its scope to those skilled in the art. The invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0021] This invention provides a voice anti-spoofing detection method based on enhanced temporal and channel modeling, such as... Figure 1 As shown, it includes the following steps: See Figure 1 and Figure 2 The embodiments of this disclosure provide a voice anti-spoofing detection method based on enhanced time-channel modeling, including the following steps: S1, input the speech signal into the pre-trained self-supervised speech model, and obtain the hidden state features of K intermediate and output layers in the self-supervised speech model.
[0022] See Figure 2 Hidden state features of each layer's output , representing a time-series matrix composed of T frames and D-dimensional features. In specific application scenarios, before feature extraction, the speech signal is first processed by unifying the sampling rate, normalizing, and unifying the duration, such as unifying the sampling rate to 16000Hz and the speech duration to 200 frames.
[0023] Preferably, the self-supervised speech model (SSL model) is one of wav2vec 2.0, HuberT, WavLM, etc.
[0024] In specific application scenarios, such as choosing the HuBERT_base model, which has 12 hidden layers and each layer outputs 768 dimensions, we can select the 3rd, 6th, 9th and 12th layers of hidden state features from low-dimensional, medium-dimensional and high-dimensional, resulting in K=4, T=200 and D=768.
[0025] S2, for each layer of features Perform nested feature enhancement processing, specifically including: Divide the features of this layer into N consecutive subgroups along the channel dimension. Each subgroup has a dimension of ; Initialize a memory variable It is a zero matrix; For the i-th subgroup Perform the following operations: Construct the current input ; Will Input a time-series-channel co-modeling core unit To obtain intermediate output = , wherein It includes a multi-head self-attention mechanism to simultaneously model inter-frame dependencies in the temporal dimension and feature interactions in the channel dimension; Calculate a gating vector ,in , This represents a global average pooling operation along the time dimension. For the Sigmoid function; Generate updated memory state ,in This represents element-wise multiplication; Output all branches Enhanced features stitched together to form a complete channel dimension ; right Perform temporal pooling to obtain the sentence-level representation vector corresponding to this layer. ; See Figure 3 The feature layer is divided into N consecutive subgroups along the channel dimension. Preferably, N=4, that is, the number of channel groups is 4, so there are consecutive subgroups. Each subgroup has a dimension of For each subgroup, after time-series-channel joint modeling and gating, a memory variable is obtained. The output of the previous subgroup is used as part of the input of the next subgroup, thus achieving multi-scale learning through nested subgroups.
[0026] See Figure 4 The The module further includes LayerNorm, Multi-Head Self-Attention (MHSA), and Feedforward Neural Network (FFN) sublayers, forming a standard Transformer block structure containing 8-head self-attention and feedforward networks.
[0027] The multi-head self-attention mechanism employs a local window attention mechanism, which restricts each time step to focusing only on its time neighborhood of a fixed length W before and after it, where 5 ≤ W ≤ 20, corresponding to a duration of 0.1 to 0.4 seconds in the speech signal, in order to reduce computational complexity and improve inference efficiency.
[0028] The multi-head self-attention mechanism introduces relative position encoding when calculating attention weights to preserve relative order information in the time series and enhances sensitivity to local forgery cues such as speech start and end boundaries and abnormal pauses.
[0029] See Figure 4 In specific implementation, with To illustrate the processing, consider the following example: if K=4, T=200, D=768, N=4, then... The size is 200×192. Let W=10, and... Input to The module first performs timing modeling using a local window MHSA module and introduces relative position encoding. The output at this point is... After summing, layer normalization is performed to obtain the temporal modeling output. Then, to ensure dimensionality consistency, the temporal modeling output is transposed and used for channel modeling via the channel MHSA module. After dimensionality restoration, it is added to the temporal modeling output and then layer normalized to obtain the channel modeling output. The channel modeling output is then fed into the FFN network layer. This output is added to the channel modeling output and layer normalized to obtain the final output. The module's output.
[0030] The gate vector During the calculation process, The weight matrix is learnable, and the gating mechanism dynamically adjusts the historical memory. Compared with the current output The contribution ratio is effectively suppressed by the start and end points of the speech signal and the feature fluctuations caused by background noise, thus improving the robustness of detecting forgery of short speech segments.
[0031] Preferably, the gate vector It can be in scalar, vector, or tensor form, and supports element-wise or channel-wise control.
[0032] The temporal pooling is attention pooling, which involves learning a set of query vectors. ,right Perform weighted aggregation to obtain ,in For based on and The calculated attention weights.
[0033] S3 represents the sentence-level representation of all layers. Input a cross-layer attention fusion module and calculate the weights of each layer. And generate the final fused embedding vector. .
[0034] The cross-layer attention fusion module uses a network layer K≥3, and the selected layers are evenly distributed in the middle and high levels of the self-supervised speech model to balance the information complementarity between low-level acoustic details and high-level semantic abstraction.
[0035] In this embodiment, the outputs of four network layers (layers 3, 6, 9, and 12) are selected. The cross-layer attention fusion module uses a single-layer attention head.
[0036] S4, Input the classifier and output the result of whether the speech is real speech or synthetic speech.
[0037] See Figure 1 In practice, a fully connected classifier (such as a two-layer multilayer perceptron (MLP) + Dropout) can be used as the classifier. The final output is 2-dimensional, which is the classification result: the confidence value of whether the speech belongs to real speech or fake speech. End-to-end training is performed through the cross-entropy loss function.
[0038] In practice, the model can be trained using the Adam optimizer with a learning rate of 1e-4 and a batch size of 32.
[0039] In a second embodiment of the present invention, a voice anti-spoofing detection device based on enhanced temporal and channel modeling can be understood as a physical device for performing the method provided in the first embodiment. This device may specifically include: The acquisition module is used to input the speech signal into the pre-trained self-supervised speech model and acquire the hidden state features of K intermediate and output layers in the self-supervised speech model. The enhancement module is used to perform nested feature enhancement processing on the state features of each layer to obtain the sentence-level representation vector corresponding to that layer. The fusion module is used to take the sentence-level representation vectors of all layers as input, calculate the weights of each layer, and generate the final fused embedding vector. The discrimination module is used to input the final fused embedded vector into the classifier and output a discrimination result that determines whether the corresponding speech is real speech or synthesized speech.
[0040] A third embodiment of the present invention provides an electronic device, such as... Figure 6 As shown, it includes: a memory 1130 and a processor 1110, which are connected to each other through a communication interface 1120; the memory 1130 stores a computer program, and the processor 1110 is used to execute the computer program to implement the voice anti-spoofing detection method based on enhanced timing and channel modeling as described in any of the above claims.
[0041] In a fourth embodiment of the present invention, a computer storage medium is provided, wherein a computer program is stored on the medium, and the computer program is executed to implement the voice anti-spoofing detection method based on enhanced timing and channel modeling as described in the first embodiment.
[0042] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.
Claims
1. A voice anti-spoofing detection method based on enhanced temporal and channel modeling, characterized in that, include: Step 1: Input the speech signal into the pre-trained self-supervised speech model and obtain the hidden state features of K intermediate and output layers in the self-supervised speech model; Step 2: Perform nested feature enhancement processing on each layer of state features to obtain the sentence-level representation vector corresponding to that layer; Step 3: Input the sentence-level representation vectors of all layers, calculate the weights of each layer, and generate the final fused embedding vector; Step 4: Input the final fused embedding vector into the classifier and output the discrimination result of whether the corresponding speech is real speech or synthesized speech.
2. The voice anti-spoofing detection method based on enhanced temporal and channel modeling according to claim 1, characterized in that, The step of performing nested feature enhancement processing on each layer of state features to obtain the sentence-level representation vector corresponding to that layer further includes: State features of this layer That is, by T frame, D The time series matrix, composed of 3D features, is divided along the channel dimension into N consecutive subgroups Each subgroup has a dimension of ; Initialize a memory variable It is a zero matrix; For the i Subgroups Perform the following operations: Construct the current input ; Will Input a time-series-channel co-modeling core unit To obtain intermediate output = , wherein This includes a multi-head self-attention mechanism, used to simultaneously model inter-frame dependencies in the temporal dimension and feature interactions in the channel dimension; Calculate a gating vector ,in , This represents a global average pooling operation along the time dimension. For the Sigmoid function; Generate updated memory state ,in This represents element-wise multiplication; Output all branches Enhanced features stitched together to form a complete channel dimension ; right Perform temporal pooling to obtain the sentence-level representation vector corresponding to this layer. .
3. The voice anti-spoofing detection method based on enhanced temporal and channel modeling according to claim 2, characterized in that, The The module further includes layer normalization and feedforward neural network sublayers, which constitute the Transformer block structure. The time neighborhood W satisfies 5≤W≤20, corresponding to the duration of 0.1~0.4 seconds in the speech signal, in order to reduce computational complexity and improve inference efficiency.
4. The voice anti-spoofing detection method based on enhanced temporal and channel modeling according to claim 2, characterized in that, The multi-head self-attention mechanism introduces relative position encoding when calculating attention weights to preserve relative order information in the time series and enhances sensitivity to local forgery cues, including speech start and end boundaries and abnormal pauses.
5. The voice anti-spoofing detection method based on enhanced temporal and channel modeling according to claim 2, characterized in that, The number of network layers K ≥ 3, and the network layers located in the middle or above of the self-supervised speech model are selected.
6. The voice anti-spoofing detection method based on enhanced temporal and channel modeling according to claim 2, characterized in that, The gate vector During the calculation process, The weight matrix is learnable, and the gating mechanism dynamically adjusts the historical memory. Compared with the current output The proportion of contribution.
7. The voice anti-spoofing detection method based on enhanced temporal and channel modeling according to claim 2, characterized in that, The temporal pooling is attention pooling, which involves learning a set of query vectors. ,right Perform weighted aggregation to obtain ,in For based on and The calculated attention weights.
8. A voice anti-spoofing detection device based on enhanced temporal and channel modeling, characterized in that, include: The acquisition module is used to input the speech signal into the pre-trained self-supervised speech model and acquire the hidden state features of K intermediate and output layers in the self-supervised speech model. The enhancement module is used to perform nested feature enhancement processing on the state features of each layer to obtain the sentence-level representation vector corresponding to that layer. The fusion module is used to take the sentence-level representation vectors of all layers as input, calculate the weights of each layer, and generate the final fused embedding vector. The discrimination module is used to input the final fused embedded vector into the classifier and output a discrimination result that determines whether the corresponding speech is real speech or synthesized speech.
9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the voice anti-spoofing detection method based on enhanced timing and channel modeling as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The medium stores a computer program that is executed to implement the voice anti-spoofing detection method based on enhanced temporal and channel modeling as described in any one of claims 1 to 7.