Local spoofed speech detection method based on improved boundary-aware attention mechanism

Abstract

The application discloses a local forged speech detection method based on an improved boundary perception attention mechanism, first proposes cross-fragment mixed data enhancement, weakens the memory dependence of the model on fixed boundary texture by increasing the diversity of sample combinations, makes the model learn more universal forged patterns, then adopts a boundary soft gating mechanism to prevent uncertain boundaries from being prematurely hard-decided, suppress unreasonable cross-boundary propagation, and reduce semantic pollution between real segments and forged segments, and further replaces a ResNet1D intra-frame feature extractor with a Conforme block, simultaneously solves two structural problems of excessively large non-SSL parameters and context loss, enhances the learning ability of the speech feature, and makes more fine distinctions on the frames near the tampered boundary.

Description

Technical Field

[0001] This invention relates to the fields of speech authenticity detection, audio deepfake recognition, and temporal signal localization, and particularly to a local fake speech detection method based on an improved boundary-aware attention mechanism. Background Technology

[0002] Localized forged speech detection is a branch of speech signal processing research. Its localization task focuses on frame-level localization of the location of forged speech and accurately determining the authenticity of the audio on a frame-by-frame basis. Although some progress has been made in this field, traditional localized speech forgery detection systems often cannot adapt to complex scenarios when forgery traces are hidden and scattered, and forged features are intertwined with real features.

[0003] With the rapid popularization and development of generative artificial intelligence technology, the social media field is flooded with deepfake audio generated by technologies such as speech synthesis and speech conversion. These highly natural-looking deepfake audios pose a significant challenge to distinguishing between genuine and fake media content. To address these issues, deepfake audio detection has received widespread attention in recent years. To date, many effective countermeasures have emerged, with some models achieving performance below 1% using equal error rate (EER) evaluation, demonstrating significant success in defending against fully deepfake audio attacks. However, every step of technological development is accompanied by new challenges, and partial deepfake audio attacks, as a more covert deception method, have once again attracted attention. These audios typically combine genuine audio segments with forged audio segments, or genuine audio segments from other corpora, increasing the complexity and difficulty of attack identification. Existing research shows that both humans and machines are susceptible to being deceived by partial deepfake audio. To address this problem, various deepfake audio detection challenges have been organized both domestically and internationally to promote the development of the audio anti-spoofing field.

[0004] The inaugural Audio Deep Synthesis Detection Challenge (ADD 2022) was held in 2022, where Partial Forgery Detection (PF) was introduced as a separate track, focusing on the real / forged binary classification task. In 2023, the second Audio Deep Forgery Detection Challenge, ADD 2023, was officially launched. Unlike ADD 2022, ADD 2023's task setting went beyond simply classifying entire speech segments as real / forged. For the first time, it set "locating the tampered regions in partially forged audio" as the competition task Track 2, using a weighted sum of sentence-level accuracy and segment-level F1 score as the evaluation metric. Currently, the systems on the leaderboard of this competition have become important benchmarks for subsequent research on the task of locating tampered regions in partially deep forged audio.

[0005] In recent years, methods and datasets for Partially Spoofed Audio Detection (PSAD) have been extensively studied. Zhang et al. developed a dataset called PartialSpoof, which contains segment-level labels, and investigated a method capable of detecting spoofed content simultaneously at the segment and speech segment levels within a multi-task learning framework. The 2023 ADD challenge introduced a partially spoofed audio localization track, where an effective method achieved the best localization score by fusing the decision results of three adversarial systems: boundary detection, frame-level spoofing detection, and speech segment-level spoofing detection. More recently, research has proposed a method combining a contrastive learning module and temporal convolution to effectively capture superior localization features. Later, Zhong et al. extracted boundary features to accurately locate the specific position of spoofed content, achieving reliable frame-level authenticity assessment.

[0006] While the aforementioned research methods have made some progress, local speech authentication systems often fail to adapt to complex scenarios where forgery traces are concealed and dispersed, and forged features are intertwined with genuine features. Similarly, the stability of the forgery boundary directly affects the system's attention allocation and discrimination capabilities. Attackers can insert forged speech segments into genuine speech, or vice versa, thus creating locally forged speech. The harm of locally forged speech lies not only in the difficulty of determining the authenticity of the entire speech segment, but also in the fact that a small number of altered words, syllables, or short segments may change the semantics.

[0007] The task of localizing locally forged speech requires providing real and fake speech discrimination results at different temporal locations of the speech. Compared with whole-speech detection, this task needs to handle finer-grained temporal boundaries, and speech frames near the boundaries often contain transitional features between real and forged components. Existing frame-level detection methods typically divide the speech into several temporal frames, extract the representation of each frame, and classify them separately. Such methods can output frame-level results, but when the forged segment is short, the boundary transition is smooth, or the context similarity is high, misclassification near the boundary is prone to occur. Existing boundary-aided methods attempt to use boundary prediction results to limit the information interaction between different segments. However, if the boundary prediction is directly binarized and used for hard masking, small errors in the boundary prediction will be amplified: missed boundaries will cause real and forged segments to continue to mix information, and false boundaries will cut off the contextual association information that should be preserved within the same segment.

[0008] Therefore, a detection system suitable for localized spoofing of speech is needed, which allows boundary information to participate in inter-frame information control in the form of continuous confidence, thereby suppressing cross-boundary error propagation while retaining necessary contextual information near uncertain boundaries, thus improving the generalization and accuracy of the speech spoofing detection system. Summary of the Invention

[0009] The technical problem this invention aims to solve is: to overcome the shortcomings of existing technologies, this invention addresses the issues of hard boundary masking's sensitivity to boundary prediction errors and the lack of continuity in inter-frame information interaction during localized spoofed speech localization. To address these problems and improve the generalization and accuracy of speech spoofing detection systems, this invention provides a localized spoofed speech localization method based on soft boundary gating. This method first acquires the speech signal to be detected and extracts the basic speech representation sequence, then generates a frame-level speech feature sequence according to a preset time resolution; subsequently, it obtains boundary enhancement features through a boundary enhancement module and generates boundary probabilities based on these features; further, it generates a soft boundary gating sequence based on the boundary probabilities and temperature parameters, uses this soft boundary gating sequence to construct an inter-frame message control matrix, continuously modulates inter-frame attention, and finally outputs the authenticity judgment results for each frame.

[0010] In this invention, the boundary probabilities are not directly transformed into binary boundaries, but rather into soft boundary gating values ​​through a temperature-modified sigmoid mapping. These soft boundary gating values ​​participate in the construction of the inter-frame message control matrix, thereby continuously weakening inter-frame message transmission across high boundary probability positions.

[0011] In a preferred embodiment, the boundary enhancement module includes an inter-frame correlation modeling branch and an intra-frame context modeling branch. The inter-frame correlation modeling branch is used to obtain the similarity and dependency relationships between frames at different times; the intra-frame context modeling branch adopts a convolution-enhanced self-attention structure to simultaneously capture long-range dependencies and local perturbations.

[0012] In a preferred embodiment, cross-segment hybrid enhancement is introduced during the training phase. This enhancement method performs a segment replacement between two training speech samples and simultaneously updates frame-level true / false labels and boundary labels, thereby increasing the diversity of boundary morphology in the training data.

[0013] The present invention also provides a localized spoofed voice localization system based on soft boundary gating and a computer-readable storage medium. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0015] Figure 2 This is a schematic diagram of the intra-frame context modeling branch in the boundary enhancement module of this invention;

[0016] Figure 3 This is a schematic diagram of constructing an inter-frame message control matrix and modulating inter-frame attention using the soft-boundary gating sequence of the present invention;

[0017] Figure 4This is a schematic diagram of the process of cross-segment hybrid enhancement and boundary label regeneration of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, in one embodiment, the input is a speech signal to be detected with a sampling rate of 8000 Hz to 48000 Hz, preferably 16000 Hz. The system extracts the temporal representation of the speech signal to be detected to obtain the basic speech representation sequence H.

[0020] Let the basic speech representation sequence be:

[0021] ,

[0022] Where N represents the number of basic speech representations, and D represents the dimension of the representations. Basic speech representations can be generated by a self-supervised speech representation network or by a speech feature extraction network with temporal context modeling capabilities.

[0023] The system operates according to a preset time resolution. Frame-level aggregation is performed on the basic speech representation sequence to obtain the frame-level speech feature sequence X. If the adjacent time intervals of the basic speech representations are... Each frame-level speech feature is obtained by aggregating K consecutive basic speech representations:

[0024] ,

[0025] In a preferred embodiment The preset time resolution r is 0.02 seconds, r is 0.16 seconds, and K is 8. The preset time resolution r can be set from 0.08 seconds to 0.32 seconds to accommodate different positioning accuracy and computational requirements.

[0026] Frame-level aggregation can employ an attention-weighted approach. For the first... The frame-level speech features are derived from the first frame within that frame. Attention weights of basic speech representations Determine their contribution:

[0027] ,

[0028] in, , and For trainable parameters, Indicates the first Each frame-level speech feature.

[0029] The system inputs the frame-level speech feature sequence X into the boundary enhancement module. The boundary enhancement module includes an inter-frame correlation modeling branch and an intra-frame context modeling branch.

[0030] In the inter-frame correlation modeling branch, for the first... Frame and the First, calculate the pairwise frame correlation features:

[0031] ,

[0032] in, This represents element-wise multiplication. Then, the inter-frame correlation attention score is calculated:

[0033] ,

[0034] in, Indicates the attention head number, , and These are trainable parameters. After normalizing the inter-frame correlation attention scores, we can obtain the ... The aggregate representation of frames to other frames.

[0035] In the intra-frame context modeling branch, the system employs a convolution-enhanced self-attention structure, the specific structure of which is as follows: Figure 2 As shown. The structure includes a first feedforward sub-layer, a multi-head self-attention sub-layer, a convolutional sub-layer, a second feedforward sub-layer, and a normalization sub-layer. For the input sequence... The structure can be represented as:

[0036] ,

[0037] ,

[0038] ,

[0039] Y ,

[0040] in, and The sublayers are denoted as feedforward, MHSA as multi-head self-attention, Conv as convolutional, and LN as normalized. The kernel length of the convolutional sublayer can be from 7 to 31, preferably 15.

[0041] The representations output by the inter-frame correlation modeling branch and the intra-frame context modeling branch are concatenated to obtain the boundary enhancement features. For the first The frame and boundary probabilities are obtained using the following formula:

[0042] ,

[0043] in, Indicates the first Frame boundary enhancement features, and For trainable parameters, Indicates the first Boundary probability of a frame.

[0044] The system generates soft boundary gating values ​​based on boundary probabilities, as illustrated in the diagram below. Figure 3 As shown, its formula is as follows:

[0045] ,

[0046] in, The threshold value for the boundary center is 0.4 to 0.6, preferably 0.5. The temperature parameter is represented, with a value ranging from 0.05 to 0.5, preferably 0.1. The smaller the temperature parameter, the steeper the gating curve; the larger the temperature parameter, the smoother the gating curve.

[0047] The system constructs an inter-frame message control matrix based on soft-boundary gating sequences. For the first Frame and the frame, The elements in the middle are determined according to the following formula:

[0048] ,

[0049] According to the above formula, when the first Frame and the When there are high boundary probability locations between frames, Continuous decrease; when two frames are located within the same real segment or the same fake segment. Maintain a high value.

[0050] The system continuously modulates inter-frame attention. Assume... Indicates the first The first one in the attention. Frame to the first Given the original attention score of the frame, the modulated attention is:

[0051] ,

[0052] The system aggregates frame-level speech features based on the modulated attention to obtain boundary-aware frame-level discriminative features S.

[0053] In one embodiment, the system uses boundary-aware frame-level discriminative features. With boundary enhancement features The mapping results are fused to obtain the fused features. :

[0054] ,

[0055] in, Indicates splicing, This represents the feature mapping. Subsequently, the classification mapping is used to output the first... Probability of a frame being real or fake:

[0056] ,

[0057] During the training phase, the system uses a weighted sum of frame-level real / false losses and boundary losses as the objective function:

[0058] ,

[0059] in, Indicates frame-level authenticity loss. Indicates boundary loss. This represents the boundary loss weight, with a value ranging from 0.1 to 1.0, preferably 0.5.

[0060] Frame-level authenticity loss can be achieved using cross-entropy loss:

[0061] ,

[0062] Boundary loss can be either the binary cross-entropy loss (balancing positive and negative samples) or the boundary focusing loss. The boundary focusing loss can be expressed as:

[0063] ,

[0064] in, Indicates the first Frame boundary labels, Indicates the first Boundary probability of a frame. The value can be between 1 and 5, with 2 being the preferred value. The value ranges from 0.1 to 0.5, with 0.25 being preferred.

[0065] In a preferred embodiment, cross-segment hybrid enhancement is employed during the training phase, as illustrated in the diagram below. Figure 4 As shown. Specifically, a first speech sample and a second speech sample are selected from the same batch, and their common effective frame length is determined. and in the interval Internal selection of segmented frame c, where The value ranges from 0.1 to 0.4, with 0.2 being preferred. The value of is between 0.6 and 0.9, preferably 0.8.

[0066] Let the first speech sample be The second voice sample is The mixed speech samples are Then it can be expressed as:

[0067] ,

[0068] Frame-level true / false tags are synchronously replaced at the same segmentation position, and boundary tags are regenerated based on the replaced tags. If the tags of two adjacent frames change, the frame corresponding to the physical splicing point is marked as the boundary frame.

[0069] In another embodiment, the multi-layer outputs of the basic speech representation can be weighted and fused. Let the output of the l-th layer be... The corresponding weight is The fusion characterization is then:

[0070] ,

[0071] This approach is used for adaptive selection between different levels of acoustic detail and semantic context.

[0072] In another embodiment, a boundary decoupling residual path can be set up to incorporate frame-level speech features that are not modulated by the boundary into the boundary-aware frame-level discriminative features according to learnable weights, so as to preserve the original context when the confidence of boundary prediction is insufficient.

[0073] The system embodiments of the present invention include a speech representation extraction module, a frame-level aggregation module, a boundary enhancement module, a soft boundary gating module, a boundary-aware attention module, and a frame-level discrimination module. These modules can be deployed in the same computing device or in multiple cooperating computing devices.

[0074] This invention transforms boundary probabilities into continuous modulation quantities through soft boundary gating, avoiding the oversensitivity of binary hard masking to boundary prediction errors. It also controls the communication strength between different time frames using an inter-frame message control matrix, preserving information within the same segment while reducing cross-boundary error propagation. Furthermore, it enhances intra-frame context extraction through convolutional self-attention structures, balancing long-range dependencies and local boundary perturbations, thus improving the separability of frames near boundaries. Additionally, it generates training samples with different splicing positions and segment combinations through cross-segment hybrid enhancement and synchronously updates boundary labels, improving the generalization ability of the localized forged speech localization model.

[0075] The above description is an exemplary embodiment of the present invention and does not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A local forged speech detection method based on an improved boundary-aware attention mechanism, characterized in that, Includes the following steps: (1) Obtain the speech signal to be detected, and extract the temporal representation of the speech signal to be detected to obtain the basic speech representation sequence; (2) Aggregate the basic speech representation sequence into a frame-level speech feature sequence according to a preset time resolution; (3) Perform inter-frame correlation modeling and intra-frame context modeling on the frame-level speech feature sequences respectively to obtain boundary enhancement features; (4) Generate the boundary probability of each frame based on the boundary enhancement features, and generate a soft boundary gating sequence based on the boundary probability; (5) Construct an inter-frame message control matrix based on the soft boundary gating sequence, and use the inter-frame message control matrix to continuously modulate the inter-frame attention to obtain boundary-aware frame-level discrimination features; (6) Integrate the boundary-assisted representation corresponding to the boundary enhancement feature with the boundary-aware frame-level discrimination feature, and output the authenticity discrimination result of each frame in the speech signal to be detected.

2. The method according to claim 1, characterized in that, The basic speech representation sequence is generated by a self-supervised speech representation network, and at least one layer of the multi-layer representations output by the self-supervised speech representation network is used to form the basic speech representation sequence. When the interval between adjacent representations of the basic speech representation sequence is At that time, each frame-level speech feature is obtained by aggregating K consecutive basic speech representations, where K satisfies: ， in This indicates the preset time resolution.

3. The method according to claim 1, characterized in that, The inter-frame correlation modeling includes: for the first... Frame and the The frame-level speech features of each frame are multiplied element-wise to form frame-related features; the pairwise related features are linearly mapped, nonlinearly transformed, and projected with learnable weights to obtain inter-frame related attention; the frame-level speech feature sequence is weighted and aggregated according to the inter-frame related attention. The intra-frame context modeling adopts a convolutional enhanced self-attention structure, which includes a first feedforward sub-layer, a multi-head self-attention sub-layer, a convolutional sub-layer, a second feedforward sub-layer, and a normalization sub-layer. Both the first feedforward sub-layer and the second feedforward sub-layer adopt semi-residual connections. The multi-head self-attention sublayer introduces relative position encoding in the query representation and key representation; the convolutional sublayer sequentially includes pointwise mapping, gated linear transformation, depthwise convolution, normalization, activation, and pointwise mapping.

4. The method according to claim 1, characterized in that, The boundary probability is obtained from the boundary enhancement feature through linear mapping and Sigmoid mapping; the first in the soft boundary gating sequence Each gate value is determined according to the following formula: ， in, Indicates the first The soft boundary gating value of the frame. Indicates the first Boundary probability of a frame. Indicates the boundary center threshold. This represents the temperature parameter.

5. The method according to claim 1, characterized in that, The first in the inter-frame message control matrix M Line 1 The elements of a column are determined according to the following formula: ， in, Indicates the first Frame and the The message passing strength between frames; when there is a position with a high boundary probability between two frames, the corresponding message passing strength decreases continuously.

6. The method according to claim 5, characterized in that, Continuous modulation of inter-frame attention using the inter-frame message control matrix includes: multiplying the inter-frame attention score by the corresponding element of the inter-frame message control matrix to obtain a controlled attention score; normalizing the controlled attention score; and aggregating the frame-level speech feature sequence based on the normalization result. .

7. The method according to claim 1, characterized in that, During the training phase, a weighted sum of frame-level true / false loss and boundary loss is used as the objective function: ， in, Indicates frame-level authenticity loss. This indicates boundary loss.

8. The method according to claim 7, characterized in that, The boundary loss is either a binary cross-entropy loss that balances positive and negative samples, or a boundary focusing loss; the boundary focusing loss is determined according to the following formula: ,in, Indicates the first Frame boundary labels, Indicates the first Frame boundary prediction probability.

9. The method according to claim 1, characterized in that, The training phase also includes a cross-segment hybrid enhancement step: selecting a first speech sample and a second speech sample in the same training batch; determining the common effective frame length of the two; selecting a segmentation frame within the range of 20% to 80% of the common effective frame length; and replacing the speech segment of the first speech sample from the segmentation frame to the end of the common effective frame length with the speech segment at the corresponding position of the second speech sample. The frame-level true / false labels are replaced synchronously, and the boundary labels are regenerated based on the replaced frame-level true / false labels.

10. A local forged speech detection system based on an improved boundary-aware attention mechanism, characterized in that, include: The module includes a speech representation extraction module, a frame-level aggregation module, a boundary enhancement module, a soft boundary gating module, a boundary-aware attention module, and a frame-level discrimination module. The speech representation extraction module is used to generate a basic speech representation sequence; the frame-level aggregation module is used to form a frame-level speech feature sequence. The boundary enhancement module is used to form boundary enhancement features and generate boundary probabilities; the soft boundary gating module is used to generate soft boundary gating sequences based on the boundary probabilities and construct an inter-frame message control matrix; the boundary-aware attention module is used to continuously modulate inter-frame attention based on the inter-frame message control matrix. The frame-level discrimination module is used to output the authenticity discrimination results of each frame.

Images