A Deep Speech Forgery Detection Method and System Based on Non-Redundant Representation Modulation
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- COMMUNICATION UNIVERSITY OF CHINA
- Filing Date
- 2026-04-23
- Publication Date
- 2026-05-26
Smart Images
Figure CN122090831A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of speech signal processing and artificial intelligence security technology, and more specifically, to an edge-side speech deep forgery detection method and system based on non-redundant representation modulation, particularly an edge-side speech deep forgery detection system with both high generalization ability and low resource consumption. Background Technology
[0002] With the rapid development of generative artificial intelligence technology, text-to-speech (TTS) and voice conversion (VC) systems can synthesize realistic speech that is difficult for humans to distinguish. While these technologies bring convenience to fields such as assisted communication and virtual assistants, they also pose serious social security risks, such as financial fraud, perjury, and political manipulation. Therefore, developing reliable voice authentication systems has become an urgent need. In real-world applications, forged voices are prevalent in everyday calls and social media voice messages, placing extremely high demands on both the high generalization ability and low resource consumption of detection models. In particular, support for local deployment on the device side is required to ensure user privacy and real-time performance.
[0003] Early speech authentication methods relied on hand-designed spectral features based on expert knowledge, such as LFCC (linear frequency cepstral coefficients) and CQCC (constant Q cepstral coefficients). These features were typically combined with shallow neural network classifiers to distinguish between real and fake speech. However, these hand-designed features have limited ability to capture subtle acoustic artifacts introduced by advanced neural vocoders, making them ill-equipped to handle increasingly sophisticated forgery techniques. To overcome the limitations of hand-designed feature representation, the RawNet series and AASIST architecture introduced an end-to-end learning paradigm, utilizing one-dimensional convolutional and graph neural networks to directly extract authentication features from the original waveform, enhancing the abstraction capability of features. Although these end-to-end models have a small number of parameters, their performance drops sharply when encountering unseen forgery algorithms or cross-domain data due to the lack of large-scale prior knowledge of speech.
[0004] With breakthroughs in self-supervised learning (SSL) for speech representation learning, models such as XLS-R, WavLM, and HuberT use Transformer encoders to learn transferable audio representations from large-scale unlabeled speech corpora, demonstrating performance superior to handcrafted features and end-to-end methods. SSL models can learn general speech representations from massive amounts of unlabeled speech, providing rich acoustic features for speech authentication tasks. Mainstream speech authentication schemes based on SSL models can be divided into two categories: Full or partial fine-tuning: Allows pre-trained encoders to update parameters in downstream tasks to improve task alignment. For example, the XLSR-AASIST method replaces the handcrafted front end of AASIST with an XLS-R encoder. Full model fine-tuning is typically performed to achieve deep task adaptation, but it is computationally expensive and prone to shortcut learning on limited supervised data.
[0005] Non-selective freeze reuse: To reduce computational cost, the frozen SSL encoder is used to train only the backend classification network. For example, the XLSR+SLS method uses a sensitive layer selection module to adaptively weight and aggregate features from multiple Transformer layers, and inputs the fused representations into a lightweight classifier, which improves efficiency but retains a large number of task-independent redundant parameters.
[0006] These SSL representation utilization schemes generally suffer from a fundamental contradiction: the pre-training goal of SSL models is Automatic Speech Recognition (ASR), which emphasizes the invariance of language semantics. Therefore, its deep representations actively suppress "noise" variables, including speaker characteristics. However, speech authentication relies on these subtle acoustic features as key discrimination criteria. Directly transferring SSL models not only introduces huge computational redundancy, but the task-independent high-level semantic priors also interfere with the recognition of forgery traces, causing performance bottlenecks.
[0007] It is evident that early models based on handcrafted features and end-to-end methods, while having a small number of parameters, lacked large-scale prior knowledge of speech and struggled to cope with unseen forgery algorithms or cross-domain data. Although SSL models offer superior representational capabilities for speech forgery detection compared to handcrafted features and end-to-end methods, existing exploitation schemes suffer from the following problems: (1) Mismatch between pre-training objectives: The SSL model is mainly pre-trained for ASR tasks, and its optimization objective is to learn invariant representations of language semantics. However, speech authentication requires capturing subtle acoustic artifacts in synthesized speech, which is fundamentally contradictory to the objective of ASR. Deep networks actively suppress acoustic details considered as "noise" during pre-training, and these details are precisely the key features required for authentication tasks; (2) Parameter redundancy and weak generalization ability: Existing methods usually adopt the full model or partial fine-tuning or non-selective freeze reuse strategy. Full model or partial fine-tuning is computationally expensive and is prone to shortcut learning under limited fake data; non-selective freeze reuse, that is, aggregating all layers or using only the output layer, inherits the semantic prior of the ASR task and introduces a lot of parameter redundancy, which limits the efficiency of the model and its cross-domain generalization ability; (3) Difficulty in deployment on the edge: SSL models generally use deep Transformer encoders with a huge number of parameters, which makes it difficult to deploy on edge devices with limited computing resources, such as mobile devices or IoT devices, thus limiting the practical application value.
[0008] In summary, existing technologies suffer from problems such as mismatched pre-training targets, parameter redundancy, weak generalization ability, and difficulties in edge deployment. Therefore, there is an urgent need for a highly generalized speech authentication scheme that can maintain high detection performance while adapting to edge deployment. Summary of the Invention
[0009] In view of the problems existing in the SSL model-based speech authentication technology, such as mismatch of pre-training objectives, parameter redundancy, weak cross-domain generalization, and difficulty in edge deployment, the purpose of this invention is to provide an edge speech deep forgery detection method and system based on non-redundant representation regulation. By performing hierarchical pruning and freezing on the pre-trained SSL model, the acoustic features that contribute most to the authentication task are extracted, and combined with a lightweight metric-aware classifier, a speech authentication scheme with small parameters, strong generalization ability, and suitable for edge deployment is achieved.
[0010] On one hand, the present invention provides an end-side speech deep forgery detection method based on non-redundant representation modulation, comprising: Non-redundant representation selection: By freezing the pre-trained SSL model XLS-R with the original speech input, the frame-level acoustic representations of all layers of the SSL model XLS-R are extracted. The contribution of the SSL model XLS-R to the speech authentication task is quantified hierarchically. Based on the contribution, deep redundant parameters with low contribution are reduced through hierarchical pruning. The frame-level acoustic representation with the highest contribution is retained as the non-redundant frame-level acoustic representation, and the parameters of the retained layer are frozen. Lightweight detection network construction: The non-redundant frame-level acoustic representations are mapped to the embedding space of the speech authentication task through a feedforward neural network. The non-redundant frame-level acoustic representations are aggregated into fixed-dimensional embedding representations through an attention statistical pooling layer. The embedding representations are input into a metric-perceptual classifier based on weighted additive angular interval loss to classify real speech and fake speech on a hypersphere, resulting in an authentication inference model with a non-redundant self-supervised learning front-end and a lightweight detection network back-end. Edge-side adaptation optimization: The fake detection inference model is exported as a general format for open neural network exchange. The disk space usage is reduced by graph-level optimization and selective static quantization method that only quantizes linear layers and attention layers, so as to adapt to the resource-constrained edge environment.
[0011] On the other hand, the present invention also provides an end-side deep forgery detection system based on non-redundant representation modulation, applied to electronic devices, comprising: The non-redundant representation selection unit is used to extract the frame-level acoustic representations of all layers of the pre-trained SSL model XLS-R by freezing the original speech input, quantize the contribution of the SSL model XLS-R to the speech authentication task in layers, reduce the deep redundant parameters with low contribution through layer pruning according to the contribution, retain the frame-level acoustic representation with the highest contribution as the non-redundant frame-level acoustic representation, and freeze the parameters of the retained layer. The lightweight detection network construction unit is used to map the non-redundant frame-level acoustic representation to the embedding space of the speech authentication task through a feedforward neural network. The non-redundant frame-level acoustic representation is aggregated into a fixed-dimensional embedding representation through an attention statistical pooling layer. The embedding representation is input into a metric-perceptual classifier based on weighted additive angular interval loss to classify real speech and fake speech on a hypersphere, thus obtaining an authentication inference model with a non-redundant self-supervised learning front end and a lightweight detection network back end. The edge-side adaptation optimization unit is used to export the fake detection inference model into a general format for open neural network exchange. It reduces disk space usage through graph-level optimization and selective static quantization methods that quantize only linear layers and attention layers, so as to adapt to resource-constrained edge-side environments.
[0012] In another aspect, the present invention also provides an electronic device, the electronic device including a memory, a processor, and an end-side speech deep spoofing detection program based on non-redundant representation regulation stored in the memory and executable on the processor, wherein the end-side speech deep spoofing detection program based on non-redundant representation regulation, when executed by the processor, implements the end-side speech deep spoofing detection method based on non-redundant representation regulation as described above.
[0013] The aforementioned edge-side speech deep forgery detection method and system based on non-redundant representation regulation according to the present invention extracts the acoustic features that contribute most to the forgery detection task by performing hierarchical pruning and freezing on the pre-trained SSL model, and combines it with a lightweight metric-aware classifier to achieve a speech forgery detection scheme with small parameters, strong generalization ability and suitable for edge deployment.
[0014] Compared with the prior art, the present invention has the following beneficial effects: (1) A systematic layer contribution analysis mechanism was established: by constructing a weighted aggregation framework, learnable layer weight parameters were introduced to quantitatively evaluate the contribution of each layer representation of the SSL model to the anti-counterfeiting task, reveal the dynamic distribution law of layer contributions, and provide a decision basis for subsequent model pruning; (2) Task-related representation regulation was achieved: Based on the layer contribution analysis results, the shallow layer that contributes the most to the anti-counterfeiting task was selectively retained, the redundant deep network was pruned, and the parameters of the retained layer were frozen to avoid the representation distribution shift introduced by fine-tuning, thereby obtaining a compact and task-aligned non-redundant representation. (3) Design a lightweight detection network. By constructing a lightweight backend classifier based on FFN and ASP, the variable-length frame-level representations are aggregated into fixed-length sentence-level embeddings, and the WAAM loss function is used to enhance the geometric separation between categories. (4) Enhanced cross-domain robustness: The RawBoost data augmentation strategy is integrated to simulate real transmission conditions (impulse noise, additive noise, channel convolution distortion) in the waveform domain, thereby improving the model’s robustness to channel changes and unseen attacks.
[0015] (5) Achieved edge-side deployability: By generalizing and quantizing the model, the disk space occupied by the model is reduced, and while maintaining detection accuracy, low-latency inference on mobile processors is achieved, meeting the needs of edge-side resource constraints and real-time detection.
[0016] To achieve the foregoing and related objectives, one or more aspects of the invention include the features which will be described in detail below and specifically pointed out in the claims. The following description and accompanying drawings illustrate certain exemplary aspects of the invention. However, these aspects indicate only a few of the various ways in which the principles of the invention can be used. Furthermore, the invention is intended to include all such aspects and their equivalents. Attached Figure Description
[0017] Other objects and results of the invention will become more apparent and readily understood with reference to the following description taken in conjunction with the accompanying drawings and the contents of the claims, and with a more complete understanding of the invention. In the drawings: Figure 1 This is a schematic diagram of the end-side speech deep forgery detection method based on non-redundant representation modulation according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the implementation framework of the end-side speech deep forgery detection method based on non-redundant representation modulation according to an embodiment of the present invention. Figure 3 This is a schematic diagram of a weighted aggregation framework structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the model generalization and quantization process according to an embodiment of the present invention; Figure 5 A heatmap of hierarchical contribution according to an embodiment of the present invention; Figure 6 The inference delay curve of the end-side device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the logical structure of an end-side speech deep forgery detection system based on non-redundant representation modulation according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the logic structure of an electronic device according to an embodiment of the present invention.
[0018] In all the accompanying drawings, the same reference numerals indicate similar or corresponding features or functions. Detailed Implementation
[0019] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description of one or more embodiments.
[0020] To address the core issues of existing SSL model-based speech spoofing detection techniques, such as mismatched pre-training objectives, parameter redundancy, weak cross-domain generalization, and difficulty in edge deployment, this invention proposes an edge-based deep forgery detection method based on non-redundant representation modulation. The overall technical chain of this invention consists of non-redundant representation selection → lightweight detection network construction → edge-based adaptation optimization. Through layer contribution analysis-guided pruning and freezing of the pre-trained SSL model (XLS-R), highly discriminative acoustic features are extracted. These features are then combined with a lightweight metric-based perceptual classifier, and edge-based low-resource, high-real-time speech spoofing detection is achieved through model generalization and quantization.
[0021] To better illustrate the technical solution of the present invention, some of the technical terms involved in the present invention will be briefly explained below.
[0022] Deep speech spoofing detection (speech authentication) refers to the technology that distinguishes between naturally generated real human speech and spoofed speech synthesized / converted by artificial intelligence technologies (such as TTS and VC). It is an intersection of artificial intelligence security and speech signal processing. In this solution, the technology is designed for deployment on edge devices and requires both high generalization ability and low resource consumption.
[0023] Voice conversion (VC) refers to the technology of converting the speech features of the source speaker into the features of the target speaker while maintaining the semantic content of the speech. It is another core generation method of forged speech and together with TTS, constitutes the main detection target of this scheme.
[0024] Equal Error Rate (EER) is a core evaluation metric in the field of speech authentication. It refers to the value at which the error rate of fake speech being judged as real speech is equal to the error rate of real speech being judged as fake speech. The lower the EER value, the stronger the model's detection accuracy and discrimination ability. In this invention, it serves as the core evaluation standard for model performance.
[0025] Cross-domain generalization capability refers to the model's detection performance on forgery algorithms, channel environments, and datasets not seen in the training set. This solution focuses on improving the model's cross-domain generalization capability against unknown forgery attacks, channel distortion in real-world scenarios, and unconstrained samples in the wild.
[0026] Edge devices refer to local hardware devices that are far from cloud servers, such as mobile phones, embedded Linux devices, and laptops. These devices generally have limited computing power, memory, and disk space. The core application scenario of this invention is for such devices, requiring the model to achieve low latency and local inference without dedicated acceleration.
[0027] Self-supervised learning (SSL) is an advanced paradigm of unsupervised learning. It learns general feature representations of data by constructing self-supervised task (such as mask prediction in speech) from unlabeled data without the need for manual labeling. The SSL model used for speech representation learning in this invention is a pre-trained general speech model that provides acoustic features for the anti-spoofing task.
[0028] XLS-R is a cross-language large-scale speech self-supervised pre-training model based on Transformer. In this invention, it serves as the core speech representation extraction model. Through layer feature analysis and pruning, non-redundant anti-spoofing features are obtained.
[0029] Frame-level representation and sentence-level representation are two feature representation granularities for speech signals. Frame-level representation divides continuous speech into segments of fixed time length, and the features corresponding to each segment are variable-length sequence features. Sentence-level representation aggregates all frame-level representations of the entire speech segment to obtain fixed-length features, which are the final input features for model classification. In this scheme, the frame-level representation is converted into fixed-dimensional sentence-level representation through the ASP mechanism.
[0030] Layer contribution refers to the degree to which the representations output by each Transformer layer of the SSL model effectively contribute to the specific task of speech authentication. In this invention, a weighted aggregation framework is used to quantitatively evaluate layer contribution, providing a basis for decision-making on model pruning.
[0031] Model pruning refers to a model compression technique that reduces the number of model parameters and computational complexity by removing layers / parameters in a neural network that contribute little or no to the task. In this invention, it is hierarchical pruning, which retains the shallowest layers with the highest contribution and prunes redundant deep layers.
[0032] In this invention, representation regulation specifically refers to the process of selecting non-redundant acoustic representations that are highly aligned with the speech authentication task from the general speech representations of the SSL model through methods such as layer contribution analysis, model pruning, and parameter freezing, while eliminating task-irrelevant semantic priors and redundant features.
[0033] Feed-Forward Neural Network (FFN) is a neural network structure in which features are passed from the input layer to the output layer in a unidirectional order without loops or feedback connections. In this invention, two layers of FFN are used as feature projection modules to map the 1024-dimensional high-dimensional frame-level representation into 128-dimensional low-dimensional embedded features specific to the speech authentication task.
[0034] The Attention Statistical Pooling (ASP) layer is a feature aggregation module that combines attention mechanisms with statistical pooling. In this invention, it is used to aggregate variable-length frame-level representations into fixed-length sentence-level representations. Its core functions are: ① to focus on time frames that are key to the anti-counterfeiting task through the attention mechanism; ② to calculate the weighted mean and variance of the frame-level representations and concatenate them to obtain statistical embedding features.
[0035] The Weighted Additive Angular Margin (WAAM) loss, in this invention, is a metric learning loss function and an improved version of the additive angular margin loss. It enforces angular decision boundaries on the hypersphere and introduces class-specific weights to alleviate the class imbalance problem between real and fake speech. Its core function is to enhance the intra-class compactness (feature clustering of similar samples) and inter-class separability (feature distancing of dissimilar samples) of real and fake speech features, thereby improving the model's generalization ability to unknown attacks.
[0036] The metric-aware classifier is a metric-learning-based classifier that achieves effective separation between categories by optimizing the distance / angle metric of samples in the feature space. In this invention, the classifier is constructed based on WAAM loss and optimizes the feature separation degree of real and fake speech on the hypersphere.
[0037] The hypersphere refers to the high-dimensional spherical space in which the feature vector is normalized. In this invention, the sentence-level representation is projected onto the hypersphere for classification, which can effectively improve the robustness of feature measurement and avoid the interference of feature amplitude on the classification result.
[0038] RawBoost data augmentation is a data augmentation strategy for raw speech waveforms. In this invention, the impulse noise, additive noise, channel convolution distortion and other real speech transmission conditions are simulated in the waveform domain to improve the robustness of the model to channel changes, environmental noise and real scene distortion.
[0039] Open Neural Network Exchange (ONNX) is a cross-framework universal format for neural network models, enabling the universal deployment of models trained using deep learning frameworks such as PyTorch / TensorFlow on different hardware / software platforms. In this invention, the trained PyTorch model is exported in ONNX format to ensure the model's compatibility with different edge devices.
[0040] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0041] Figure 1 , Figure 2 The flow and implementation framework of the edge-side speech deep forgery detection method based on non-redundant representation regulation in the embodiments of the present invention are illustrated by way of example.
[0042] like Figure 1 and Figure 2 As shown in the figure, the edge-side speech deepfake detection method based on non-redundant representation regulation (SDD-NRR) provided in this embodiment involves speech deepfake detection (speech authentication) technology. It is applicable to edge-side devices with limited computing power and real-time speech authentication schemes with high cross-domain generalization ability. The main steps are as follows: S110: Selection of non-redundant representations: By freezing the pre-trained SSL model XLS-R with the original speech input, the frame-level acoustic representations of all layers of the SSL model XLS-R are extracted. The contribution of the SSL model XLS-R to the speech authentication task is quantified hierarchically. Based on the contribution, deep redundant parameters with low contribution are reduced through hierarchical pruning. The frame-level acoustic representation with the highest contribution is retained as the non-redundant frame-level acoustic representation, and the parameters of the retained layer are frozen.
[0043] S120: Lightweight detection network construction: The non-redundant frame-level acoustic representation is mapped to the embedding space of the speech authentication task through a feedforward neural network. The non-redundant frame-level acoustic representation is aggregated into a fixed-dimensional embedding representation through an attention statistical pooling layer. The embedding representation is input into a metric-perceptual classifier based on weighted additive angular interval loss to classify real speech and fake speech on a hypersphere, thus obtaining an authentication inference model with a non-redundant self-supervised learning front end and a lightweight detection network back end. S130: End-side adaptation optimization: The fake detection inference model is exported as a general format for open neural network exchange. The disk space usage is reduced by graph-level optimization and selective static quantization method that only quantizes linear layers and attention layers, so as to adapt to the resource-constrained end-side environment.
[0044] The purpose of this invention is to solve the problems of high parameter redundancy, weak cross-domain generalization ability, and difficulty in edge deployment faced by existing speech authentication technologies. It helps to ensure that the pre-trained SSL model is consistent with the goal of speech authentication task, reduces task-irrelevant redundant parameters and semantic priors, and provides an edge speech authentication solution with both high generalization ability and low resource consumption.
[0045] The above steps of the present invention will now be described in detail with reference to more specific embodiments.
[0046] like Figure 2 As shown, the SDD-NRR method provided in this embodiment is mainly implemented through two core components: 1. Non-redundant XLS-R extractor: Extracts a task-aligned compact representation by performing layer pruning and parameter freezing guided by layer contribution analysis on the SSL model XLS-R; 2. Metric-aware FFN-ASP classifier: A lightweight FFN layer combined with the ASP mechanism and WAAM loss achieves efficient discriminative learning.
[0047] The non-redundant XLS-R extractor is used to implement the non-redundant representation selection in step S110. To evaluate the contribution of each XLS-R layer to the speech authentication task, a weighted aggregation framework is designed in the non-redundant representation selection stage. Figure 3 A weighted aggregation framework diagram according to an embodiment of the present invention is shown, such as Figure 3 As shown, in this embodiment, the original waveform is input into the frozen complete XLS-R model, and the hidden states of all layers are extracted using formula (1.1): (1.1) in, This represents the number of Transformer layers in XLS-R. Indicates the first The frame-level feature sequence output by the layer. For time frames, For feature dimensions.
[0048] Set independent learnable scalar parameters for each layer's output. The layer weights are obtained by normalization using the Softmax function. , l This indicates the XLS-R Transformer's... l layer: (1.2) The denominator of formula (1.2) indicates that the sum of all learnable scalar parameters is a fixed value, while the numerator represents the variable values. This means that the sum of all learnable scalar parameters is a fixed value. lLearnable scalar parameters of the layer, molecules change with the number of layers. l change.
[0049] Through time-normalized stable training, the frame-level acoustic representations of each layer are time-normalized and then weighted and summed according to the layer weights to obtain the aggregated representation: (1.3) in, Temporal normalization of the representation layer is used to stabilize training.
[0050] During the training process, Parameters are updated via backpropagation. After training converges, the parameters are updated through the layer weights. Quantify the contribution of each layer to the counterfeit detection task.
[0051] In this invention, the learning objective of layer weights is linked to their contribution to the anti-spoofing task (higher weights for layers with greater contributions, and vice versa). Based on contribution, layer pruning reduces low-contribution, deeply redundant parameters, retaining the frame-level acoustic representation with the highest contribution as the non-redundant frame-level acoustic representation. Pruning reduces low-contribution, deeply redundant parameters relative to all other deeply redundant parameters besides the highest contribution. This contribution-oriented update relies on the loss feedback from the anti-spoofing task. Only by involving layer weights in the training of the anti-spoofing task and backpropagating through the model's predicted loss can the weights be updated in a direction that improves anti-spoofing performance. The aggregated representation, as the fused feature after weighting the layer weights, is the only training feature that can be input into the anti-spoofing classifier.
[0052] Therefore, the aggregated representation is a dedicated intermediate carrier designed in this invention to quantitatively evaluate the contribution of each layer of frame-level representations to anti-counterfeiting. Since layer weights are independent, learnable scalar parameters without any practical feature meaning, they cannot be input alone into subsequent anti-counterfeiting classifiers for training; while each layer of frame-level acoustic representations are high-dimensional sequence features, representing the model's actual representation of speech features. In this embodiment, the aforementioned aggregated representation is used to weightedly fuse meaningless scalar layer weights with layered frame-level acoustic representations that have practical feature meaning, integrating the scattered layered features into a single, fused feature that can be recognized by the anti-counterfeiting task. This establishes a strong correlation between the originally isolated layer weights and layered representations, making the layer weights the core parameters influencing the fused feature expression. This allows the scalar weights to have a practical carrier for participating in anti-counterfeiting task training, which is the basis for the layer weights to be updated by the task training.
[0053] Because the frame-level acoustic representations of different layers exhibit inconsistencies in feature distribution and numerical scale due to differences in model hierarchy (e.g., shallow features are biased towards acoustic details and have a small numerical scale; deep features are biased towards semantic abstraction and have a large numerical scale), direct weighted fusion would overweight layers with large numerical scales, causing the layer weights to be learned biased towards numerical scale rather than actual contribution to the anti-spoofing task, resulting in distorted learning results. Temporal normalization is used to achieve unified calibration of features between layers, making the learning of layer weights more stable and fairer.
[0054] This invention eliminates the inter-layer distribution differences of frame-level representations by using temporal normalization, so that the features of all layers are at the same numerical scale and distribution range. This ensures that the weighting of layer weights is a fair weighting based on task contribution, rather than a biased weighting based on numerical scale. Ultimately, this makes the learning results of layer weights more stable and more in line with the actual needs of the anti-counterfeiting task.
[0055] To evaluate the independent discriminative power of a specific layer, according to the weight distribution Select the level with the highest contribution and use only a single level representation. As input for training the classifier : (1.4) This verifies whether information from a single level is sufficient to achieve high-performance detection without being affected by irrelevant levels.
[0056] Based on the layer contribution analysis results, a single optimal layer was selected as the non-redundant XLS-R extractor. Only the first layer was retained. Layered Transformer, discarding deeper layers with lower contributions, significantly reduces parameter redundancy: Representation of the representation Perform pruning operations, retaining the previous branches. b layer; (1.5) Take the output of the Kth layer It serves as a frame-level feature input to the classifier network.
[0057] Freeze all parameters of the XLS-R layer to avoid introducing representation distribution shifts during downstream training.
[0058] In this embodiment, the SDD-NRR method of this invention is implemented using a deep learning model developed, trained, and derived based on the PyTorch framework. Therefore, the model used in this embodiment is a standard PyTorch model. Accordingly, the above-mentioned non-redundant representation selection stage can be simply summarized as follows: load the pre-trained XLS-R model in PyTorch, then propagate forward to extract frame-level acoustic representations of all layers, construct learnable layer weights alpha (PyTorch parameters), calculate aggregate representations (PyTorch tensor operations), train and update weights → obtain layer contribution, prune → retain the optimal single layer → form a non-redundant representation extractor.
[0059] In the lightweight detection network construction phase, this embodiment employs a two-layer FFN to map the 1024-dimensional frame-level features output by the non-redundant XLS-R extractor to a 128-dimensional task-specific embedding space, resulting in 128-dimensional low-dimensional task-specific frame-level embedding features. Each FFN block consists of a linear transformation, a ReLU activation function, and a Dropout regularization layer. Represents the ReLU activation function. This indicates the discard layer regularization operation; (1.6) in, For the first The input representation of a frame, , The weight matrix is the linear transformation matrix. For the corresponding bias vector, D hidden =128, These are the frame-level features after projection.
[0060] After completing the FFN feature projection, the ASP mechanism is used to aggregate the variable-length frame sequence to generate a fixed-length sentence-level statistical embedding. At the same time, the attention mechanism is used to focus on the time frames that are key to the anti-counterfeiting task.
[0061] Specifically, as an example, methods for aggregating non-redundant frame-level acoustic representations into fixed-dimensional embedded representations using ASP include: Calculate the attention score for the low-dimensional task-specific frame-level embedding features in each frame: (1.7) Here is the attention weight matrix. For attention bias vectors, It is a learnable attention scoring vector used to project frame-level embeddings into scalar scores, thereby measuring the importance of each frame to the detection result.
[0062] The weights for frame t are obtained by normalizing the attention scores: (1.8) The weighted mean of the low-dimensional task-specific frame-level embedding features is calculated based on the time weight. With weighted variance : (1.9) Finally, the weighted mean and weighted variance are concatenated to obtain the fixed-dimensional embedding representation as the final statistical embedding. .
[0063] After obtaining the fixed-dimensional embedding representation, the embedding representation can be input into the metric-aware classifier (WAAM) based on weighted additive angular interval loss to classify real speech and fake speech on the hypersphere, resulting in a fake speech detection inference model (PyTorch model) with a non-redundant self-supervised learning front end and a lightweight detection network back end.
[0064] Specifically, WAAM enhances the intra-class compactness and inter-class separability of real and fake samples in the embedding space by enforcing angular decision boundaries on the hypersphere, thereby improving the model's generalization ability to unknown attacks. It projects statistical embeddings onto the metric embedding space: (1.10) For the projection matrix, For the corresponding bias vector; Training batches include A statement sampled from either real or fake speech. Forged tags for individual statements ,in Represents real voice. This indicates forged voice. Let's assume... For normalized embedding, To normalize the true category centers, For the first i The angle between the normalized embedding of each statement and the normalized true class center. WAAM loss is defined as: (1.11) in, Indicates a non-target class. Class-specific weights are assigned to mitigate class imbalance. An additive angular interval is used to enhance class separation. This is the scaling factor.
[0065] After completing the lightweight detection network construction and obtaining a counterfeit detection reasoning model with a non-redundant self-supervised learning front-end and a lightweight detection network back-end, edge adaptation and optimization can be performed.
[0066] Figure 4 This illustrates a process for generalizing and quantifying the counterfeit detection reasoning model during the edge-side adaptation optimization phase, according to an embodiment of the present invention. Specifically, as... Figure 4 As shown, in the edge-side adaptation and optimization stage, the fake detection inference model is first subjected to serialization and graph-level optimization. The trained PyTorch model is exported to the ONNX universal format, and graph-level optimization is performed through ONNX Runtime, including operator fusion (merging continuous operations into a single kernel call) and constant folding (pre-computing static nodes), to reduce memory access overhead, improve inference efficiency, and ensure model compatibility across different hardware platforms.
[0067] Static quantization is an offline quantization method performed before inference. It estimates the distribution of activation values in each layer of the model using a small amount of representative calibration data, and determines the scaling factor and zeros required for quantization accordingly. Compared to dynamic quantization, statically quantized models can be computed entirely using INT8 during the inference phase, resulting in higher execution efficiency and lower power consumption on edge devices.
[0068] Specifically, as an example, 120 training speech samples were randomly selected to form a calibration dataset, and a percentile calibration strategy was used to obtain a stable quantization range. The model weights were mapped from 32-bit floating-point numbers (FP32) to 8-bit integers (INT8), reducing the storage requirement for each parameter from 4 bytes to 1 byte, compressing disk space usage by approximately 75%. Since the CNN feature extractor in the XLS-R front end is extremely sensitive to numerical changes, quantizing the convolutional layers can cause a qualitative change in feature representation. Therefore, the static quantization used in this embodiment is selective static quantization, which only quantizes the linear layers and attention layers in the fake-detection inference model.
[0069] The optimized model consumes low resources, performs inference entirely on the CPU, does not rely on a dedicated NPU for acceleration, and is compatible with various edge operating systems such as Android and embedded Linux. It can achieve sub-second inference on mobile chips and real-time inference within 500ms on mainstream laptop CPUs, meeting the needs of real-time voice authentication on edge devices.
[0070] To verify the effectiveness of the non-redundant representations selected by the proposed end-side speech deep forgery detection method (SDD-NRR) based on non-redundant representation modulation in speech forgery detection tasks, this section designs multiple sets of experiments in hierarchical contribution analysis and comparison with mainstream models, covering objective evaluation of the experiments and ablation analysis.
[0071] Table 1. Statistical information of the dataset
[0072] The specific experimental setup is as follows: 1.1 Experimental Environment Based on the PyTorch deep learning framework, the training hardware is a single NVIDIA 4090D 24G, and the inference test covers multiple platforms including mobile, laptop, and desktop CPUs.
[0073] 1.2 Speech Preprocessing
[0074] All audio was sampled at a uniform 16kHz rate and a fixed length of 4 seconds (64,000 sampling points). Short audio segments were concatenated in a loop, while long audio segments were truncated. Amplitude normalization was also performed, and then RawBoost was used to enhance the audio.
[0075] Before selecting non-redundant representations, the training data is processed to simulate real transmission conditions in the waveform domain, including impulse noise, additive noise, and channel convolutional distortion, to obtain the original speech for which non-redundant representations are selected in step S110. In the above speech preprocessing, all input audio signals are normalized to a fixed duration of 4 seconds (corresponding to 64,000 sampling points at a 16 kHz sampling rate). For shorter speech segments, padding is performed to the target length; for longer speech segments, truncation is performed. Subsequently, the RawBoost data augmentation method is applied to the waveform to simulate channel distortion in a real environment, specifically including the following three types of noise: Impulsive Signal-Dependent Additive Noise (ISD):
[0076] Stationary Signal-Independent Additive Noise (SSI):
[0077] Linear and non-linear convolutive noise (LnL):
[0078] in: Represents the original audio waveform. Indicates impulse response, Indicates noise signal, and This is the scaling factor. This represents the convolution operation. This indicates element-wise multiplication.
[0079] 1.3 Training Hyperparameters
[0080] The batch size is 32, the training epochs are 50, the optimizer is Adam, the learning rate adopts a linear warm-up strategy, and stabilizes at 1e-4 after 1000 warm-up steps, the weight decays by 1e-4, and the gradient clipping threshold is 1.0.
[0081] 1.4 Evaluation Indicators
[0082] We adopted the Equal Error Rate (EER), a commonly used metric in the field of speech authentication. The lower the EER value, the higher the detection accuracy and the stronger the discrimination ability of the model.
[0083] 1.5 Dataset
[0084] ASVspoof 2019 LA (19LA): Used for training, development, and evaluation. The industry-standard training set contains various TTS / VC spoofing attacks. The development set has no speaker overlap with the training set. The test set contains 13 unknown spoofing attacks not seen in the training set, verifying the model's generalization ability to unknown attacks.
[0085] ASVspoof 2021 LA / DF (21 LA / DF): Used to evaluate channel robustness, simulate real transmission compression and coding distortion, and verify the model's robustness to channel interference.
[0086] Fake-or-Real (FoR): Used to evaluate cross-domain generalization ability. It is a cross-domain fake speech generated by an open-source TTS system to verify the model's ability to generalize to unknown synthesis engines.
[0087] In-The-Wild (ITW): Used to evaluate real-world scenarios in the wild, using fake celebrity voice samples collected from social media platforms in an unconstrained acoustic environment to verify performance in real-world scenarios.
[0088] Under the above experimental setup, an experiment was conducted to verify the effectiveness of the non-redundant characterization selection: A weighted aggregation framework is constructed, incorporating the outputs of all 24 Transformer layers in the XLS-R frontend. During training, each layer's output is assigned an independent, learnable weight parameter. When the model updates parameters during backpropagation, the changes in the weights directly reflect the contribution of that layer's representation to the final classification decision. By monitoring the evolution of the weights at each layer during training, a heatmap of layer contribution is generated to qualitatively determine which layers in the model retain the richest speech authentication information.
[0089] After sufficient iterative training on the 19LA training set, the trainable weights are normalized to obtain... Figure 5The weight heatmap shown reveals that the contributions of layers 5 and 19 significantly increase with training, exhibiting a bimodal distribution; conversely, the contributions of deeper layers closer to the output layer (such as layer 24) continuously decrease. Excessive semantic abstraction in deep networks can filter out acoustic forgery traces, making XLS-R's shallower networks more suitable for speech forgery detection tasks.
[0090] Based on the qualitative analysis, quantitative validation was further conducted. According to the heatmap, shallow layers (layers 4, 5, and 6), deep layers (layers 18, 19, 20, and 21), and the output layer (layer 24) were selected as independent front-end characterization extractors, and trained and validated with identical classifiers. The experimental results are as follows: Table 2 Comparison of characterization layer performance (bold for best value, underline for second best value)
[0091] The experimental results in Table 2 show that the fifth layer representation achieved the lowest equal error rate (EER) on all test sets while maintaining a low parameter count. In contrast, the output layer has a parameter count as high as 316M, but its EER on the cross-domain test set ITW reaches 21.4136%, which is much higher than the 5.2256% of the fifth layer. This indicates that deeper representations are more prone to overfitting the training domain, leading to a significant decrease in cross-domain generalization ability.
[0092] Furthermore, this invention attempts to aggregate representations from different layers using a weighted fusion network, including the fusion of the bimodal layers 5 and 19, and the fusion of shallow representations from layers 4, 5, and 6. The performance comparison of the corresponding representation layers is shown in Table 3. It can be seen that the average performance of both fusion methods is not better than using layer 5 alone: the average EER of the 5+19 fusion is 2.5291%, and the average EER of the 4+5+6 fusion is 2.7619%, both higher than the 2.2626% of a single layer 5. This indicates that simple inter-layer feature superposition does not bring stable gains. The reason may be that the representations from layer 19 and deeper have stronger abstract semantic information, incorporating more redundant features related to the training domain during training; while the shallow representations, although retaining richer acoustic features, have weaker representation capabilities for forgery traces than layer 5, and do not form effective complementarity after fusion with layer 5, thus the final performance fails to surpass that of a single-layer representation.
[0093] Table 3. Model Performance Comparison (Bold for best value, underline for second-best value)
[0094] As shown in Table 3 above, although end-to-end methods (RawNet2, AASIST) have a small number of parameters, their average EER is 24-35%, and their performance degrades significantly on cross-domain datasets, verifying the generalization bottleneck of training from an initial state without prior speech knowledge. Existing SSL-based schemes (XLSR-AASIST, XLSR-Conformer, XLSR-Mamba) outperform end-to-end methods, but their parameter counts all exceed 300M, and their performance degrades significantly on cross-domain datasets such as FoR and ITW. Our invention, SDD-NRR, achieves the best average performance of 2.78% on all evaluation datasets with only 76.4M parameters, achieving the optimal balance between performance and efficiency. On the challenging ITW dataset, it achieves a low EER of 5.22%, significantly outperforming the comparison methods.
[0095] This invention systematically analyzes the actual contributions of the RawBoost data augmentation and WAAM loss function optimization strategies through ablation experiments. As shown in Table 4, after removing RawBoost, the EER of the ITW field dataset increased from 5.2256% to 8.1838%, indicating that the method can effectively enhance the model's adaptability to unknown attacks. The WAAM loss function is used to improve inter-class discrimination. When it is replaced by the traditional weighted cross-entropy (WCE) loss, the EER of both the 19LA and 21LA datasets within the domain shows a significant increase, indicating that the angular interval penalty mechanism introduced by WAAM can simultaneously improve intra-class compactness and inter-class separability, thereby strengthening the ability to distinguish between real and fake speech features. It should be noted that WCE shows little difference from WAAM on the 21DF dataset, and even has a slight advantage. Therefore, the improvement of WAAM is not absolutely superior on all datasets. In summary, the combined use of RawBoost and WAAM achieves better or more stable results on multiple datasets and optimizes the model performance, verifying the effectiveness of the proposed optimization strategy.
[0096] Table 4. Results of ablation experiments using the optimized strategy (optimal values are in bold).
[0097] This experiment also verified the effectiveness of the edge-side optimization scheme of the present invention, which is divided into two parts: quantization performance comparison and edge-side inference latency test.
[0098] 1. Static quantization performance comparison
[0099] Table 5 below shows the performance comparison results before and after model quantization, proving that the static quantization scheme achieves a good balance between model compression and performance preservation. Notably, the quantized model outperforms the original FP32 model in terms of EER on the 21LA / DF test set. Analysis suggests that INT8 quantization, by discretizing the weights, introduces a slight but beneficial regularization effect, thereby improving the model's robustness and generalization ability, further validating the feasibility of this invention in practical applications.
[0100] Table 5 Performance Comparison of Quantization Models
[0101] 2. End-side inference latency test
[0102] This test targets the end-to-end inference latency of a single 4-second audio clip, using only the device's CPU without enabling any hardware acceleration. The test results are as follows: Figure 6 As shown, the optimized model of this invention achieves sub-second inference on mobile chips and low-latency inference within 500ms on mainstream laptops and desktop devices, fully meeting the needs of real-time authentication scenarios such as real-time calls and live streaming, and possessing extremely strong edge-side deployment capabilities.
[0103] The above describes an edge-side speech deep forgery detection method based on non-redundant representation modulation according to embodiments of the present invention. It can be determined that using the above-described edge-side speech deep forgery detection method based on non-redundant representation modulation of the present invention for speech deep forgery detection can effectively meet the requirements of limited edge resources and real-time detection. The above-described edge-side speech deep forgery detection method based on non-redundant representation modulation of the present invention can be implemented in software, in hardware, or in a combination of both.
[0104] Corresponding to the above-mentioned end-side speech deep forgery detection method based on non-redundant representation modulation, the present invention also provides an end-side speech deep forgery detection system based on non-redundant representation modulation.
[0105] Figure 7 The framework structure of an end-side speech deep forgery detection system 700 based on non-redundant representation modulation according to an embodiment of the present invention is shown. Figure 7 As shown, the edge-side speech deep forgery detection system based on non-redundant representation modulation provided by the present invention utilizes the edge-side speech deep forgery detection method based on non-redundant representation modulation as described above for speech deep forgery detection. According to the implemented functions, the edge-side speech deep forgery detection system 700 based on non-redundant representation modulation mainly includes a non-redundant representation selection unit 710, a lightweight detection network construction unit 720, and an edge-side adaptation optimization unit 730.
[0106] The unit of this invention can also be called a module, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.
[0107] Furthermore, in one possible implementation of the present invention, the functions of each module / unit are as follows: The non-redundant representation selection unit 710 is used to extract the frame-level acoustic representations of all layers of the pre-trained SSL model XLS-R by freezing the original speech input, quantize the contribution of the SSL model XLS-R to the speech authentication task in layers, reduce the deep redundant parameters with low contribution through layer pruning according to the contribution, retain the frame-level acoustic representation with the highest contribution as the non-redundant frame-level acoustic representation, and freeze the parameters of the retained layer. The lightweight detection network construction unit 720 is used to map the non-redundant frame-level acoustic representation to the embedding space of the speech authentication task through a feedforward neural network, aggregate the non-redundant frame-level acoustic representation into a fixed-dimensional embedding representation through an attention statistical pooling layer, and input the embedding representation into a metric-perceptual classifier based on weighted additive angular interval loss to classify real speech and fake speech on a hypersphere, thereby obtaining an authentication inference model with a non-redundant self-supervised learning front end and a lightweight detection network back end. The edge-side adaptation optimization unit 730 is used to export the fake detection inference model into a general format for open neural network exchange. It reduces disk space usage through graph-level optimization and selective static quantization methods that quantize only linear layers and attention layers, so as to adapt to resource-constrained edge-side environments.
[0108] The aforementioned edge-side speech deep forgery detection system based on non-redundant representation modulation is an implementation method corresponding to the aforementioned edge-side speech deep forgery detection method based on non-redundant representation modulation. Its specific execution steps can be referred to the specific embodiments of the aforementioned edge-side speech deep forgery detection method based on non-redundant representation modulation, and will not be described in detail here.
[0109] like Figure 8 As shown, the present invention also provides an electronic device 1 for implementing an end-side speech deep forgery detection method based on non-redundant representation modulation.
[0110] The electronic device 1 may include a processor 10, a memory 11, and a bus. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as an edge-side deep speech spoofing detection program 12 based on non-redundant representation control. The memory 11 may include both internal storage units of the aforementioned video effects editing system and / or edge-side deep speech spoofing detection system based on non-redundant representation control, and external storage devices. The memory 11 can be used not only to store application software and various types of data, such as the code of the edge-side deep speech spoofing detection program based on non-redundant representation control, but also to temporarily store data that has been output or will be output.
[0111] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory, magnetic storage, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of an edge-side deep speech spoofing detection program based on non-redundant representation control, but also to temporarily store data that has been output or will be output.
[0112] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units, microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, connecting various components of the entire electronic device via various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., end-side deep speech spoofing detection programs based on non-redundant representation regulation) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.
[0113] The bus can be a peripheral component interconnection standard bus or an extended industry standard structure bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.
[0114] Figure 8 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 8The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0115] The end-side speech deep forgery detection program 12 based on non-redundant representation control stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can realize the end-side speech deep forgery detection method based on non-redundant representation control as described above.
[0116] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the above embodiments, and will not be repeated here. Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.
[0117] This invention also provides a computer-readable storage medium, which may be non-volatile or volatile, wherein a computer program is stored, which, when executed by a processor, implements the end-side deep speech forgery detection method based on non-redundant representation modulation as described above.
[0118] Specifically, the specific implementation method of the computer program when executed by the processor can be referred to the description of the relevant steps in the embodiment of the edge-side speech deep forgery detection method based on non-redundant representation regulation, and will not be repeated here.
[0119] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0120] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0121] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0122] The edge-side speech deep forgery detection method and system based on non-redundant representation modulation according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the edge-side speech deep forgery detection method and system based on non-redundant representation modulation proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the contents of the appended claims.
Claims
1. A method for end-side speech deep spoofing detection based on non-redundant representation regulation, characterized in that, The method comprises the following steps: Non-redundant representation selection: through a pre-trained SSL model XLS-R with frozen original speech input, frame-level acoustic representations of all layers of the SSL model XLS-R are extracted, the contribution of the SSL model XLS-R to a speech authentication task is quantified layer by layer, deep-layer redundant parameters with low contribution are reduced through layer-wise pruning according to the contribution, and frame-level acoustic representations with the highest contribution are reserved as non-redundant frame-level acoustic representations, and parameters of the reserved layers are frozen; Lightweight detection network construction: the non-redundant frame-level acoustic representations are mapped to an embedding space of the speech authentication task through a feedforward neural network, the non-redundant frame-level acoustic representations are aggregated into embedding representations with a fixed dimension through an attention statistical pooling layer, the embedding representations are input into a metric-aware classifier based on a weighted additive angular margin loss, and real speech and fake speech are classified on a hypersphere to obtain an authentication reasoning model with a non-redundant self-supervised learning front end and a lightweight detection network back end; End-side adaptation optimization: the authentication reasoning model is exported into an open neural network exchange common format, disk space occupation is reduced through graph-level optimization and selective static quantization methods for only linear layers and attention layers, and the authentication reasoning model is adapted to a resource-limited end-side environment.
2. The end-side voice deepfake detection method based on non-redundant representation regulation according to claim 1, wherein, The contribution of the SSL model XLS-R to the speech authentication task is quantified layer by layer, comprising the following steps: Setting independent learnable scalar parameters for frame-level acoustic representations of each layer , normalized by a Softmax function to get layer weights , l Transformer layer for XLS-R l : Through time series normalization stabilization training, the frame-level acoustic representations of each layer are normalized in time series, and the aggregated representations are obtained by layer weight weighted summation, and the formula is: wherein, L is the number of Transformer layers of the SSL model XLS-R, is the number of the frame-level feature sequence output by the layer, T =200 is the number of time frames, D =1024 is the feature dimension, is the normalized layer weight, denotes layer time normalization; During the training process, The parameters are updated by backpropagation, and after the training converges, the layer weights Quantify the contribution of each layer to the speech discrimination task.
3. The end-side voice deepfake detection method based on non-redundant representation regulation according to claim 2, wherein, The dimension of the non-redundant frame-level acoustic representations is 1024; in the process of constructing the lightweight detection network, Two-layer feedforward neural networks are adopted to map the 1024-dimensional non-redundant frame-level acoustic representations into 128-dimensional low-dimensional task-specific frame-level embedding features; wherein each feedforward neural network is composed of a linear transformation, a ReLU activation function and a Dropout regularization layer.
4. The end-side voice deepfake detection method based on non-redundant representation regulation according to claim 3, wherein, The non-redundant frame-level acoustic representations are aggregated into embedding representations with a fixed dimension through an attention statistical pooling layer, comprising the following steps: The attention scores of each frame low-dimensional task-specific frame-level embedding feature are calculated, and the time weights are obtained by normalizing the attention scores; The weighted mean and weighted variance of the low-dimensional task-specific frame-level embedding features are calculated according to the time weights; The weighted mean and weighted variance are spliced to obtain the embedding representations with a fixed dimension.
5. The end-side voice deepfake detection method based on non-redundant representation regulation according to claim 4, wherein, The metric-aware classifier is a loss calculation formula: wherein, N is the number of sentences in the training batch, is the true label of the i th sentence, wherein, denotes real speech, denotes fake speech; is the normalized embedding of the i th sentence and the normalized real class center, denotes a non-target class, is a class-specific weight, is an additive angular margin, S is a scaling factor.
6. The end-side voice deepfake detection method based on non-redundant representation regulation according to claim 5, wherein, Before the non-redundant representation selection, the following steps are further included: RawBoost data enhancement is performed on the training data to simulate real transmission conditions such as pulse noise, additive noise and channel convolution distortion in the waveform domain to obtain original speech for non-redundant representation selection.
7. The end-side voice deepfake detection method based on non-redundant representation regulation according to claim 6, wherein, The graph-level optimization includes operator fusion and constant folding; wherein the operator fusion is used to combine consecutive operations into a single kernel call, and the constant folding is used to precompute static nodes.
8. The end-side voice deepfake detection method based on non-redundant representation regulation, according to claim 7, wherein, The selective static quantization of the linear layer and the attention layer comprises: adopting a static quantization method, selecting a calibration data set, and determining a quantization scale factor and a zero point through a percentile calibration strategy; and performing 8-bit integer INT8 quantization on only the linear layer and the attention layer operators in the anti-spoofing inference model, and mapping model weights from 32-bit floating-point numbers FP32 to INT8.
9. A non-redundant representation-based regulation end-side voice deep fake detection system applied to an electronic device, characterized in that, Comprise: A non-redundant representation selection unit configured to extract frame-level acoustic features of all layers of a pre-trained SSL model XLS-R with frozen original speech input, quantize contributions of the SSL model XLS-R to a speech anti-spoofing task in layers, reduce deep redundant parameters with low contributions through layer pruning according to the contributions, and reserve frame-level acoustic features with the highest contributions as non-redundant frame-level acoustic features, and freeze parameters of the reserved layers; A lightweight detection network building unit configured to map the non-redundant frame-level acoustic features to an embedding space of a speech anti-spoofing task through a feedforward neural network, aggregate the non-redundant frame-level acoustic features into fixed-dimension embedding features through an attention statistical pooling layer, input the embedding features into a metric-aware classifier based on a weighted additive angular separation loss, and classify real speech and fake speech on a hypersphere to obtain an anti-spoofing inference model with a non-redundant self-supervised learning front end and a lightweight detection network back end; An end-side adaptation optimization unit configured to export the anti-spoofing inference model into an open neural network exchange common format, reduce disk space occupation through graph-level optimization and selective static quantization of only linear layers and attention layers, and adapt to resource-constrained end-side environments.
10. An electronic device, comprising: The electronic device comprises a memory, a processor, and a non-redundant representation regulation based end-side speech deep spoofing detection program stored on the memory and executable on the processor, and the non-redundant representation regulation based end-side speech deep spoofing detection program, when executed by the processor, implements the non-redundant representation regulation based end-side speech deep spoofing detection method of any one of claims 1 to 8.