Variable voice authentic identification and traceability method and system based on multi-task learning

Abstract

The invention discloses a variable voice authentic identification and traceability method and system based on multi-task learning, and relates to the field of artificial intelligence and computer security. Obtaining an original voice sample; constructing a variable voice authentic identification and traceability model; inputting the original voice sample into a final variable voice authentic identification and traceability model to obtain a voice forgery probability and an original speaker; the variable voice authentic identification and traceability model integrates a feature extraction module, a shared encoder module, an authentic identification task branch module and a traceability task branch module; according to the invention, a joint learning framework capable of collaborative optimization and mutual promotion is constructed, multi-task learning is utilized to improve the authenticity identification accuracy and traceability of the voice-changing voice at the same time, and on the basis of high-precision discrimination of whether an original voice sample is forged or not, the original speaker voiceprint in the voice-changing voice is further reversely recovered, so that the voice-changing voice identification accuracy and traceability of the voice-changing voice are improved. And a high-performance defense technical scheme for variable voices is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer security technology, and more specifically to a method and system for voice distortion detection and source tracing based on multi-task learning. Background Technology

[0002] With the rapid development of deep learning and speech representation technologies, high-fidelity voice changing technology (also known as voice conversion or voice cloning) has become increasingly mature and readily available. This technology can generate a forged voice that is highly similar to the target speaker's timbre using only a very small number of voice samples as input. The lowered technological barrier has led to a sharp increase in the risk of voice-changing being misused, potentially triggering a series of serious social security problems such as identity impersonation fraud, the spread of false information, manipulation of public opinion, and even threats to national security.

[0003] Currently, research on defense technologies against voice-altered speech mainly focuses on voice spoofing detection. Its core objective is to determine whether a voice recording is genuine or fabricated using voice-altering technology, essentially a binary classification problem. However, defense methods that can only determine authenticity have significant limitations: they cannot reveal the identity of the original speaker behind the voice-altered audio. This allows malicious users (i.e., the original speaker who provides the original audio and performs the voice-altering) to conceal their true identity and evade accountability, greatly weakening the regulatory effectiveness and legal deterrent against technology abuse.

[0004] To achieve effective source tracing of altered speech (i.e., identifying the original speaker), current technologies face three main challenges:

[0005] First, there is the perturbation and overlay of the dominant voiceprint. The core of the voice-changing process is to map the original speaker's speech features onto the target speaker's acoustic space. During this process, the target speaker's voiceprint features (such as fundamental frequency profile, formant structure, spectral envelope, etc.) are forcibly imposed and dominate, severely overlaying and distorting the original speaker's original voiceprint features. Traditional voiceprint recognition systems are easily misled by this dominant "pseudo-target" voiceprint, resulting in mismatches.

[0006] Secondly, the original voiceprint information is extremely sparse and hidden. Advanced voice-changing models (especially those based on adversarial training or implicit representation) deliberately or inevitably erase or highly compress the residual voiceprint information of the original speaker in pursuit of high naturalness and high similarity. The original voiceprint features remaining in the changed speech are not only extremely weak, but also highly fragmented and nonlinearly embedded, making them difficult to capture effectively by conventional feature extraction methods.

[0007] Third, the high degree of coupling and decoupling of voiceprint features is difficult. In the generated altered speech, the target voiceprint features are deeply coupled and entangled with the residual original voiceprint features. Existing voiceprint recognition technologies lack the ability to effectively de-entangle and reverse recover these coupled features. They can usually only identify the most prominent target voiceprint (which is precisely the purpose of voice alteration), but cannot separate and reconstruct the pure voiceprint representation of the original speaker from the altered and mixed acoustic features.

[0008] Therefore, current defense systems have serious shortcomings in dealing with voice-altered speech threats, and there is an urgent need to develop a new source tracing technology capable of penetrating voice-altering interference, reversing the process, and identifying the original speaker's identity. Meanwhile, considering the inherent correlation between authentication and source tracing tasks (for example, effective original voiceprint features may also be important clues for authentication), separating the two would result in the loss of potential synergistic gains. The key to overcoming these bottlenecks lies in constructing a joint learning framework that can synergistically optimize and mutually promote each other; that is, utilizing multi-task learning to simultaneously improve the accuracy of voice-altered speech authentication and source tracing capabilities is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides a method and system for voice-changing speech authentication and source tracing based on multi-task learning. It constructs a joint learning framework that can be collaboratively optimized and mutually promoted. By utilizing multi-task learning, it can simultaneously improve the accuracy of voice-changing speech authentication and source tracing capabilities. Based on the high-precision identification of whether an original speech sample is forged, it can further reverse-engineer the original speaker's voiceprint in the voice-changing speech, thereby realizing a high-performance defense technology solution against voice-changing speech.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: a method for voice distortion detection and source tracing based on multi-task learning, comprising:

[0011] Obtain raw speech samples;

[0012] Construct a model for voice-changing speech authentication and source tracing;

[0013] The original speech sample is input into the final voice-changing speech authentication and source tracing model to obtain the speech forgery probability and the original speaker;

[0014] The voice-changing speech authentication and source tracing model integrates a feature extraction module, a shared encoder module, an authentication task branch module, and a source tracing task branch module.

[0015] The feature extraction module extracts multi-scale acoustic features from the original speech sample, and performs feature fusion and deep processing on the multi-scale acoustic features to convert them into preliminary features;

[0016] The shared encoder module uses multi-level convolutional-pooling layers to extract acoustically invariant features of the preliminary features, and uses bidirectional GRU to model long-term speech dependencies to obtain a temporal feature sequence.

[0017] Based on the temporal feature sequence, the anti-spoofing task branch module focuses on local abnormal features through attention statistical pooling, combines dual-path feature fusion to obtain dual-path fused features, performs binary classification on the obtained dual-path fused features to obtain the probability of voice-changing speech forgery.

[0018] Based on the temporal feature sequence, the source tracing task branch module extracts speaker identity features through context-aware modeling and multi-granularity feature pooling, combines decoupling regularization to separate voice-changing interference, and uses a sub-center angle interval classifier to identify the original speaker of the voice-changing speech.

[0019] An adaptive multi-task loss function is constructed, and a two-stage progressive joint optimization training is performed on the voice-changing speech authentication and source tracing model through a hierarchical optimization strategy to obtain the final voice-changing speech authentication and source tracing model.

[0020] Preferably, the multi-scale acoustic features of the original speech sample are extracted, including: Mel feature extraction and MFCC feature extraction;

[0021] Among them, the Mel filter bank is used to extract the Mel spectrogram, and the original speech samples are converted into a dimension [f Mel The spectral characteristics of [×T], where f Mel Here, T represents the Mel characteristic frequency, and T represents the number of time frames.

[0022] Calculate the MFCC and dynamic features of the original speech sample to obtain the dimension [f MFCC The spectral characteristics of [×T], where f MFCC Here, T represents the characteristic frequency of MFCC and the number of time frames.

[0023] Preferably, the multi-scale acoustic features are subjected to feature fusion and depth processing to convert them into preliminary features, including:

[0024] The Mel and MFCC features are concatenated along the frequency dimension, adding a channel dimension to form a tensor of dimension [c×f×T], where c is the number of channels and f = f Mel +f MFCC ;

[0025] Deep features are extracted through convolutional layers, batch normalization and ReLU activation functions are added, and max pooling is used to compress the feature dimensions, outputting a deep acoustic feature map with dimensions [c'×f'×T'], where c' is the number of channels after batch normalization, f'=f / k, T'=T / k, and k is the window size of max pooling.

[0026] Preferably, the anti-spoofing task branch module focuses on local abnormal features through attention statistical pooling, combines dual-path feature fusion to obtain dual-path fused features, and performs binary classification anti-spoofing on the obtained dual-path fused features to obtain the probability of voice-changing speech forgery, including:

[0027] The temporal feature sequences are weighted and summed to obtain a context feature vector of dimension [d×1], thereby focusing on local abnormal features in voice-changing speech; where d is the output dimension of the fully connected layer;

[0028] Calculate the average features in the time-series dimension to obtain global statistical features of dimension [d×1]. Concatenate the global statistical features and the local anomaly features to obtain dual-path fusion features.

[0029] The obtained dual-path fusion features are subjected to binary classification for fake detection using a fully connected layer, Dropout random deactivation function, ReLU activation function and Sigmoid activation function connected in sequence. The single-value fake probability is output. When the single-value fake probability exceeds the set threshold, the original speech sample is a fake voice-changing speech sample.

[0030] Preferably, the source tracing task branch module extracts speaker identity features through context-aware modeling and multi-granularity feature pooling, combines decoupling regularization to separate voice-changing interference, and uses a sub-center angle interval classifier to identify the original speaker of the voice-changing speech, including:

[0031] The temporal feature sequence is input into a 2-layer Transformer encoder, each layer including a feedforward network and an 8-head attention mechanism, which are used to simultaneously pay attention to information at different locations, capture the long-term dependencies of the speech sequence, and output dimensionality-invariant time series features.

[0032] Based on the time series features, attention statistical pooling is used to extract global macro features, and GhostVLAD pooling is used to extract local micro features; the global macro features and the local micro features are concatenated to obtain multi-granularity pooling fusion features;

[0033] The multi-granularity pooling fusion features are processed to obtain the speaker's original embedding vector. The original embedding vector is then decoupled and regularized to obtain a decoupled feature vector. Based on the original embedding vector and the decoupled feature vector, the speaker's identity features and voice-changing interference features are obtained.

[0034] A subcenter matching mechanism and an angle interval penalty mechanism are designed using a subcenter angle interval loss classifier to obtain the original speaker information of the voice-changing speech.

[0035] Preferably, the sub-center matching includes:

[0036] For each speaker, construct k subcenters and a learnable subcenter matrix SC of dimension [kN×192]. Initialize the weights using a Xavier uniform distribution, where N is the total number of speakers to be classified. Normalize the decoupled feature vector E' and the subcenter matrix SC, and calculate the cosine similarity between them. The recombined matrix has a dimension of [k×N]; for each speaker category, the subcenter with the highest similarity is taken as the speaker's score, resulting in a cosine matrix with a dimension of [1×N].

[0037] The angle interval penalty includes: introducing an interval penalty of radians m at the decision boundary. Where θ = arccos(cosine) is the feature angle, which forces the minimum angular interval between features of different classes. Setting the monotonicity threshold th = cos(π-m) and the linear compensation term mm = sin(π-m)·m, we get:

[0038]

[0039] The speaker's identity label is converted into a [1×N]-dimensional one-hot encoding, and a boundary-based Softmax is used to construct a [1×N]-dimensional logical output. The position of the maximum logical value is the original speaker number of the finally recognized voice-changing speech.

[0040] Preferably, the voice-changing speech authentication and source tracing model integrates a feature extraction module, a shared encoder module, an authentication task branch module, and a source tracing task branch module, including:

[0041] The concatenated feature extraction layer, shared encoder, anti-spoofing task header, and source tracing task header form an end-to-end architecture, as shown in the following expression:

[0042]

[0043] Where, ξ feat For feature extraction layer, ξ shared For shared encoders, T det To identify fake mission heads, T track For tracing the source of the task, This indicates that the output of the module to the right of the symbol is used as the input of the module to the left. This indicates the combination and construction of multi-task learning modules;

[0044] Heteroscedastic task uncertainty is introduced as a learnable loss weight parameter. By optimizing Θunc To achieve adaptive balance of loss scale, where, To detect the noise variance of the fake goods detection task, The noise variance for the source tracing task.

[0045] Preferably, constructing an adaptive multi-task loss function includes: constructing a fake detection task loss function and a source tracing task loss function respectively to obtain a joint loss function;

[0046] The loss function for the anti-spoofing task is constructed using focused binary cross-entropy, and its expression is as follows:

[0047]

[0048] In the formula, y i p represents the true label for speech sample X, where 0 indicates a real speech sample and 1 indicates a fake speech sample; i α represents the forgery probability ultimately output by the Sigmoid activation function in the forgery detection module; α and γ are hyperparameters for mitigating class imbalance; B is the batch size.

[0049] The source tracing task loss function is constructed based on the improved subcenter angle interval, and its expression is as follows:

[0050]

[0051] In the formula, ψ(θ,m)=cos(θ+m)-ω·sin m·θ, where ω is the compensation coefficient; To decouple the eigenvector E' i With speaker i sub-center SC i The included angle; s is the feature scaling factor, used to amplify the gradient change near the decision boundary; m is the radian angle interval;

[0052] Based on the loss function L of the counterfeit detection task det and the source tracing task loss function L track The joint loss function is constructed as follows:

[0053]

[0054] In the formula, λ is the loss balance factor. To detect the noise variance of the fake goods detection task, The noise variance for the source tracing task.

[0055] Preferably, the hierarchical optimization strategy includes: allocating learning rates for parameter groups, configuring optimizer hyperparameters, and performing dynamic parameter scheduling;

[0056] The voice-changing speech authentication and source tracing model is subjected to a two-stage progressive joint optimization training, including: first, pre-training the shared encoder module, and then performing global end-to-end fine-tuning.

[0057] Preferably, a voice-changing speech authentication and source tracing system based on multi-task learning includes:

[0058] The data acquisition module is used to acquire raw speech samples;

[0059] The model building module is used to build models for voice-changing speech authentication and source tracing.

[0060] The result output module is used to input the original speech sample into the final voice-changing speech authentication and source tracing model to obtain the speech forgery probability and the original speaker.

[0061] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method and system for voice distortion detection and source tracing based on multi-task learning, which has the following beneficial effects:

[0062] 1. Compared with traditional methods that treat counterfeit detection and source tracing as independent tasks, this invention designs a shared feature interaction mechanism. It uses the counterfeit detection task branch to focus on local anomalies and the source tracing task branch to focus on global features. Through joint training, it achieves dual-task collaborative enhancement and improves the multi-dimensional defense capability against voice-altered speech.

[0063] 2. Compared with existing voice-changing speech tracing methods, which struggle to capture and model highly fragmented and non-linearly embedded original voiceprint information, the tracing task branch module designed in this invention can alleviate the problem of extremely sparse and hidden original voiceprint information. It uses context-aware modeling and multi-granularity feature pooling to sensitively capture multi-dimensional abnormal features, and further uses decoupling regularization and sub-center matching mechanisms to classify identity tags, effectively improving the tracing accuracy of the original speaker.

[0064] 3. This invention adopts a lightweight architecture with a shared encoder module and a dual-task branch module in parallel. It designs an adaptive multi-task loss function and uses a two-stage progressive training strategy for joint optimization, which can achieve efficient deployment and strong robustness of the multi-task inference model, providing an efficient and reliable solution for voice distortion detection and source tracing. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0066] Figure 1 This is a schematic diagram of a method for voice distortion detection and source tracing based on multi-task learning provided by the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] This invention discloses a method for voice distortion detection and source tracing based on multi-task learning, such as... Figure 1 As shown, it includes:

[0069] Obtain raw speech samples;

[0070] Construct a model for voice-changing speech authentication and source tracing;

[0071] The original speech sample is input into the final voice-changing speech authentication and source tracing model to obtain the speech forgery probability and the original speaker;

[0072] The voice-changing speech authentication and source tracing model integrates a feature extraction module, a shared encoder module, an authentication task branch module, and a source tracing task branch module.

[0073] The feature extraction module extracts multi-scale acoustic features from the original speech sample, and performs feature fusion and deep processing on the multi-scale acoustic features to convert them into preliminary features;

[0074] The shared encoder module uses multi-level convolutional-pooling layers to extract acoustically invariant features of the preliminary features, and uses bidirectional GRU to model long-term speech dependencies to obtain a temporal feature sequence.

[0075] Based on the temporal feature sequence, the anti-spoofing task branch module focuses on local abnormal features through attention statistical pooling, combines dual-path feature fusion to obtain dual-path fused features, performs binary classification on the obtained dual-path fused features to obtain the probability of voice-changing speech forgery.

[0076] Based on the temporal feature sequence, the source tracing task branch module extracts speaker identity features through context-aware modeling and multi-granularity feature pooling, combines decoupling regularization to separate voice-changing interference, and uses a sub-center angle interval classifier to identify the original speaker of the voice-changing speech.

[0077] An adaptive multi-task loss function is constructed, and a two-stage progressive joint optimization training is performed on the voice-changing speech authentication and source tracing model through a hierarchical optimization strategy to obtain the final voice-changing speech authentication and source tracing model.

[0078] Specifically, the steps for constructing a voice-changing speech authentication and source tracing model include:

[0079] S100: Feature Extraction Layer Module Construction. This module extracts multi-scale acoustic features from the original speech signal, and after feature fusion and deep processing, converts them into a preliminary representation suitable for neural network processing.

[0080] S200: Shared encoder module construction. This module uses multi-level convolutional-pooling layers to extract acoustically invariant features and uses a bidirectional GRU to model long-term speech dependencies, thereby learning a general representation of speech and providing shared features for anti-spoofing and source tracing tasks.

[0081] S300: Construction of the anti-spoofing task branch module. This module focuses on local abnormal features through attention statistical pooling, enhances representation capabilities by combining dual-path feature fusion, and finally achieves binary classification anti-spoofing of voice-changing speech through abnormal pattern detection, providing the system with a basis for determining the probability of forgery.

[0082] S400: Source tracing task branch module construction. This module extracts speaker identity features through context-aware modeling and multi-granularity feature pooling, separates voice-changing interference by decoupling regularization, and finally identifies the original speaker of the voice-changing speech through a sub-center angle interval classifier.

[0083] S500: Multi-task learning and joint training. This module integrates all module components and constructs an adaptive multi-task loss function. Through a hierarchical optimization strategy, it achieves two-stage progressive joint optimization training.

[0084] Specifically, the method for constructing the feature extraction layer module described in S100 is as follows:

[0085] S101, Multi-scale acoustic feature extraction, including:

[0086] S1011, Mel Feature Extraction: Mel spectrograms are extracted using the Mel filter bank, transforming the original speech sample X into a [f] dimension. Mel The spectral characteristics of [×T], where f Mel Let denoted as Mel feature frequency and T be the number of time frames. In particular, logarithmic compression log(Mel+ε) is used to enhance the numerical stability of Mel features, where ε is a set small constant, which can be set to 1e-6.

[0087] Specifically, if the duration of all original speech samples X is fixed at 10 seconds and a time step of 10 milliseconds is used, then the number of time frames T = 1000.

[0088] S1012, MFCC Feature Extraction: Calculate the MFCC (Mel-frequency cepstral coefficients) and dynamic features of the original speech sample X, obtaining the dimension [f MFCC The spectral characteristics of [×T], where f MFCC Here, T represents the characteristic frequency of MFCC and the number of time frames.

[0089] Specifically, f can be set MFCC =128, T=1000.

[0090] S102, Feature fusion and deep processing, including:

[0091] S1021. Multi-scale feature concatenation: Concatenate the Mel features and MFCC features along the frequency dimension, adding a channel dimension to form a tensor of dimension [c×f×T], where c is the number of channels (c=1 before convolution), and f=f Mel +f MFCC Therefore, the dimension of this tensor is [1×256×1000].

[0092] S1022, Deep Convolutional Feature Enhancement: Deep features are further extracted through convolutional layers. Batch normalization and ReLU activation functions are added, and max pooling is used to compress the feature dimension, outputting a deep acoustic feature map of dimension [c'×f'×T'], where c' is the number of channels after batch normalization (specifically, c' = 32), f' = f / k, T' = T / k, and k is the max pooling window size (specifically, k = 2). Therefore, the output deep acoustic feature map has a dimension of [32×128×500].

[0093] Specifically, the method for constructing the shared encoder module described in S200 is as follows:

[0094] S201, Convolutional-pooling layer connections, including:

[0095] Two convolutional layers are used to progressively compress the frequency dimension and extract acoustically invariant features. The first convolutional layer has a batch normalized channel count of c”1 (specifically, c”1 can be set to 64), uses the GELU activation function and max pooling, and the max pooling window size is k’ (specifically, k’ can be set to 2). The second convolutional layer has a batch normalized channel count of c”2 (specifically, c”2 can be set to 128), uses the GELU activation function and adaptive average pooling. In particular, the adaptive average pooling compresses the frequency dimension to 1, resulting in high-level acoustic features of dimension [c”2×T”], where T” = T’ / k’. Therefore, the output high-level acoustic feature dimension is [128×250].

[0096] S202, Time-series correlation modeling, including:

[0097] Long-term speech dependencies are modeled using a bidirectional GRU (Gated Recurrent Unit), and the acoustic features are converted into a time series format. The hidden layer size is h (specifically, h = 256 can be set). After concatenating the bidirectional hidden states, the output is a time series feature sequence with a dimension of [2h×T”], which serves as a shared feature for subsequent anti-spoofing and source tracing tasks. Therefore, the dimension of the output shared feature is [512×250].

[0098] Specifically, the method for constructing the counterfeit detection task branch module described in S300 is as follows:

[0099] S301, Attention Statistical Pooling, including:

[0100] The time step weights are calculated by using a fully connected layer, Tanh activation function, and Softmax. The temporal feature sequence obtained from S202 is then weighted and summed to obtain a context feature vector of dimension [d×1], thereby focusing on local abnormal segments in the voice-changing speech. Here, d is the output dimension of the fully connected layer (specifically, d = 128 can be set).

[0101] S302, Dual-path feature fusion, including:

[0102] The average feature in the time dimension is calculated to obtain the global statistical feature with dimension [d×1]. The local anomaly feature obtained by S301 is concatenated with the global statistical feature to obtain the dual-path fusion feature with dimension [2d×1] (i.e. [256×1]).

[0103] S303, Abnormal mode detection, including:

[0104] By using a fully connected layer, Dropout random deactivation, ReLU activation function, and Sigmoid activation function, the dual-path fusion features obtained from S302 are used for fake binary classification. Dropout random deactivation is used to prevent overfitting. The output of the Sigmoid activation function is a single-valued fake probability p (0~1). The closer it is to 1, the higher the probability of fake. If it exceeds the set threshold, the original speech sample X is considered to be a fake voice-changing speech sample.

[0105] Specifically, the method for constructing the source tracing task branch module described in S400 is as follows:

[0106] S401, Context-Aware Modeling, including:

[0107] The temporal feature sequence obtained from S202 is input into a 2-layer Transformer encoder. Each layer contains a feedforward network and an 8-head attention mechanism, which allows the model to pay attention to information at different positions at the same time, thereby capturing the long-term dependencies of the speech sequence and enhancing the understanding of the temporal structure of the speech. The output dimension remains unchanged at [2h×T”] (i.e. [512×250]).

[0108] S402, Multi-granularity feature pooling, including using attention statistical pooling to extract global macro features and using GhostVLAD pooling to extract local micro features:

[0109] S4021, Attention Statistical Pooling: The time step weights are calculated through the Attention layer. The calculated time step weights and the temporal feature sequence output by S401 are used to calculate the weighted mean and standard deviation. This is used to capture and focus on the local spectral abnormalities caused by voice change and suppress irrelevant time segments. The weighted mean and standard deviation are concatenated as the output, and the output dimension is increased to [4h×T”] (i.e. [1024×250]).

[0110] S4022, GhostVLAD Pooling: First, use convolution + Softmax to calculate soft-assignment weights and soft-assign features to 8 cluster centers; then, calculate the residuals between the features and cluster centers to capture local patterns; next, multiply the residuals with the soft-assignment weights and perform weighted aggregation; finally, perform double normalization on the results, including independent normalization for each cluster center and overall normalization after reshape flattening, to ensure that the features are uniformly distributed on the hypersphere, and the output dimension is increased to [8h×T”] (i.e. [2048×250]).

[0111] S4023, Multi-granularity pooling fusion: The attention statistical pooling feature output obtained in S4021 and the GhostVLAD pooling feature output obtained in S4022 are concatenated to obtain a multi-granularity pooling fusion feature with a dimension of [12h×T”] (i.e. [3072×250]).

[0112] S403, the construction of the multi-speaker classifier, is the core part of the source tracing task branch module, including:

[0113] S4031, Speaker Embedding Extraction: The multi-granularity pooling fusion features obtained in S4023 are reduced to [4h×T”] (i.e. [1024×250]) through a fully connected layer, and then batch normalization, adaptive negative slope PReLU activation function and fully connected layer are passed in sequence to output a 192-dimensional speaker embedding vector E.

[0114] S4032, Decoupling Regularization: Through a linear projection layer, the speaker embedding vector E obtained in S4031 is mapped to a decoupled feature vector E' with the same 192 dimensions; through the mean squared error loss function, the distance between the original embedding vector E and the decoupled feature vector E' is constrained, forcing the model to separate speaker identity features from voice-changing interference features.

[0115] S4033, Multi-speaker Classification: Employs a subcenter angular margin loss classifier, specifically including:

[0116] S40331, Subcenter Matching: Construct k subcenters for each speaker (specifically, k = 3), i.e., construct a learnable subcenter matrix SC with dimension [kN×192], and initialize the weights using a Xavier uniform distribution, where N is the total number of speakers to be classified (specifically, the total number of suspected target speakers can be set to N = 100); perform L2 normalization on the decoupled feature vector E' and the subcenter matrix SC, and calculate the cosine similarity between them. The recombined matrix has a dimension of [k×N] (i.e., [3×100]); for each speaker category, the subcenter with the highest similarity is taken as the speaker's score, resulting in a cosine matrix with a dimension of [1×N] (i.e., [1×100]).

[0117] S40332, Angle Interval Penalty: Introduce an interval penalty of radians m (specifically, m = 0.2) at the decision boundary. Where θ = arccos(cosine) is the feature angle, which forces the minimum angular interval between features of different classes. A monotonicity threshold of th = cos(π-m) is set to ensure that changes in the angle interval do not lead to a non-monotonic loss function; a linear compensation term mm = sin(π-m)·m is set to avoid the gradient vanishing problem when θ approaches π; finally, the following is obtained:

[0118]

[0119] S40333, Logical Output Construction: Convert the speaker's identity label to be classified into a one-hot encoding of dimension [1×N] (i.e., [1×100]) (denoted as Onehot). Use boundary-based Softmax to construct a logical output of dimension [1×N] (i.e., [1×100]). The position of the maximum logical value corresponds to the original speaker number of the voice-changing speech finally identified by the source tracing task branch module.

[0120] Specifically, the multi-task learning joint training method described in S500 is as follows:

[0121] S501, Model Integration and Parametricization, including:

[0122] S5011, Cascaded Model Topology Construction: The feature extraction layer (S100), shared encoder (S200), fake detection task head (S300), and source tracing task head (S400) are connected in series to form an end-to-end architecture.

[0123]

[0124] Where, ξ feat For feature extraction layer, ξ shared For shared encoders, T det To identify fake mission heads, T track For tracing the source of the task, This indicates that the output of the module to the right of the symbol is used as the input of the module to the left. This indicates the combination and construction of multi-task learning modules.

[0125] S5012, Construction of Learnable Uncertainty Parameters: Introducing heteroscedastic task uncertainty as a learnable loss weight parameter. By optimizing Θ unc It can achieve adaptive balance of loss scale, whereby, To detect the noise variance of the fake goods detection task, The noise variance for the source tracing task.

[0126] S502, Construction of Adaptive Multi-Task Loss Function, including:

[0127] S5021. Construction of the loss function for the counterfeit detection task: The loss function for the counterfeit detection task is designed using focused binary cross-entropy as follows:

[0128]

[0129] Among them, y i p represents the true label for speech sample X, where 0 indicates a real speech sample and 1 indicates a fake speech sample; i α represents the forgery probability output by the Sigmoid activation function in the forgery detection module; α and γ are hyperparameters for mitigating class imbalance; and B is the batch size.

[0130] Specifically, α = 0.8, γ = 2, and B = 32 can be set.

[0131] S5022. Construction of the source tracing task loss function: The source tracing task loss function is designed based on the improved sub-center angle interval as follows:

[0132]

[0133] Among them, ψ(θ,m)=cos(θ+m)-ω·sin m·θ, ω is the compensation coefficient; To decouple the eigenvector E' i With speaker i sub-center SC i The angle between the two sides; s is the feature scaling factor, used to amplify the gradient change near the decision boundary; m is the radian angle interval.

[0134] Specifically, ω = 0.1, s = 30, and m = 0.2 can be set.

[0135] S5023, Joint Loss Function Construction: Based on the two task branch sub-loss functions L designed in S5021 and S5022, det and L track The joint loss function is designed as follows:

[0136]

[0137] Where λ is the loss balance factor.

[0138] Specifically, λ can be set to 0.5.

[0139] S503, Layered optimization strategy settings, including:

[0140] S5031, Parameter group learning rate allocation: For feature extraction layer parameter Φ feat A low learning rate, specifically 1e-4), is used to fine-tune the underlying acoustic feature extractor, protecting pre-trained knowledge from catastrophic forgetting; for shared encoder parameters Φ shared A moderate learning rate (specifically, 5e-4) is used to balance general feature representation with task adaptability, thus optimizing the temporal modeling capability of GRU; for the two task branches of counterfeit detection and source tracing, the parameter Φ det and Φ track A high learning rate (specifically, 1e-3) is used to accelerate the convergence of task-specific components (attention pooling, classifier) ​​and enhance the sharpness of the decision boundary; for the loss weight parameter Θ unc It employs the highest learning rate (specifically, it can be set to 1e-2) to quickly converge the uncertainty parameter and dominate the multi-task loss balance.

[0141] S5032, Optimizer Hyperparameter Configuration: The AdamW optimizer is used. The first-order moment decay rate β1 can be set to 0.9 to control the inertia of the gradient moving average and reduce the impact of stochastic gradient noise. The second-order moment decay rate β2 can be set to 0.999 to adaptively adjust the learning step size of each parameter and improve the ill-conditioned curvature problem. Adaptive weight decay intensity is used.

[0142]

[0143] Negatively correlated with the learning rate, it enables high-frequency updates of the two task branch modules to apply weaker regularization, where η Φ This represents the relative learning rate for the corresponding module parameters.

[0144] S5033, Parameter Dynamic Scheduling: Set the linear preheating step number t warm , t warm =500, using an initial 500 steps of linearly increasing learning rate to stabilize the gradient direction in the early training phase; set the number of epochs t for validating the loss stagnation. stag , t stag =5, when the verification loss does not decrease for 5 consecutive rounds, the decay is triggered to escape the local optimum; in addition, gradient pruning is used to constrain the upper bound of the gradient norm to prevent numerical explosion in the steep region of the loss surface.

[0145] S504, a two-stage progressive training program, includes:

[0146] S5041, Shared encoder pre-training: Freezing the parameters Φ of the feature extraction layer feat (Disable gradient calculation for feature extraction layer parameters) to protect pre-trained acoustic knowledge (Mel / MFCC feature extraction capabilities), while preventing low-level features from being dominated and corrupted by task branch modules, and reducing optimization complexity in the early stages of training. The optimization objective is...

[0147]

[0148] S5042, Global End-to-End Fine-Tuning: Employs a gradual unfreezing fine-tuning strategy, initially unfreezing the parameters Φ of the two task branches: authentication and source tracing. det Φ track and Θ unc Continuing the optimized trajectory of stage 1 described in S5041; after 20% of the total rounds, further unfreeze the parameters Φ of the shared encoder module. shared Enhance the adaptability of temporal modeling; after 50% of the total rounds, further unfreeze the parameter Φ of the feature extraction layer module. feat This enhances the ability to extract underlying acoustic features. At this point, all modules have been unfrozen, enabling global end-to-end fine-tuning.

[0149] This invention employs several methods: First, it constructs a feature extraction layer module to extract multi-scale acoustic features from the original speech, forming a preliminary fusion representation. Second, it constructs a shared encoder module to extract acoustically invariant features using multi-level convolution-pooling and models long-term dependencies through bidirectional GRU, providing shared basic features for downstream tasks. Third, it constructs a fake detection task branch module, using attention-based statistical pooling to focus on local anomaly features and combining dual-path fusion to enhance the representation, ultimately performing binary classification for fake detection. Fourth, it constructs a source tracing task branch module, extracting speaker features through context-aware modeling and multi-granularity pooling, separating voice-changing interference using decoupling regularization, and finally using a sub-center angle interval classifier to identify the original speaker. Fifth, it performs multi-task learning joint training, integrating all modules, designing an adaptive multi-task loss function, and employing a hierarchical optimization strategy for two-stage progressive joint training. This invention aims to construct a collaboratively optimized joint framework that simultaneously improves the accuracy of voice-changing speech fake detection and speaker source tracing capabilities through multi-task learning, deterring the abuse of voice-changing speech forgery technology.

[0150] Because the anti-spoofing and source tracing tasks are highly correlated in feature learning, the anti-spoofing task focuses on detecting abnormal patterns in speech (such as artifacts introduced by voice changing), while the source tracing task needs to recover the original speaker's identity features from the changed speech. These two tasks share underlying acoustic feature representations (through a shared encoder), enabling the model to simultaneously learn more discriminative and robust feature representations. The anomaly detection capabilities provided by the anti-spoofing task help the source tracing task focus more on extracting the original speaker's features; conversely, the sensitivity to identity details in the source tracing task helps the anti-spoofing task more accurately distinguish between genuine and fake speech. Therefore, this embodiment of the invention employs a multi-task collaborative learning mechanism, where parallel processing during training improves the performance of both tasks due to better feature learning.

[0151] Specifically, in model structure design and training, learning both tasks simultaneously can improve model performance. The fake detection task head and the source tracing task head achieve implicit feature interaction and collaborative optimization through a shared encoder module, which is precisely the advantage of a multi-task learning framework. After training, when deployed, fake detection is performed first, followed by source tracing. That is, the fake detection task branch is used to determine whether the speech is fake, and the source tracing task further identifies the original speaker based on this.

[0152] Therefore, during the training phase, the two tasks are performed simultaneously, jointly optimized using a multi-task loss function. During the inference phase, if only falsification is required, only the falsification branch can be used; if further source tracing is needed, both branches can be used simultaneously or sequentially. This design satisfies the logical sequence of tasks while ensuring the model's efficiency and flexibility. This embodiment of the invention achieves synergistic enhancement of falsification and source tracing tasks through a multi-task learning framework, achieving deep integration and mutual reinforcement at the feature level.

[0153] In one specific embodiment of the present invention, a voice-changing speech authentication and source tracing system based on multi-task learning includes:

[0154] The data acquisition module is used to acquire raw speech samples;

[0155] The model building module is used to build models for voice-changing speech authentication and source tracing.

[0156] The result output module is used to input the original speech sample into the final voice-changing speech authentication and source tracing model to obtain the speech forgery probability and the original speaker.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0158] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for voice distortion detection and source tracing based on multi-task learning, characterized in that, include: Obtain raw speech samples; Construct a model for voice-changing speech authentication and source tracing; The original speech sample is input into the final voice-changing speech authentication and source tracing model to obtain the speech forgery probability and the original speaker; The voice-changing speech authentication and source tracing model integrates a feature extraction module, a shared encoder module, an authentication task branch module, and a source tracing task branch module. The feature extraction module extracts multi-scale acoustic features from the original speech sample, and performs feature fusion and deep processing on the multi-scale acoustic features to convert them into preliminary features; The shared encoder module uses multi-level convolutional-pooling layers to extract acoustically invariant features of the preliminary features, and uses bidirectional GRU to model long-term speech dependencies to obtain a temporal feature sequence. Based on the temporal feature sequence, the anti-spoofing task branch module focuses on local abnormal features through attention statistical pooling, combines dual-path feature fusion to obtain dual-path fused features, performs binary classification on the obtained dual-path fused features to obtain the probability of voice-changing speech forgery. Based on the temporal feature sequence, the source tracing task branch module extracts speaker identity features through context-aware modeling and multi-granularity feature pooling, combines decoupling regularization to separate voice-changing interference, and uses a sub-center angle interval classifier to identify the original speaker of the voice-changing speech. An adaptive multi-task loss function is constructed, and a two-stage progressive joint optimization training is performed on the voice-changing speech authentication and source tracing model through a hierarchical optimization strategy to obtain the final voice-changing speech authentication and source tracing model.

2. The method for voice distortion detection and source tracing based on multi-task learning according to claim 1, characterized in that, Extracting multi-scale acoustic features from the original speech samples, including Mel feature extraction and MFCC feature extraction; Among them, the Mel filter bank is used to extract the Mel spectrogram, and the original speech samples are converted into a dimension [f Mel The spectral characteristics of [×T], where f Mel Here, T represents the Mel characteristic frequency, and T represents the number of time frames. Calculate the MFCC and dynamic features of the original speech sample to obtain the dimension [f MFCC The spectral characteristics of [×T], where f MFCC These are the characteristic frequencies of MFCC.

3. The method for voice distortion detection and source tracing based on multi-task learning according to claim 2, characterized in that, The multi-scale acoustic features are fused and deep processed to convert them into preliminary features, including: The Mel and MFCC features are concatenated along the frequency dimension, adding a channel dimension to form a tensor of dimension [c×f×T], where c is the number of channels and f = f Mel +f MFCC ; Deep features are extracted through convolutional layers, batch normalization and ReLU activation functions are added, and max pooling is used to compress the feature dimensions, outputting a deep acoustic feature map with dimensions [c'×f'×T'], where c' is the number of channels after batch normalization, f'=f / k, T'=T / k, and k is the window size of max pooling.

4. The method for voice distortion detection and source tracing based on multi-task learning according to claim 1, characterized in that, The anti-spoofing task branch module focuses on local abnormal features through attention statistical pooling, combines dual-path feature fusion to obtain dual-path fused features, and performs binary classification anti-spoofing on the obtained dual-path fused features to obtain the probability of voice-changing speech forgery, including: The temporal feature sequences are weighted and summed to obtain a context feature vector of dimension [d×1], thereby focusing on local abnormal features in the voice-changing speech; where d is the output dimension of the fully connected layer; Calculate the average features in the time-series dimension to obtain global statistical features of dimension [d×1]. Concatenate the global statistical features and the local anomaly features to obtain dual-path fusion features. The obtained dual-path fusion features are subjected to binary classification for fake detection using a fully connected layer, Dropout random deactivation function, ReLU activation function and Sigmoid activation function connected in sequence. The single-value fake probability is output. When the single-value fake probability exceeds the set threshold, the original speech sample is a fake voice-changing speech sample.

5. The method for voice distortion detection and source tracing based on multi-task learning according to claim 1, characterized in that, The source tracing task branch module extracts speaker identity features through context-aware modeling and multi-granularity feature pooling, combines decoupling regularization to separate voice-changing interference, and uses a sub-center angle interval classifier to identify the original speaker of the voice-changing speech, including: The temporal feature sequence is input into a 2-layer Transformer encoder, each layer including a feedforward network and an 8-head attention mechanism, which are used to simultaneously pay attention to information at different locations, capture the long-term dependencies of the speech sequence, and output dimensionality-invariant time series features. Based on the time series features, attention statistical pooling is used to extract global macro features, and GhostVLAD pooling is used to extract local micro features; the global macro features and the local micro features are concatenated to obtain multi-granularity pooling fusion features; The multi-granularity pooling fusion features are processed to obtain the speaker's original embedding vector. The original embedding vector is then decoupled and regularized to obtain a decoupled feature vector. Based on the original embedding vector and the decoupled feature vector, the speaker's identity features and voice-changing interference features are obtained. A subcenter matching mechanism and an angle interval penalty mechanism are designed using a subcenter angle interval loss classifier to obtain the original speaker information of the voice-changing speech.

6. The method for voice distortion detection and source tracing based on multi-task learning according to claim 5, characterized in that, The subcenter matching includes: For each speaker, construct k subcenters and a learnable subcenter matrix SC of dimension [k×N×192]. Initialize the weights using a Xavier uniform distribution, where N is the total number of speakers to be classified. Normalize the decoupled feature vector E' and the subcenter matrix SC, and calculate the cosine similarity between them. The recombined matrix has a dimension of [k×N]; for each speaker category, the subcenter with the highest similarity is taken as the speaker's score, resulting in a cosine matrix with a dimension of [1×N]. The angle interval penalty includes: introducing an interval penalty of radians m at the decision boundary. Where θ = arccos(cosine) is the feature angle, which forces the minimum angular interval between features of different classes. Setting the monotonicity threshold th = cos(π-m) and the linear compensation term mm = sin(π-m)·m, we get: The speaker's identity label is converted into a [1×N]-dimensional one-hot encoding, and a boundary-based Softmax is used to construct a [1×N]-dimensional logical output. The position of the maximum logical value is the original speaker number of the finally recognized voice-changing speech.

7. The method for voice distortion detection and source tracing based on multi-task learning according to claim 1, characterized in that, The voice-changing speech authentication and source tracing model integrates a feature extraction module, a shared encoder module, an authentication task branch module, and a source tracing task branch module, including: The concatenated feature extraction layer, shared encoder, anti-spoofing task header, and source tracing task header form an end-to-end architecture, as shown in the following expression: Where, ξ feat For feature extraction layer, ξ shared For shared encoders, T det To identify fake mission heads, T track For tracing the source of the task, This indicates that the output of the module to the right of the symbol is used as the input of the module to the left. This indicates the combination and construction of multi-task learning modules; Heteroscedastic task uncertainty is introduced as a learnable loss weight parameter. By optimizing Θ unc To achieve adaptive balance of loss scale, where, To detect the noise variance of the fake goods detection task, The noise variance for the source tracing task.

8. The method for voice distortion detection and source tracing based on multi-task learning according to claim 1, characterized in that, Constructing an adaptive multi-task loss function includes: constructing a loss function for the fake detection task and a loss function for the source tracing task respectively, and obtaining a joint loss function; The loss function for the anti-spoofing task is constructed using focused binary cross-entropy, and its expression is as follows: In the formula, y i p represents the true label for speech sample X, where 0 indicates a real speech sample and 1 indicates a fake speech sample; i α represents the forgery probability ultimately output by the Sigmoid activation function in the forgery detection module; α and γ are hyperparameters for mitigating class imbalance; B is the batch size. The source tracing task loss function is constructed based on the improved subcenter angle interval, and its expression is as follows: In the formula, ψ(θ,m)=cos(θ+m)-ω·sinm·θ, where ω is the compensation coefficient; To decouple the eigenvector E' i With speaker i sub-center SC i The included angle; s is the feature scaling factor, used to amplify the gradient change near the decision boundary; m is the radian angle interval; Based on the loss function L of the counterfeit detection task det and the source tracing task loss function L track The joint loss function is constructed as follows: In the formula, λ is the loss balance factor. To detect the noise variance of the fake goods detection task, The noise variance for the source tracing task.

9. The method for voice distortion detection and source tracing based on multi-task learning according to claim 1, characterized in that, The hierarchical optimization strategy includes: allocating learning rates for parameter groups, configuring optimizer hyperparameters, and performing dynamic parameter scheduling. The voice-changing speech authentication and source tracing model is subjected to a two-stage progressive joint optimization training, including: first, pre-training the shared encoder module, and then performing global end-to-end fine-tuning.

10. A voice-changing speech authentication and source tracing system based on multi-task learning, employing the voice-changing speech authentication and source tracing method based on multi-task learning as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire raw speech samples; The model building module is used to build models for voice-changing speech authentication and source tracing. The result output module is used to input the original speech sample into the final voice-changing speech authentication and source tracing model to obtain the speech forgery probability and the original speaker.

Images