Non-learnable speech data generation method against model fine-tuning attacks

Abstract

The application discloses a kind of unlearnable speech data generation methods for model fine-tuning attack, comprising: based on pre-training speech representation network and differentiable classification head, proxy speaker recognition model is constructed;Initialize perturbation variable and calculate timing significance mask;Through candidate buffer pool and cosine similarity rule, determine target identity label for source sample;With inner layer optimization simulation attacker, use protected speech and real speaker label to fine-tune model;In outer layer optimization, the model parameter obtained in inner layer optimization is used as temporary model, and the outer layer loss is calculated for target identity label, and the gradient of perturbation is solved according to first-order approximation strategy;Update perturbation and execute projection and spectral difference constraint;Protected speech is output by repeating iteration.The application method can improve the effectiveness of the target identity attack, enhance the identity confusion ability, reduce the optimization waste caused by invalid target, and improve the stability and practicality of the voice identity protection process.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing, machine learning and data security technology, and in particular to a method for generating unlearnable speech data to counter model fine-tuning attacks. Background Technology

[0002] In recent years, with the widespread adoption of social media, short video platforms, and various voice interaction systems, massive amounts of personal voice data containing clear identifiers have been publicly released on the internet, posing a serious privacy and security risk of unauthorized large-scale scraping. Simultaneously, self-supervised pre-trained speech representation models and deep speaker recognition systems have made breakthrough progress. Current pre-trained networks have learned highly generalizable underlying acoustic representations and strong identity discrimination priors from massive amounts of unlabeled data. The evolution of techniques for learning strong representations solely from speech audio has directly given rise to a new, low-cost, and highly threatening few-sample fine-tuning attack scenario. By fine-tuning on small datasets, attackers only need to utilize a very small amount of publicly available target victim speech data, fine-tuning an open-source pre-trained model to quickly obtain a high-precision speaker recognition model or a high-fidelity speech clone model for that victim at extremely low computational cost.

[0003] However, existing voice privacy protection schemes are ineffective against such highly targeted fine-tuning attacks. Traditional unlearnable sample generation methods typically assume that the attacker's model parameters are frozen or randomly initialized, failing to align with the attacker's true two-layer evolutionary path of "pre-trained representation extraction plus dynamic fine-tuning update." Faced with the powerful feature adaptation capabilities of pre-trained models and the momentum mechanism of fine-tuning optimizers, the static perturbations applied by traditional defenses often only cause initial training oscillations. With multiple rounds of gradient descent, these shallow perturbations are easily canceled out, making it difficult to continuously and effectively suppress the malicious model's learning of the real speaker's identity features. Furthermore, most existing methods only cause samples to deviate from their original feature cluster centers, lacking explicit guidance for misidentification. This leads to the fine-tuned model still easily re-associating deep features with the real speaker in the metric learning space.

[0004] Currently, the speech domain severely lacks a systematic proactive defense solution that can effectively resist two-layer optimization attacks, ensure the original speech quality, and be efficiently solved and implemented in engineering. Summary of the Invention

[0005] To address the problem that existing methods for voice identity protection or protected voice generation involving target identity designation often suffer from insufficient attack targeting due to the random, targeted, or manually preset nature of target identities, this invention proposes a non-learnable voice data generation method to counter model fine-tuning attacks. By simulating the fine-tuning process of an attacker on a pre-trained speaker recognition model, and driving perturbed voice to shift towards non-real identity target categories on the fine-tuned model, a publicly releasable non-learnable voice is generated. This method improves the effectiveness against targeted identity attacks, enhances identity obfuscation capabilities, reduces optimization waste caused by invalid targets, and thus improves the stability and practicality of the voice identity protection process.

[0006] This invention adopts the following technical solution: a method for generating unlearnable speech data against model fine-tuning attacks, comprising the following steps:

[0007] Step 1: Obtain raw speech samples and its real speaker tags Based on a pre-trained speech representation network and a differentiable classification head, a proxy speaker recognition model is constructed. ;

[0008] Step 2: Initialize the perturbation variables And calculate the temporal saliency mask. ;

[0009] Step 3: Determine the target identity label for the source sample using the candidate buffer pool and cosine similarity rule. The proxy speaker recognition model Perform two-layer optimization;

[0010] Step 4: Simulate an attacker using inner-layer optimization, employing protected speech. and real speaker tags Fine-tuning the pre-trained model yields the model parameters. ;

[0011] Step 5: In outer layer optimization, with For temporary models, for target identity labels Calculate the outer layer loss and apply the perturbation variable according to the first-order approximation strategy. Solve for the gradient;

[0012] Step 6: Update the perturbation variables according to the gradient update rule. and execute Projection and spectral difference constraints;

[0013] Step 7: Repeat steps 3 to 6 until the set number of rounds is reached or the convergence condition is met, then output the protected voice. .

[0014] As a preferred embodiment, in step 1, the proxy speaker recognition model For fine-tuning attack and defense tasks, it is constructed using a pre-trained speech representation network. Classification of speakers The combination forms a speaker classification head, which can be an AAM-Softmax classification head, a fully connected classification head, or an ArcFace head.

[0015] As a preferred embodiment, in step 2, the temporal saliency mask The gradient magnitude of the input speech is calculated based on the classification loss to obtain the temporal saliency score, and then the score is normalized to generate the result.

[0016] As a preferred option, in step 3, a non-real target identity label is determined for each source sample. The specific steps are as follows:

[0017] Step 3.1: Use a pre-trained speech representation network to extract the normalized speaker embeddings of the current batch of samples. , For source samples Normalized speaker embedding;

[0018] Step 3.2: Embed the speakers of the current batch and its labels Write to candidate buffer pool , Candidate samples The normalized speaker embedding; the candidate buffer pool adopts a first-in-first-out structure with a maximum capacity of ;

[0019] Step 3.3, for the source sample Exclude all candidates that meet the criteria from the candidate buffer pool. For candidates, only those not belonging to this category will be retained. ;

[0020] Step 3.4: Select the non-native identity with the highest cosine similarity to the source sample as the target identity, thus obtaining the source sample. Corresponding target identity tags ;

[0021] Step 3.5: When there are no available non-class samples in the candidate buffer pool, a backoff strategy is adopted to randomly select a preset label from all non-real identity labels, or to select one according to the cyclic offset rule.

[0022] As a preferred embodiment, in step 4, the inner layer optimization simulates an attacker exploiting protected voice and real speaker tags. The proxy speaker recognition model Perform one or more fine-tuning steps; use cross-entropy speaker classification loss for the inner layer loss, and update the inner layer accordingly to obtain the optimized model parameters. .

[0023] As a preferred embodiment, in step 5, the outer layer optimization is performed to... As a temporary model, the same protected speech is directed toward a non-real target identity label. The offset is achieved by weighting the target identity classification loss with the spectral difference constraint.

[0024] The spectral difference constraint adopts the L1 norm difference, L2 norm difference, or Mel spectral difference after short-time Fourier transform.

[0025] The first-order approximation strategy, in the inner update phase, will protect the speech... Considered as the input to the stopping gradient, this completes the transformation of the model parameters from... arrive The update; during the outer layer optimization phase, Considered relative to the disturbance variable The constant is used to calculate only the gradient from the perturbation variable to the outer loss.

[0026] As a preferred option, in step 6, the perturbation variable is updated using the signed gradient descent method. After each update implement Projection ensures that the disturbance amplitude does not exceed a preset threshold. It also performs spectral difference constraints to constrain the short-term spectral differences between the protected speech and the original speech, ensuring the publishability of the protected speech.

[0027] As a preferred option, step 2 also includes multi-subject perturbation processing to alleviate the averaging problem caused by similar samples sharing a single perturbation;

[0028] Each speaker category maintains multiple sub-perturbation templates. ,in , This indicates the number of sub-perturbation templates maintained for each speaker category; when a sample to be protected enters the system, the sample-level index is obtained from the sample path, file name, or unique number through a deterministic mapping function.

[0029] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0030] 1. This invention generates perturbations by explicitly simulating the fine-tuning process of an attacker on a pre-trained speaker recognition model, making the perturbation design path consistent with the actual attack path. Therefore, it is more suitable for pre-training fine-tuning scenarios than methods that only target unlearnable samples trained from scratch.

[0031] 2. This invention assigns a clear erroneous identity offset direction to each source sample through a target identity selection mechanism, transforming the protection target from general "disruptive training" to "inducing erroneous identity mapping", thereby increasing the probability of attackers establishing erroneous identity associations after fine-tuning.

[0032] 3. This invention uses a first-order approximation method to solve the perturbation gradient in bi-layer optimization. Without explicitly calculating the second-order gradient, it realizes a practical engineering solution process, balancing the effectiveness of the method with the complexity of training.

[0033] 4. This invention is achieved through... Amplitude constraints, STFT spectral difference constraints, and optional temporal saliency masks enable active protection of protected speech while preserving its audibility and publishability as much as possible.

[0034] 5. This invention, by employing a pre-trained speech representation network and an AAM-Softmax classification head structure, can be directly adapted to existing mainstream speaker recognition systems and has strong practical feasibility. Attached Figure Description

[0035] Figure 1 This is a schematic diagram illustrating the overall application scenario of the method of the present invention;

[0036] Figure 2 This is a flowchart of the non-learnable speech data generation method of the present invention for model fine-tuning attacks;

[0037] Figure 3 This is a schematic diagram of the target identity selection mechanism of the present invention;

[0038] Figure 4 This is a schematic diagram of the two-layer optimization solution of the present invention;

[0039] Figure 5 This is a schematic diagram illustrating the generation of the temporal saliency mask in this invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0041] The overall application scenarios of this invention's non-learnable speech data generation method against model fine-tuning attacks are as follows: Figure 1 As shown, the overall application scenario revolves around the "voice privacy ecosystem," which includes two parallel processes: an unprotected attack path and the defense path of this invention. Specifically, it includes the overall relationship of: public voice release, third-party collection, attackers using public voice to fine-tune the pre-trained speaker recognition model, and interfering with this path through the method of this invention.

[0042] Specifically, Figure 1 The upper part illustrates the unprotected attack path: First, in the "public voice release" stage, the "user / speaker" releases the "raw voice"; then, the raw voice is "collected by a third party," enters a "third-party voice library," and is "data scraped"; next, the attacker enters the "attacker exploitation" stage, "fine-tunes" the collected raw voice, and updates the "fine-tuned pre-trained model" in the "inner loop optimization" according to the following formula:

[0043] ;

[0044] in, Indicates model parameters, This represents the updated model parameters. Indicates the learning rate. Represents the inner loss function. Represents the original speech. This tag indicates the actual speaker.

[0045] After passing through this unprotected attack path, the system enters the "vulnerable model and usage" stage, which leads to risky outcomes such as "model vulnerability", "successful targeted impersonation", and "voice spoofing". The above-mentioned parts of the path are connected by arrows and clearly marked as "unprotected attack path".

[0046] Figure 1The lower half illustrates the defense path of this invention: First, in the "publishing (perturbed speech)" stage, the "user / speaker" processes the original speech through the "speech perturbation generation system," and obtains the protective perturbation in the "perturbation generation" step, as shown in the figure. This illustrates the perturbation optimization process.

[0047] Subsequently, the perturbation is superimposed on the speech sample to form "perturbed speech". Afterwards, the perturbed speech also enters the "third-party speech library" and may be "data scraped"; when the attacker attempts to exploit the perturbed speech again, in the "attacker exploitation (failure)" stage, the attacker still performs "damaged fine-tuning" and updates the model in the "inner loop optimization" according to the following formula:

[0048] ;

[0049] However, since the input sample has been changed from the original speech Transformed into perturbed speech The image shows "The attacker attempted to..." The "fine-tuning" indicator and the red cross indicate that the fine-tuning process has been suppressed. After passing through the defense path of this invention, the system enters the "robust model and protection" stage, obtains the "robust speaker model", and achieves "successful interference" and "deception prevention".

[0050] also, Figure 1 The central concept of "voice privacy ecosystem" represents the overall environment in which this invention is situated, including the release, flow, collection, utilization, and protection of voice data. Each arrow indicates the flow of voice data, model updates, and attack / defense processes. The upper part corresponds to the process in which unprotected voice is collected by third parties and exploited by attackers in real-world scenarios, while the lower part corresponds to how this invention generates perturbative voice and interferes with the attacker's fine-tuning process, thereby blocking the risk propagation link from publicly available voice to vulnerable models.

[0051] This invention provides a method for generating unlearnable speech data to counter model fine-tuning attacks. The overall process is as follows: Figure 2 As shown, the process includes raw voice input, target identity selection, inner-layer simulation fine-tuning, outer-layer erroneous identity offset, perturbation constraint, and protected voice output. The specific steps are as follows:

[0052] 1. Raw speech input: Obtain raw speech samples and its real speaker tags Constructing a proxy speaker recognition model ;

[0053] 2. Initialize the disturbance: Initialize the disturbance variables And optionally compute a temporal saliency mask. ;

[0054] 3. Target Identity Selection: Target identity labels are determined for source samples using a candidate buffer pool and cosine similarity rules. ;

[0055] 4. Inner layer simulation fine-tuning: Inner layer optimization simulates attackers using protected speech. and real labels Fine-tuning the pre-trained model yields... ;

[0056] 5. In outer layer error identity offset: In outer layer optimization, with For temporary models, as Calculate the outer layer loss for the target label, and apply it according to the first-order approximation strategy. Find the gradient;

[0057] 6. Perturbation gradient update and perturbation constraints: Update according to the signed gradient or other gradient update rules. and execute Projection and spectral difference constraints;

[0058] 7. Iterative Judgment and Protected Voice Output: Repeat the iteration until the set number of rounds is reached or the convergence condition is met, then output the protected voice. .

[0059] in, , indicating a length of The original speech waveform; Represents the original speech The "real speaker" tag; , representing the protection disturbance variable to be optimized; , representing a temporal saliency mask; This indicates protected audio. If masking is not enabled, then... Take all 1s.

[0060] This invention simulates the fine-tuning process of an attacker on a pre-trained speaker recognition model, and then drives perturbed speech to shift towards a non-real-identity target category on the fine-tuned model, thereby generating publicly available, non-learnable speech. The core of this invention is supported by a "target identity selection mechanism" and a "two-layer optimization update mechanism"; while spectral difference constraints, temporal saliency masks, multi-sub-perturbation templates, and deterministic mapping rules are enhancements to the method of this invention.

[0061] (a) Agent speaker recognition model structure

[0062] This invention provides a proxy speaker recognition model. , by pre-trained speech representation network Classification of speakers Combining, represented as:

[0063] ;

[0064] in, For extracting high-level speaker representations from input speech, pre-trained speech representation networks such as Wav2Vec2 are preferred. Used to identify speakers based on their representations, preferably using the AAM-Softmax classification head, but can also be replaced with a regular fully connected classification head, ArcFace head, or other speaker classification heads.

[0065] It should be noted that the core of this invention does not lie in using a pre-trained representation network or a classification head alone, but rather in combining a pre-trained speech representation network with a differentiable classification head into a proxy attack model that can participate in two-layer optimization. This model can realistically simulate the attacker's fine-tuning and updating process of the pre-trained speaker recognition system in the inner layer stage, and can apply targeted optimization to the protected speech based on the same model state in the outer layer stage, which is oriented towards error identity mapping.

[0066] Unlike existing methods that only target static models, freeze feature extractors, or construct perturbations from randomly initialized models, the proxy speaker recognition model structure in this invention explicitly retains the linkage and update relationship between pre-trained representation extraction capabilities and downstream identity discrimination capabilities. This allows for a more accurate depiction of the technical paths of pre-trained representation extraction, classification head adaptation, and parameter fine-tuning in real-world attack scenarios. The protective perturbations generated based on this structure no longer only generate one-time interference for fixed decision boundaries but can optimize for erroneous identity association results in the fine-tuned model. Therefore, it is more suitable for protection scenarios involving pre-trained speaker recognition models.

[0067] In particular, the proxy speaker recognition model structure of this invention also has good versatility and transferability. Under the premise of keeping the main structure of the pre-trained representation network combined with the differentiable classification head unchanged, it can be implemented by replacing different types of pre-trained encoders or classification heads without changing the core idea of ​​this invention to simulate fine-tuning attacks and induce incorrect identity mapping through two-layer optimization.

[0068] Therefore, this structure is not a simple patchwork of existing speaker recognition models, but a functional proxy modeling structure designed for fine-tuning attack and defense tasks.

[0069] (II) Mathematical Definition of Bi-level Optimization

[0070] This invention's two-layer optimization includes inner-layer optimization and outer-layer optimization. The solution process for the two-layer optimization is as follows: Figure 4 As shown, the inner layer update is obtained. The outer layer will Treat it as a constant and treat the disturbance variable The complete process of calculating the first-order approximate gradient.

[0071] Specifically, let the original speech be... Authentic labels Target identity tags are Protected voice is ;in, This is an optional temporal saliency mask.

[0072] (1) Inner layer optimization: simulate an attacker using protected speech and real labels to fine-tune the pre-trained model in one or more steps.

[0073] In this embodiment, cross-entropy speaker classification loss is preferably used; when using the AAM-Softmax classification head, the inner loss is written as:

[0074] ;

[0075] The corresponding inner update step is:

[0076] ;

[0077] in, is the inner learning rate. The inner optimization is used to simulate the process by which an attacker fine-tunes the collected protected speech as if it were real supervised samples.

[0078] (2) Outer layer optimization: in the temporary model Above, directing the same protected voice toward a non-real target identity. Offset. The outer layer loss is preferably a weighted sum of the target identity classification loss and the spectral difference constraint, expressed as:

[0079] ;

[0080] The first term is used to induce the model to misclassify the source sample as the target identity after fine-tuning; the second term is used to constrain the spectral difference between the protected speech and the original speech to reduce the impact on audibility.

[0081] In particular, the spectral difference term of the present invention can also be replaced by the STFT domain L2 norm, Mel spectral difference, or other perceived similarity indicators.

[0082] (3) Gradient solution method in bi-level optimization

[0083] Considering that when rigorously solving the two-layer optimization, the outer gradient needs to penetrate the inner model parameter update process, which involves second-order gradients and high engineering overhead, this invention preferably adopts a first-order approximate gradient calculation solution strategy.

[0084] Specifically, during the inner layer update phase, the protected voice... Considered as the input to the stopping gradient, the model parameters are first processed from... arrive Update;

[0085] In the outer layer optimization stage, Considered relative to The constant is used to calculate only the gradient from the perturbation variable to the outer loss, i.e.:

[0086] ;

[0087] This strategy is essentially a first-order look-ahead approximation: ignoring inner pairs. Higher-order dependencies, only retaining the outer loss pairs The direct gradient is used, and the inner layer update is only responsible for approximating the model state after the attacker's fine-tuning, while the outer layer optimizes the perturbation variables on the model state.

[0088] Furthermore, after obtaining the outer layer loss with respect to the perturbation variable... After the gradient, for the perturbation variable Perform the update. Preferably, the update is performed using signed gradient descent, as follows:

[0089] ;

[0090] in, Indicates the first The perturbation variable in the next iteration This represents the updated perturbation variable. Indicates the perturbation update step size. Represents a symbolic function. express The constrained projection operator is used to limit the updated perturbation variable to a preset perturbation amplitude threshold. Within the corresponding feasible region.

[0091] After updating the perturbation variables, a temporal saliency mask can be further incorporated. Generate new protected speech:

[0092] ;

[0093] thus, Figure 4 The two-layer optimization process can be fully represented as follows: first, temporary model parameters are obtained through inner layer update. Based on Calculate the gradient of the outer layer loss with respect to the perturbation variable δ, and apply the gradient to... Update and project constraints are performed to achieve iterative optimization of the protected speech.

[0094] Because this invention does not explicitly backpropagate the gradient terms of the perturbation variables in the inner layer model parameter update chain during the outer layer optimization process. This avoids the explicit solution of high-order gradient information in strict bi-level optimization. Therefore, it can significantly reduce the implementation difficulty and engineering complexity of the overall solution process, reduce the consumption of computing resources and memory overhead, and improve the feasibility of the method in practical systems.

[0095] Compared with solution methods that require complete tracking of the coupling gradients of inner and outer layers, the first-order approximation strategy adopted in this invention does not require the construction of a complex high-order backpropagation process, making it easy to implement directly under the existing deep learning training framework. At the same time, it helps to improve the stability and iterative efficiency of the solution process.

[0096] Based on this, the improvement does not change the main technical concept of the invention, and still retains the core logic of "first simulating the attacker's fine-tuning process on the pre-trained speaker recognition model through inner layer updates, and then optimizing the perturbation variables based on the updated model state in the outer layer, thereby inducing the model to establish an incorrect identity mapping." Therefore, although the present invention has simplified the gradient solution method, it can still effectively approximate the model evolution path under real fine-tuning attack scenarios, making the generated protection perturbation both targeted and practical.

[0097] Specifically, this invention preferably employs a one-step inner-layer fine-tuning approach. This effectively simulates the attacker's parameter adaptation process while further reducing the computation time required for a single iteration, lowering the overall optimization cost, and thus making it more suitable for the engineering requirements of multi-round perturbation generation in voice protection scenarios. When it is necessary to improve the fitting degree of the attack process, the inner-layer update method can also be extended to a multi-step fine-tuning approximation to enhance the simulation capability for more complex attacker training behaviors. Therefore, this invention achieves a good balance between protection effectiveness, implementation complexity, and computational efficiency.

[0098] (III) Quantitative Definition of Target Identity Selection Mechanism

[0099] To ensure that the outer layer optimization has a clear misleading direction, this invention assigns a non-real target identity label to each source sample. The target identity selection mechanism process of this invention is as follows: Figure 3 The diagram illustrates the complete process of input sample processing, embedding extraction, FIFO candidate pooling, elimination of similar samples, and argmax target identity determination. The specific steps are as follows:

[0100] 1. Use a surrogate model to extract the normalized speaker embeddings of the current batch of samples. ;

[0101] 2. Write the current batch embedding and its tags into the candidate buffer pool. The buffer pool preferably adopts a first-in, first-out (FIFO) structure, with a maximum capacity denoted as [missing information]. ;

[0102] 3. For the source sample Eliminate all candidates that meet the criteria from the candidate pool. For candidates of the same category, only candidates not belonging to this category are retained. ;

[0103] 4. In the preferred implementation, calculate Based on the target identity selection rules, according to Select Select the non-native identity with the highest cosine similarity to the source sample as the target identity, and input the target label. :

[0104] ;

[0105] This definition suggests selecting the competing identity in the representation space that is most easily confused with the source sample, thereby increasing the probability of an attacker establishing an incorrect identity mapping after fine-tuning. For even stronger conservatism, a similarity threshold can also be used. After selecting the candidate set, argmax is then taken.

[0106] Specifically, when no available non-class samples are available in the candidate pool, a backoff strategy is adopted, randomly selecting a preset label from all non-real identity labels, or selecting according to the cyclic offset rule. Where mod represents the mathematical modulo operation, This represents the total number of categories.

[0107] The update logic for the buffer pool is as follows: whenever a new batch of data is written, the oldest batch of data is popped from the buffer pool to keep the capacity within a certain limit. Preferably, Take the most recent 5 to 10 batches of embedded sets to balance the stability of target identity selection and storage overhead.

[0108] (iv) Constraints and Enhancements

[0109] (1) Spectral difference constraint:

[0110] To ensure the publishability of protected speech, this invention imposes constraints on the short-time spectral differences between the protected speech and the original speech.

[0111] The present invention preferably uses the L1 norm difference after short-time Fourier transform (STFT), expressed as:

[0112] ;

[0113] It can also be replaced by L2 norm difference, Mel spectrum difference, or other perceptual similarity indicators.

[0114] (2) Temporal saliency mask:

[0115] In a preferred enhanced implementation of the present invention, the temporal saliency mask generation process is as follows: Figure 5 As shown. Figure 5 From left to right, the process includes four stages: “input speech and loss”, “gradient calculation”, “gradient magnitude”, and “normalization to obtain temporal mask”. These stages illustrate how to construct a temporal saliency mask from the gradient information of the input speech obtained from the classification loss.

[0116] Specifically, first input the original voice. and classification loss Let the original speech be of length . The time-domain waveform, i.e.:

[0117] ;

[0118] in, Indicates the time of input voice. The sampled values, This represents the total number of sampling points for the input speech. Based on the classification loss... The gradient of the input speech is calculated point by point to obtain the time step. Gradient value at:

[0119] ;

[0120] in, Represents classification loss Input speech sample values The gradient.

[0121] Then, the gradient is magnituded, i.e., its absolute value is taken, to construct the time-series significance score:

[0122] ;

[0123] in, Indicates time The temporal saliency score at that point. This saliency score characterizes the sensitivity to speaker identification at that moment. The larger the value, the greater the impact of the corresponding time location on the classification loss, and the more suitable it is as a key area for perturbation.

[0124] Furthermore, the saliency scores are normalized to obtain the temporal saliency mask:

[0125] ;

[0126] in, Indicates time The normalized temporal saliency mask value, To prevent extremely small constants with a denominator of zero, Indicates time The significance score at the location.

[0127] This yields the complete temporal saliency mask vector:

[0128] .

[0129] Figure 5 The middle section further illustrates the temporal variation process from gradient magnitude to mask, where, Figure 5 (a) in the text represents the input speech waveform. , Figure 5 In the equation (b), the significance score is obtained from the gradient magnitude. , Figure 5 (c) in the text represents the mask after normalization. .from Figure 5 As can be seen from (a) to (c), the normalized mask has a higher value in the time segment where identity determination is more sensitive, thus guiding subsequent perturbations to be preferentially allocated to key time domain locations.

[0130] Specifically, the significance score and the normalization formula are:

[0131] ;

[0132] ;

[0133] The mask works by masking the temporal saliency. With disturbance variables Perform element-wise multiplication and then superimpose it onto the original speech to generate protected speech. In this way, the perturbations will be concentrated on the sensitive periods of identity discrimination, thereby improving the efficiency of perturbation utilization under the same perturbation budget and enhancing the interference effect on model fine-tuning attacks.

[0134] In some implementations, if a frame-based approach is used, the gradient magnitude can be calculated frame by frame first, and the intra-frame gradients can be pooled to obtain a frame-level saliency score. Then, the frame-level saliency score can be mapped back to the sampling point level through interpolation or upsampling to generate the final temporal saliency mask. .

[0135] (3) Multi-subject perturbation template:

[0136] In an optional implementation of the invention, each speaker category can maintain multiple sub-perturbation templates. ;in, The sample-level index can be obtained from the sample path, filename, or unique number through a deterministic mapping function to avoid inconsistencies in the index under different operating environments. This module is used to alleviate the averaging problem caused by the sharing of a single perturbation among similar samples, and does not constitute a necessary limitation of the main invention.

[0137] Example 1

[0138] A method for generating unlearnable speech data to counter model fine-tuning attacks is provided, and the overall implementation is as follows:

[0139] Assuming the input speech sampling rate is 16 kHz, fixed-duration speech segments are extracted as training samples. The surrogate model uses a pre-trained Wav2Vec2 encoder as the backbone network, followed by an AAM-Softmax speaker classification head. For any sample... and its real label First initialize the perturbation variables And limit the disturbance to Within the constraints.

[0140] In each iteration, the target identity label is first determined based on the current batch of samples and the candidate buffer pool. Then, protected voice was used. With real labels Perform one inner-layer fine-tuning step to obtain temporary model parameters. In the outer layer stage, the same protected speech is input again into the temporary model. , as the target identity Calculate the target classification loss and add a spectral difference constraint term to obtain the outer layer loss. Then, under the first-order approximation condition... Calculate the gradient and update using signed gradient descent. After each update... implement Projection is used to ensure that the disturbance amplitude does not exceed the threshold. Repeat this process until the preset number of iterations is reached.

[0141] Example 2

[0142] A preferred implementation for target identity selection is provided, as follows:

[0143] For each sample in the current batch, first extract the normalized speaker embedding. And embed the current batch and write the tag to a capacity of The FIFO candidate buffer pool.

[0144] For the sample to be protected Exclude all true labels from the candidate pool For identical candidates, the remaining candidates are sorted by cosine similarity, and the label of the non-candidate with the highest similarity is taken as the [label of the candidate]. .

[0145] When the candidate pool is empty or there are no available non-candidates of this type, the preset fallback rule is applied. .

[0146] Example 3

[0147] An implementation of temporal saliency and perceptual constraint enhancement is provided, as follows:

[0148] Based on Example 1, the gradient of the input waveform is first calculated using the classification loss of the current proxy model on the input speech. The absolute value of the gradient is taken as the temporal saliency score, and a temporal saliency mask is obtained through normalization. .

[0149] Subsequently, the protected speech was rewritten as An STFT domain L1 difference term is added to the outer layer loss to ensure that the audibility of the original speech is maintained as much as possible while improving the ability to mislead the listener.

[0150] Example 4

[0151] A specific implementation of multi-sub perturbation is provided, as follows:

[0152] Based on Example 1, maintain for each speaker category Individual perturbation template When a sample to be protected enters the system, a deterministic index is calculated using the sample path or unique number. Then, the corresponding sub-perturbation template is called to participate in the update.

[0153] This implementation can reduce the over-averaging phenomenon caused by different speech samples sharing a single perturbation template.

[0154] Furthermore, to verify the effectiveness of the method of this invention, an experiment was conducted using a speech dataset containing 100 speakers. Each speaker included 25 speech samples, of which 20 were used as training samples and 5 as validation samples. The speech sampling rate was 16 kHz, and each input speech was truncated to 3 seconds. The attack model consisted of a pre-trained speech representation network and a speaker classification head, and was trained using the last2 fine-tuning mode.

[0155] In the experiment, equal error rate (EER) was used as the performance index for speaker verification. The higher the EER, the weaker the attack model's ability to learn speaker identity features using the speech samples, indicating that the corresponding protection method has a stronger audio unlearnability effect.

[0156] Under unprotected conditions, the attack model trained using the original speech has an EER of approximately 0.072–0.081, indicating that attackers can effectively learn speaker identity features. After generating protected speech using the traditional SLEM method in HiddenSpeaker, the attack model's results at the 50th training epoch are: training loss approximately 0.0831, validation loss approximately 1.7437, recognition accuracy approximately 0.736, and EER approximately 0.071, indicating that this method has limited protection effectiveness in scenarios with pre-trained attack models.

[0157] In contrast, after generating protected speech using the target identity selection mechanism described in this invention, the training model at the 50th training cycle has the following results: training loss of approximately 0.1838, validation loss of approximately 5.6395, recognition accuracy of approximately 0.294, and EER of approximately 0.164; the corresponding attack model at the 50th training cycle has the following results: training loss of approximately 0.1989, validation loss of approximately 5.3724, recognition accuracy of approximately 0.318, and EER of approximately 0.160.

[0158] Therefore, compared with unprotected baselines and traditional methods, the present invention can significantly improve the error rate of the attack model and significantly reduce its speaker recognition accuracy, thereby more effectively weakening the attack model's ability to learn the identity features of the real speaker and improving the audio non-learnable protection effect.

[0159] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating unlearnable speech data to counter model fine-tuning attacks, characterized in that, Includes the following steps: Step 1: Obtain raw speech samples and its real speaker tags Based on a pre-trained speech representation network and a differentiable classification head, a proxy speaker recognition model is constructed. ; Step 2: Initialize the perturbation variables And calculate the temporal saliency mask. ; Step 3: Determine the target identity label for the source sample using the candidate buffer pool and cosine similarity rule. The proxy speaker recognition model Perform two-layer optimization; Step 4: Simulate an attacker using inner-layer optimization, employing protected speech. and real speaker tags The proxy speaker recognition model Fine-tuning was performed to obtain the model parameters. ; Step 5: In outer layer optimization, with For temporary models, for target identity labels Calculate the outer layer loss and apply the perturbation variable according to the first-order approximation strategy. Solve for the gradient; Step 6: Update the perturbation variables according to the gradient update rule. and execute Projection and spectral difference constraints; Step 7: Repeat steps 3 to 6 until the set number of rounds is reached or the convergence condition is met, then output the protected voice. .

2. The method for generating unlearnable speech data according to claim 1, characterized in that, In step 1, the proxy speaker recognition model For fine-tuning attack and defense tasks, it is constructed using a pre-trained speech representation network. Classification of speakers The combination forms the following formula: ； in, For model parameters, Used to extract high-level speaker representations from input speech. Used to determine identity based on the speaker's characteristics.

3. The method for generating unlearnable speech data according to claim 1, characterized in that, In step 2, the temporal saliency mask The gradient magnitude of the input speech is calculated based on the classification loss to obtain a temporal saliency score, which is then normalized to generate the following formula: ； ； in, For classifying losses, For input voice at time The sampled values, The significance score for the time series. For a moment The time-series significance score at the position, This represents the total number of sampling points for the input speech. To prevent extremely small constants with a denominator of zero, For a moment The normalized temporal saliency mask value.

4. The method for generating unlearnable speech data according to claim 2, characterized in that, In step 3, a non-real target identity label is determined for each source sample. The specific steps are as follows: Step 3.1: Use a pre-trained speech representation network to extract the normalized speaker embeddings of the current batch of samples. : ； in, This indicates a normalization operation. For source sample index, This represents the voice source samples input in the current batch. For source samples Normalized speaker embedding; Step 3.2: Embed the speakers of the current batch and its labels Write to candidate buffer pool : ； in, This is a candidate sample index used to identify samples written into the candidate buffer pool. Candidate samples The normalized speaker embedding; the candidate buffer pool adopts a first-in-first-out structure with a maximum capacity of ; Step 3.3, for the source sample Exclude all candidates that meet the criteria from the candidate buffer pool. For candidates, only those not belonging to this category will be retained. : ； in, For source samples The corresponding set of candidate indices not belonging to this class. For source samples The "real speaker" tag; Step 3.4: Select the non-native identity with the highest cosine similarity to the source sample as the target identity: ； in, The candidate sample index that maximizes the cosine similarity. For index Speaker labels corresponding to candidate samples, Used in candidate index sets that are not of this class The candidate sample index with the highest cosine similarity to the source sample is selected. For source samples Corresponding target identity tags; Step 3.5: When no available non-class samples are found in the candidate buffer pool, a backoff strategy is adopted, randomly selecting a preset label from all non-real identity labels, or taking one according to the cyclic offset rule: ; where mod represents the mathematical modulo operation, This represents the total number of categories.

5. The method for generating unlearnable speech data according to claim 2, characterized in that, In step 4, the inner layer optimization simulates an attacker exploiting protected voice and real speaker tags. The proxy speaker recognition model Perform one or more fine-tuning steps; The protected voice , represented as: ； in, This is element-wise multiplication; The inner loss uses cross-entropy speaker classification loss: ； The corresponding inner update step is: ； in, The inner learning rate, The cross-entropy loss function is used for speaker classification. To adjust model parameters gradient operator, For inner layer loss, These are the optimized model parameters.

6. The method for generating unlearnable speech data according to claim 5, characterized in that, In step 5, the outer layer optimization is performed to... As a temporary model, the same protected speech is directed toward a non-real target identity label. Offset; The outer layer loss is a weighted sum of the target identity classification loss and the spectral difference constraint: ； in, This is a temporary model. For the weights of the spectrum difference constraint term, It is the short-time Fourier transform function. For outer layer loss; first term For target identity classification loss, the second term For spectral difference constraints.

7. The method for generating unlearnable speech data according to claim 6, characterized in that, The speaker classification head may be an AAM-Softmax classification head, a fully connected classification head, or an ArcFace head. The spectral difference constraint adopts the L1 norm difference, L2 norm difference, or Mel spectral difference after short-time Fourier transform.

8. The method for generating unlearnable speech data according to claim 6, characterized in that, In step 5, the first-order approximation strategy, during the inner layer update phase, will protect the speech... Considered as the input to the stopping gradient, this completes the transformation of the model parameters from... arrive The update; during the outer layer optimization phase, Considered relative to the disturbance variable The constant is used to calculate only the gradient from the perturbation variable to the outer loss, as shown in the formula: ； in, Indicates the perturbation variable The gradient operator.

9. The method for generating unlearnable speech data according to claim 1, characterized in that, Step 6 includes the following sub-steps: Step 6.1: Update the perturbation variables using signed gradient descent. After each update implement Projection ensures that the disturbance amplitude does not exceed a preset threshold. ; Step 6.2: Perform spectral difference constraints to impose constraints on the short-term spectral differences between the protected speech and the original speech, ensuring the publishability of the protected speech.

10. The method for generating unlearnable speech data according to claim 1, characterized in that, Step 2 also includes multi-subject perturbation processing to alleviate the averaging problem caused by similar samples sharing a single perturbation; Each speaker category maintains multiple sub-perturbation templates. ,in , This indicates the number of sub-perturbation templates maintained for each speaker category; when a sample to be protected enters the system, the sample-level index is obtained from the sample path, file name, or unique number through a deterministic mapping function.

Images