Counterfeit voice attribution method and system based on robust confrontation defense boundary

Abstract

The invention discloses a forged voice attribution method and system based on a robust confrontation defense boundary. The forged voice attribution method comprises the following steps: acquiring a voice sample; inputting the voice sample into a trained forged voice attribution model, and outputting the probability that the voice sample belongs to various voice forging methods as the recognition result of the forged voice attribution; the faked voice attribution model comprises a random transformation module which is used for carrying out random combination transformation on voice samples and obtaining time-frequency characteristics of the voice samples; the feature extraction module is used for extracting depth features of the voice samples from the time-frequency features; and the feature classification module is used for carrying out category judgment on the depth features and outputting the probability that the voice sample belongs to various voice counterfeiting methods. When faked voice attribution is carried out, the voice faking method category can be accurately judged under the condition of no adversarial disturbance, and the recognition precision of the voice faking method category can be greatly improved under the condition of existence of adversarial disturbance.

Description

Technical Field

[0001] The present invention belongs to the technical field of forged speech attribution, and in particular relates to a forged speech attribution method and system based on robust adversarial defense boundaries. Background Art

[0002] The emergence and rapid development of deepfakes have increased the content security risks in social media. Defense against voice and image deepfakes has become an important research direction in information security. In recent years, defense against voice deepfakes has evolved from detecting the authenticity of voice information to tracing the source of voice forgery methods. Forged Speech Attribution (FSA) is one of the key technologies in tracing the source of voice forgery methods, and is also a key link in establishing a robust and explainable voice anti-spoofing strategy.

[0003] However, most of the existing FSA works use deep learning models and do not consider the impact of factors such as adversarial sample attacks. In real scenarios, attackers may add tiny adversarial perturbations to forged speech to form adversarial samples to mislead the recognition model's judgment of the forgery method category, which is equivalent to disguising the forgery method to prevent it from being traced. This adversarial perturbation is imperceptible to human hearing. Although improving the adversarial robustness of deep learning models has always been a key issue in the field of artificial intelligence, preliminary experiments have shown that the adversarial robustness of existing FSA models is weak and cannot accurately identify forged speech that has been disguised by perturbations. For this reason, it is urgent to conduct FSA algorithm research for forged speech under adversarial attack conditions and improve the adversarial robustness of FSA models. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present invention provides a forged speech attribution method and system based on robust adversarial defense boundaries. When attributing forged speech, the present invention can not only accurately identify the speech forgery method without adversarial disturbance, but also greatly improve the recognition accuracy of the speech forgery method in the presence of adversarial disturbance.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] In a first aspect, a method for attributing a forged voice is provided, comprising: collecting a voice sample; inputting the voice sample into a trained forged voice attribution model, and outputting the probability that the voice sample belongs to various voice forgery methods as a recognition result of the forged voice attribution; wherein the forged voice attribution model comprises: a random transformation module for performing random combination transformation on the voice sample and obtaining its time-frequency features; a feature extraction module for extracting deep features of the voice sample from the time-frequency features; and a feature classification module for performing category judgment on the deep features and outputting the probability that the voice sample belongs to various voice forgery methods.

[0007] Furthermore, the training method of the forged speech attribution model includes: inputting the speech samples in the training set into the random transformation module to obtain the time-frequency features of the speech samples after random combination transformation; using the time-frequency features as input, and using the feature extraction module to extract the deep features of the speech samples; outputting the deep features to the feature classification module, and outputting the predicted logical values ​​of the speech samples belonging to each category; using the decision boundary distance regularization module, using the predicted logical values ​​as input, constructing the adversarial robustness upper limit of the model, and calculating the decision boundary distance regularization loss; according to the predicted logical values, obtaining the predicted label of the sample, and calculating the classification loss; using the linear combination of the decision boundary distance regularization loss and the classification loss as the robust adversarial defense boundary loss function to supervise the training of the feature extraction module and the feature classification module, and obtaining the weight parameters of the feature extraction module and the feature classification module.

[0008] Furthermore, the speech samples are randomly combined and transformed to obtain their time-frequency features, including: based on the set n transformation types, randomly generating the number of quasi-sampling transformations , n represents the number of set transformation types; for the number of transformations obtained by sampling Q, for the first Secondary transformation , the type of randomly generated transformation , the probability of using this transformation and the intensity of the transformation ; Among them, for the type of transformation , different Corresponding to different transformation types, randomly select from the set n transformation types; use transformation Probability follows a Bernoulli distribution with parameter p, i.e. using the transformation The probability is p; using the transformation Strength Follow the uniform distribution of given parameters; apply Q transformations to the input speech samples in sequence to obtain samples that have been randomly combined and transformed :

[0009] (1-1)

[0010] in, represents an arrangement scheme of the Q transformations; each transformation is a three-element random parameter group, consisting of These three random variables are used to describe; Indicates that the parameter is The transformation function of the random combination of transformed speech samples is converted into logarithmic Mel spectrum using short-time Fourier transform, Mel filtering and logarithmic operation, that is, the time-frequency characteristics of the random combination of transformed speech samples. , expressed as ,in, represents the random transformation module, represents the short-time Fourier transform, Mel filtering and logarithmic operation.

[0011] Furthermore, the n types of transformations are set, including: reverberation effect transformation, environmental noise superposition transformation, background music superposition transformation and noisy human voice superposition transformation.

[0012] Furthermore, we construct the upper limit of the model's adversarial robustness, including: first, defining a random neural network model The adversarial defense boundary is

[0013] (1) Among them, For the Class relative to The adversarial defense boundary of the class is:

[0014] (2)

[0015] in, It is a voice sample The true label of It is the model obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. The weight of It is obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. A randomly transformed version of ; The random transformation parameters sampled by the Monte Carlo algorithm are used when implementing adversarial sample attacks; It is obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. The input is calculated as:

[0016] (3)

[0017] in, To implement adversarial sample attacks, add The adversarial perturbation on ; To solve the optimal disturbance when counteracting the disturbance, the calculation formula is:

[0018] (4)

[0019] in, and They represent the models obtained by sampling using the Monte Carlo algorithm when solving the optimal perturbation. The weights and transformation parameters of the input, Indicates the number of sampling times, Represents the loss function used by the Transformation Expectation Algorithm; the Transformation Expectation Algorithm is used in combination with the gradient-based attack method to solve the optimal perturbation for the random neural network; when the gradient-based attack method is used to generate adversarial samples, in each iteration of updating the adversarial sample, the Transformation Expectation Algorithm uses the Monte Carlo algorithm to sample multiple inputs or target model weights, and uses the average of the gradients calculated based on the sampled inputs or model weights as the final gradient;

[0020] Next, based on the adversarial defense boundary described in formula (2), the model is obtained About Categories The adversarial robustness condition is:

[0021] (5)

[0022] That is, when Any Monte Carlo sampling transformation version of , All located at the border When the side is greater than 0, the model Still able to correctly identify The label of the model About Categories is robust against adversarial forces;

[0023] Then use Taylor's formula at point Expand the left side of formula (5) to obtain:

[0024] (6)

[0025] in, represents the Euclidean norm operator, yes The higher-order infinitesimals of represents the gradient, yes and The angle between them; Formula (6) is obtained by identity transformation:

[0026] (7)

[0027] in, It is one of the factors that affect the model's adversarial robustness; It is a voice sample The true label, therefore, , the first term on the left side of formula (7) is greater than 0, and the numerator of the second term on the left side is is an infinitesimal term, much smaller than ; In the design of adversarial samples, Relative to is very small; therefore, the second term does not affect the overall sign of the left side, and the sign of the left side of formula (7) is determined by the sign of the first term on the left side; therefore, the left side of formula (7) is greater than 0;

[0028] because ,distinguish Degrees, 0 degrees 90 degrees and 90 degrees 180 degrees, three situations, respectively analyzing the factors that affect the model's adversarial robustness;

[0029] ① When 90 degrees At 180 degrees, ; At this time, the left side of formula (7) is greater than 0, because , so the left side of formula (7) is greater than , that is, formula (7) holds, then inequality (5) holds; that is, when 90 degrees At 180 degrees, Able to correctly predict the category of adversarial samples, the model is adversarially robust;

[0030] ② When 0 degrees At 90 degrees, ; Strengthen the condition of formula (7) to:

[0031] (8)

[0032] As long as formula (8) holds, then formula (7) holds; formula (8) is transformed by identity to obtain:

[0033] (9)

[0034] ③ When Degrees, ,Right now and The direction is consistent, which includes the following special cases: is a special SNN, i.e. a fixed neural network; when When the neural network is fixed, the attacker targets the same model weight parameters when solving the optimal perturbation and using the perturbation to attack, and the transformation parameters of the input are equivalent to the identity transformation; at this time, the optimal perturbation obtained is in the gradient direction of the loss function with respect to the input, that is, and The direction is consistent. ; At this time, formula (7) is equivalent to formula (8), and after identity transformation, formula (9) is obtained;

[0035] For the second and third cases, if , formula (9) holds, then formula (7) holds, and similarly formula (5) holds; the left side of formula (9) is defined as ,Right now:

[0036] (10)

[0037] because , Equivalent in form to a sample With Class and Decision boundary of the class The distance between The model is defined as The upper bound of adversarial robustness in dimensionality; The larger the value, the greater the possibility that formula (9) is true. The stronger the adversarial robustness in the dimension, the stronger the It is a factor that affects the adversarial robustness of the model. Taking into account the adversarial robustness of the model in various dimensions, the upper limit of the adversarial robustness of the model is constructed as:

[0038] (11)

[0039] is the minimum upper limit of the model’s adversarial robustness in each dimension; therefore, increasing , will help increase each , thus increasing the possibility that formula (9) is true, which is beneficial to enhancing the adversarial robustness of the model;

[0040] For a fixed neural network, the condition for the network to be robust against adversarial forces is that formula (8) holds true, that is, the left side of formula (8) is greater than 1; while for an SNN, the condition for the network to be robust against adversarial forces is that formula (7) holds true, that is, the left side of formula (7) is greater than ; Since the left sides of formula (7) and formula (8) are the same, and ,Therefore, under the condition that other factors are the same, the adversarial robustness condition of SNN is easier to be met, that is, SNN has stronger adversarial robustness.

[0041] Further, the decision boundary distance regularization loss is calculated, including: for the speech sample , whose true label is , the time-frequency characteristics of the sample after random combination transformation are ,in, They are The width and height of extract The deep features ,in, They are deep features The number of channels, width and height; use the feature classification module , output A vector of predicted logical values ​​belonging to each class ,in, is the number of categories, ; Based on the adversarial robustness upper limit of the constructed model, calculate the decision boundary distance regularization loss for:

[0042] (12)

[0043] in, is the threshold parameter, represents the gradient; minimize Encourage the model to learn satisfy ;in, The formula is:

[0044] (13)

[0045] in, express Belong to The predicted logical value of the class, express Belong to The predicted logical value of the class, Represents the logical value vector right gradient.

[0046] Further, calculating the classification loss includes: using a Softmax cross entropy loss function or an AM-Softmax loss function to calculate the classification loss;

[0047] Consider a batch of samples , the number is , taking the logical value and label output by the feature classification module as input,

[0048] When the Softmax cross entropy loss function is used to calculate the classification loss, the formula is:

[0049] (14)

[0050] in, Indicates samples, express Belong to the categorical logical value of the class, Representation sample The corresponding one-hot encoded vector, Represents logarithmic operation;

[0051] When the AM-Softmax loss function is used to calculate the classification loss, the calculation formula is as follows: (14). The only difference is the calculation method of the logical value. The logical value used by the AM-Softmax loss function is:

[0052] (14-1)

[0053] in, The samples are The deep features and classification modules of Class classifier weights The intra-class and inter-class similarities of is the margin imposed on the intra-class similarity, where .

[0054] Furthermore, the robust adversarial defense boundary loss function is:

[0055] (15)

[0056] in, is a parameter that balances the importance of the corresponding loss.

[0057] Furthermore, deep features of the speech samples are extracted, including: taking the time-frequency features as input, and using ResNet-50 to extract the deep features of the speech samples; ResNet-50 has four convolution blocks, and the output of the last convolution block is used as the deep features.

[0058] In a second aspect, a forged voice attribution system is provided, comprising: a data collection module for collecting voice samples; an attribution module for inputting the voice samples into a trained forged voice attribution model, and outputting the probability that the voice samples belong to various voice forgery methods as the recognition result of the forged voice attribution; wherein the forged voice attribution model comprises: a random transformation module for performing random combination transformation on the voice samples and obtaining their time-frequency features; a feature extraction module for extracting deep features of the voice samples from the time-frequency features; and a feature classification module for performing category judgment on the deep features and outputting the probability that the voice samples belong to various voice forgery methods.

[0059] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for attributing fake speech as described in the first aspect is implemented.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] (1) The present invention inputs the time-frequency features of the speech samples that have been randomly combined and transformed into a trained forged speech attribution model, and outputs the probability that the time-frequency features belong to various speech forgery methods as the recognition result of the forged speech attribution. When performing forged speech attribution, the method can not only accurately identify the type of speech forgery method without adversarial disturbance, but also greatly improve the recognition accuracy of the forgery method of forged speech disguised by adversarial disturbance.

[0062] (2) We simulate the interference that fake speech may be subject to in the real world and design a random transformation module. By combining multiple random transformations, we enhance the randomness of the input samples and improve the adversarial robustness of the model.

[0063] (3) We define the concept of adversarial defense boundary, analyze the adversarial robustness of the model based on it, and design a decision boundary distance regularization module. By maximizing the difference between the logical value of the correct category and the logical value of other categories and minimizing the gradient of the difference with respect to the input, we reduce the sensitivity of the model output logical value to adversarial perturbations and improve the adversarial robustness.

[0064] (4) Compared with existing forged speech attribution methods, the method based on robust adversarial defense boundary can not only improve the recognition effect of speech forgery methods under unperturbed conditions, but also significantly improve the recognition accuracy of forgery methods for forged speech disguised with adversarial perturbations;

[0065] (5) Compared with existing adversarial defense methods, the method based on robust adversarial defense boundary analyzes the adversarial robustness of the model from the perspective of adversarial defense boundary for the first time. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1It is a main flow chart of a forged speech attribution method based on a robust adversarial defense boundary provided by an embodiment of the present invention;

[0067] Figure 2 It is a visualization diagram of features learned on the evaluation set of the 2023 Chinese Fake Audio Detection (CFAD2023) dataset, where (a) is a visualization of features learned by the WCA method, and (b) is a visualization of features learned by the method of the present invention;

[0068] Figure 3 It is a curve showing the change of the upper limit of the adversarial robustness of the model learned by the method of the present invention and other methods as the training progresses when training on the training set of the CFAD2023 dataset;

[0069] Figure 4 It is a flow chart of a training method for a forged speech attribution model in a forged speech attribution method based on a robust adversarial defense boundary provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0071] Embodiment 1

[0072] like Figure 1~Figure 4 As shown, a method for attributing fake speech based on robust adversarial defense boundaries includes: collecting speech samples; inputting the speech samples into a trained fake speech attribution model, and outputting the probability that the speech samples belong to various speech forgery methods as the recognition result of the fake speech attribution; wherein the fake speech attribution model includes: a random transformation module, which is used to perform random combination transformation on the speech samples and obtain their time-frequency features; a feature extraction module, which is used to extract the deep features of the speech samples from the time-frequency features; a feature classification module, which is used to perform category judgment on the deep features and output the probability that the speech samples belong to various speech forgery methods.

[0073] The feature extraction module in the present invention uses a deep convolutional neural network to extract the deep features of the speech sample, including: taking the time-frequency features as input, using ResNet-50 to extract the deep features of the speech sample; ResNet-50 has four convolution blocks, and the output of the last convolution block is used as the deep features; the deep features of the obtained speech sample are output to the feature classification module. It is worth noting that other feature extraction networks based on deep learning can also be used as a solution for extracting the deep features of speech samples, such as ResNet-18. The feature classification module performs category judgment on the deep features of the speech sample; the feature classification module is a fully connected layer followed by a Softmax function, and the number of output nodes of the fully connected layer is the same as the total number of speech forgery method categories.

[0074] For the input speech sample , use the random transformation module to obtain a random combination transformation version of the speech sample, and use short-time Fourier transform, Mel filtering and logarithmic operation to convert the random transformation speech sample into a logarithmic Mel spectrum as a time-frequency feature ,in, They are width and height.

[0075] Optionally, use ResNet-50 as the feature extraction module ResNet-50 has four convolutional blocks and uses the output of the last convolutional block as the deep feature. ,in, They are deep features The number of channels, width and height.

[0076] Use the fully connected layer as a feature classification module. This module takes deep features as input and outputs a vector of predicted logical values ​​for each category. ,in, is the number of categories, The index corresponding to the maximum value in the probability vector corresponds to the category label of the sample.

[0077] The present invention trains the forged speech attribution model by the following method.

[0078] (1) Input the speech samples in the training set into the random transformation module to obtain the time-frequency features of the speech samples after random combination transformation.

[0079] Perform random combination transformation on speech samples and obtain their time-frequency features, including:

[0080] Based on the set n transformation types, randomly generate the number of quasi-sampling transformations , n represents the number of set transformation types;

[0081] For the number of transformations obtained by sampling, Q, Secondary transformation , the type of randomly generated transformation , the probability of using this transformation and the intensity of the transformation ;

[0082] Among them, for the type of transformation , different Corresponding to different transformation types, randomly select from the set n transformation types; use transformation Probability follows a Bernoulli distribution with parameter p, i.e. using the transformation The probability is p; using the transformation Strength follows a uniform distribution with given parameters;

[0083] Apply Q transformations to the input speech sample in sequence to obtain a sample that has been randomly combined and transformed. :

[0084] (1-1)

[0085] in, represents an arrangement scheme of the Q transformations; each transformation is a three-element random parameter group, consisting of These three random variables are used to describe; Indicates that the parameter is The transformation function of

[0086] Use short-time Fourier transform, Mel filtering and logarithmic operation to convert the randomly combined transformed speech samples into logarithmic Mel spectrum, that is, the time-frequency characteristics of the randomly combined transformed speech samples , which can be expressed as ,in, represents the random transformation module, represents the short-time Fourier transform, Mel filtering and logarithmic operation.

[0087] In the present invention, the n types of transformations set include at least: reverberation effect transformation, environmental noise superposition transformation, background music superposition transformation and noisy human voice superposition transformation.

[0088] The reverberation effect is transformed to simulate the propagation of fake speech in the room, which is achieved by convolution of fake speech and room impulse response. , randomly select a room impulse response audio from the room impulse response dataset, and Perform convolution operation to obtain the fake speech after reverberation transformation.

[0089] Environmental noise superposition transformation simulates the situation where the fake voice is interfered by environmental noise. , randomly extract a piece of environmental noise audio from the NOISE subset of the MUSAN dataset. Then crop or zero-fill it to make its length the same as same, and added to the random signal-to-noise ratio according to The signal-to-noise ratio is randomly selected between 0dB and 15dB.

[0090] The background music is superimposed and changed to simulate the situation where the fake voice is disturbed by music. , randomly select a piece of audio from the MUSIC subset of the MUSAN dataset and adjust it to the same size as of the same length, then added to The signal-to-noise ratio is randomly selected between 5dB and 15dB.

[0091] The noisy human voice is superimposed and transformed to simulate the interference of other speakers’ real voices on the fake voice. , randomly selected from the SPEECH subset of the MUSAN dataset A voice sample. exist The signal-to-noise ratio of each selected speech sample is randomly selected between 13dB and 20dB.

[0092] (2) Taking the time-frequency features as input, a feature extraction module is used to extract deep features of the speech sample; the deep features are output to a feature classification module, and a predicted logical value of each category of the speech sample is output.

[0093] (3) Using the decision boundary distance regularization module, taking the predicted logical value as input, constructing the upper limit of the adversarial robustness of the model, and calculating the decision boundary distance regularization loss.

[0094] Constructing the upper bound of the model’s adversarial robustness, including:

[0095] First, define the random neural network model The adversarial defense boundary is

[0096] (1) Among them, For the Class relative to The adversarial defense boundary of the class is:

[0097] (2)

[0098] in, It is a voice sample The true label of It is the model obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. The weight of It is obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. A randomly transformed version of ; To implement adversarial sample attacks, use the Monte Carlo algorithm to sample the random transformation parameters; It is obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. The input is calculated as:

[0099] (3)

[0100] in, To implement adversarial sample attacks, add The adversarial perturbation on ; To solve the optimal disturbance when counteracting the disturbance, the calculation formula is:

[0101] (4)

[0102] in, and They represent the models obtained by sampling using the Monte Carlo algorithm when solving the optimal perturbation. The weights and transformation parameters of the input, Indicates the number of sampling times, Represents the loss function used by the Transformation Expectation Algorithm; the Transformation Expectation Algorithm can be used in conjunction with the gradient-based attack method to solve the optimal perturbation for a random neural network. Specifically, when using the gradient-based attack method to generate adversarial samples, in each iteration of updating the adversarial sample, the Transformation Expectation Algorithm uses the Monte Carlo algorithm to sample multiple inputs or target model weights, and uses the average of the gradients calculated based on the sampled inputs or model weights as the final gradient.

[0103] Next, based on the adversarial defense boundary described in formula (2), the model is obtained About Categories The adversarial robustness condition is:

[0104] (5)

[0105] That is, when Any Monte Carlo sampling transformation version of , All located at the border When the side is greater than 0, the model Still able to correctly identify The label of the model About Categories is robust against .

[0106] Then use Taylor's formula at point Expand the left side of formula (5) to obtain:

[0107] (6)

[0108] in, represents the Euclidean norm operator, yes The higher-order infinitesimals of represents the gradient, yes and The angle between them; Formula (6) is obtained by identity transformation:

[0109] (7)

[0110] From formula (7), we can get: It is one of the factors that affect the adversarial robustness of the model. , the first term on the left side of formula (7) is greater than 0, and the numerator of the second term on the left side is is an infinitesimal term, much smaller than In the design of adversarial samples, Relative to Therefore, the second term does not affect the overall sign of the left side, and the sign of the left side of formula (7) is determined by the sign of the first term on the left side. Therefore, the left side of formula (7) is greater than 0.

[0111] because ,distinguish Degrees, 0 degrees 90 degrees and 90 degrees The three cases of 180 degrees analyze the factors affecting the adversarial robustness of the model respectively.

[0112] ① When 90 degrees At 180 degrees, At this time, the left side of formula (7) is greater than 0, because , so the left side of formula (7) is greater than , that is, formula (7) holds, then inequality (5) holds; that is, when 90 degrees At 180 degrees, Able to correctly predict the category of adversarial samples, the model is adversarially robust.

[0113] ② When 0 degrees At 90 degrees, ; Strengthen the condition of formula (7) to:

[0114] (8)

[0115] As long as formula (8) holds, then formula (7) holds; formula (8) is transformed by identity to obtain:

[0116] (9)

[0117] ③ When Degrees, ,Right now and The direction is consistent, which includes the following special cases: is a special SNN, i.e. a fixed neural network. When the neural network is fixed, the attacker targets the same model weight parameters when solving the optimal perturbation and using the perturbation to attack, and the transformation parameters of the input are equivalent to the identity transformation. At this time, the optimal perturbation obtained is in the gradient direction of the loss function with respect to the input, that is, and The direction is consistent. At this time, formula (7) is equivalent to formula (8), and formula (9) is obtained after identity transformation.

[0118] For the second and third cases, if , formula (9) holds, then formula (7) holds, and it can also be deduced that formula (5) holds. The left side of formula (9) is defined as ,Right now:

[0119] (10)

[0120] Observe formula (10), since , Equivalent in form to a sample With Class and Decision boundary of the class The distance between The model is defined as Upper Bound on Adversarial Robustness (UBAR) in dimensionality. The larger the value, the greater the possibility that formula (9) is true. The stronger the adversarial robustness is in the dimension. Therefore, It is a factor that affects the adversarial robustness of the model. Taking into account the adversarial robustness of the model in various dimensions, the upper limit of the adversarial robustness of the model can be constructed as:

[0121] (11)

[0122] It can be seen that is the minimum upper limit of the model’s adversarial robustness in each dimension. Therefore, increasing , will help increase each , thereby increasing the possibility that formula (9) is true, which will be more conducive to enhancing the adversarial robustness of the model.

[0123] It is worth noting that for a fixed neural network, the condition for the network to be robust against adversarial forces is that formula (8) holds true, that is, the left side of formula (8) is greater than 1; while for an SNN, the condition for the network to be robust against adversarial forces is that formula (7) holds true, that is, the left side of formula (7) is greater than Since the left sides of formula (7) and formula (8) are the same, and Therefore, under the condition that other factors are the same, the adversarial robustness condition of SNN is easier to be met, that is, SNN has stronger adversarial robustness; therefore, a random transformation module is designed when randomly combining and transforming speech samples and obtaining their time-frequency features. By randomly combining and transforming the input speech samples, the network becomes SNN, thereby enhancing the adversarial robustness of the network.

[0124] Compute the decision boundary distance regularization loss, including:

[0125] Voice Sample , whose true label is , for the samples after random combination transformation, extract their logarithmic Mel spectrum as time-frequency features ,in, They are The width and height of extract The deep features ,in, They are deep features The number of channels, width and height; use the feature classification module , output A vector of predicted logical values ​​belonging to each class ,in, is the number of categories, ; Calculate the decision boundary distance regularization loss for:

[0126] (12)

[0127] in, is the threshold parameter, represents the gradient; minimize Encourage the model to learn satisfy ;in, The formula is:

[0128] (13)

[0129] in, express Belong to The predicted logical value of the class, express Belong to The predicted logical value of the class, Represents the logical value vector right gradient.

[0130] By minimizing the decision boundary distance regularization loss, it is beneficial to maximize the difference between the logical value of the correct category and the logical value of other categories, minimize the gradient of the difference with respect to the input, reduce the sensitivity of the model output logical value to adversarial perturbations, and improve adversarial robustness.

[0131] (4) Based on the predicted logical value, obtain the predicted label of the sample and calculate the classification loss.

[0132] Calculate classification loss, including: using Softmax cross entropy loss function or AM-Softmax loss function to calculate classification loss;

[0133] Consider a batch of samples , the number is , taking the logical value and label output by the feature classification module as input,

[0134] When the Softmax cross entropy loss function is used to calculate the classification loss, the formula is:

[0135] (14)

[0136] in, Indicates samples, express Belong to the categorical logical value of the class, Representation sample The corresponding one-hot encoded vector, Represents logarithmic operation;

[0137] When the AM-Softmax loss function is used to calculate the classification loss, the calculation formula is as follows: (14). The only difference is the calculation method of the logical value. The logical value used by the AM-Softmax loss function is:

[0138] (14-1)

[0139] in, The samples are The deep features and classification modules of Class classifier weights The intra-class and inter-class similarities of is the margin imposed on the intra-class similarity, where .

[0140] It should be noted that other loss functions that can be used for sample classification can be used here to calculate the classification loss.

[0141] By minimizing the classification loss, the model's classification performance for samples can be improved.

[0142] (5) A linear combination of the decision boundary distance regularization loss and the classification loss is used as the robust adversarial defense boundary loss to supervise the training of the feature extraction module and the feature classification module, and the weight parameters of the feature extraction module and the feature classification module are obtained.

[0143] The above two loss functions are linearly combined to obtain the final robust adversarial defense boundary loss function:

[0144] (15)

[0145] in, is a parameter that balances the importance of the corresponding loss.

[0146] Use the linear combination of the above two losses to jointly supervise the training of the feature extraction module and the classification module. Train for T rounds in total. Update the weight parameters of the feature extraction module and the classification module according to the gradient back propagation algorithm in each round, and calculate the classification accuracy on the validation set. After the training is completed, use the model with the highest classification accuracy on the validation set as the final model. Evaluate the performance of the final model on the evaluation set and output the predicted label of the sample.

[0147] In an embodiment of the present invention, robust adversarial defense boundary-based speech forgery attribution (RADB-FSA) is implemented using two modules: random transform (RT) and decision boundary distance regularization (DBDR). The RT module simulates the interference that forged speech may be subject to in real-world scenarios. During training and inference, it performs random combination transformations on the input speech, using randomness to improve adversarial robustness. The DBDR module introduces a decision boundary distance regularization loss function to encourage the model to improve the upper limit of adversarial robustness and reduce the sensitivity of the model's category prediction to adversarial perturbations. This method can not only achieve higher forgery method attribution performance without adversarial perturbations, but also significantly improve the model's forgery method attribution performance for forged speech disguised by adversarial perturbations.

[0148] In combination with a specific implementation of an embodiment of the present invention, a method flow for implementing voice forgery attribution using a forged voice attribution method based on a robust adversarial defense boundary is introduced below on the CFAD2023 dataset. Figure 4 As shown, this embodiment relies on the CFAD2023 dataset to carry out forged speech attribution based on robust adversarial defense boundaries. The following steps may be included:

[0149] S401, dividing the speech data into training samples, verification samples and test samples.

[0150] S402, using a random transformation module to perform random combination transformation on the speech samples, and using short-time Fourier transform, Mel filtering and logarithmic operation to convert the randomly transformed speech data into a logarithmic Mel spectrum.

[0151] S403, input the logarithmic MelP into the feature extraction module ResNet-50, and output the deep features.

[0152] S404, input the deep features into the classification module and output the category logic value.

[0153] S405, using the category logic value, input sample and sample label, calculate the decision boundary distance regularization loss .

[0154] S406, using the category logic value and sample label, using Softmax cross entropy loss to calculate the classification loss .

[0155] S407, combine the above two losses to obtain the final robust adversarial defense boundary loss .

[0156] S408, use Train the feature extraction module and the classification module.

[0157] S409, using the trained feature extraction module and classification module, identifying the data disguised by the adversarial disturbance.

[0158] 2023 Chinese Fake Audio Detection (CFAD2023) has three subsets: training set, validation set, and evaluation set, which contain 9 forgery methods, defined as F00 to F08, where real speech (denoted by F00) is considered a special category. The remaining 8 categories are speech synthesis methods based on traditional vocoders or neural vocoders, including commonly used vocoders such as Hifi-GAN, WaveNet, and STRAIGHT. The training set, validation set, and evaluation set all contain these 9 categories. In the training set, each category contains 3,200 speech samples. In the validation set, each category contains 1,200 speech samples. In the evaluation set, each category contains 3,500 speech samples.

[0159] The Adam optimizer and batch size of 128 were used for training for 50 rounds. The initial learning rate was set to 2e-4 and the weight decay was 1e-5. The learning rate was reduced to 0.2 times the original value every 10 rounds. The weight parameter in formula (5) is and threshold parameters Set them to 0.1, 100 respectively.

[0160] Calculations are performed to compare RADB with current typical defense methods, and the results are listed in Table 1.

[0161] Table 1 Comparison of RADB and other methods (%)

[0162]

[0163] In Table 1, the leftmost column indicates the attack methods used. Specifically, "clean" means no attack. Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), Basic Iterative Method (BIM), and Jitter are gradient-based white-box adversarial sample attack methods. CW is an optimization-based white-box adversarial sample attack method. EOT-FGSM / BIM / PGD / Jitter is an iterative version of FGSM / BIM / PGD / Jitter. The adversarial sample attack method based on EOT can enhance the attack performance of the attack method against random neural networks. Random neural networks refer to neural network models that introduce randomness in the input or the network itself. The network enhances the adversarial robustness of the network by introducing randomness. Among them, EOT represents the Expectation Over Transformation (EOT) algorithm. This algorithm can be used in conjunction with gradient-based adversarial sample attack methods. Specifically, when using the above-mentioned gradient-based attack methods (such as PGD) to generate adversarial samples, EOT uses the Monte Carlo algorithm to sample multiple inputs or target model weights in each iteration of updating the adversarial sample, and uses the average of the gradients calculated based on the sampled inputs or model weights as the final gradient. One-Pixel and Square are black-box attack methods, where One-Pixel refers to a single-pixel attack method that generates adversarial samples by adding perturbations to a single data point in the input. Square is a query-based attack method that attacks by adding perturbations to a local square area.

[0164] The line starting with "Vanilla" indicates a defense method. Specifically, "Vanilla" means that no defense method is used. "AT" indicates a defense method based on adversarial training, which uses adversarial samples generated by PGD for adversarial training. DIO is the method described in the article "Robust Neural Networks via Orthogonal Decomposition" published in the Journal of Pattern Recognition in 2024; UniG is the method described in the article "Robust Neural Networks via Orthogonal Decomposition" published in the ACM on Intelligent Systems and Technology in 2023. The method recorded in the article "Statistical Gradients for Improving the Robustness of Deep Networks in the Real World" in the Transactions journal; WCA is the method recorded in the article "Weight Covariance Alignment for Adversarially Robust Neural Networks" published at the International Conference on Machine Learning in 2021; DTDD is the method recorded in the article "Defense through Diversified Directions" published at the International Conference on Machine Learning in 2020; L2P is the method recorded in the article "Learn2Perturb: An End-to-End Feature Perturbation Learning for Improving Adversarial Robustness" published at the International Conference on Computer Vision and Pattern Recognition in 2020; PNI is the method recorded in the article "PNI: Trainable Randomness for Improving the Robustness of Neural Networks against Adversarial Attacks" published at the International Conference on Computer Vision and Pattern Recognition in 2019; IGR is the method recorded in the article "Improving Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Input Gradients" published at the AAAI Conference on Artificial Intelligence in 2018; RADB is the method of the present invention.

[0165] As can be seen from Table 1, compared with all baseline defense methods, the proposed method achieves higher accuracy on both clean and adversarial samples. When EOT is not used, under the strongest white-box attack (PGD), compared with the most effective baseline method WCA, the proposed method can improve the accuracy of the model from 55.78% to 65.13%. Under the condition of One-Pixel black-box attack, compared with the most effective baseline method WCA, the proposed method can improve the accuracy of the model from 88.35% to 93.98%. When EOT is used, under the strongest white-box attack (EOT-PGD), the corresponding accuracy of the proposed method is still 7.55% higher than that of WCA. These research results show that the proposed method is not only conducive to improving the performance of the model on clean samples, but also conducive to improving the forgery method attribution performance of the model in the presence of adversarial sample attacks.

[0166] from Figure 2It can be seen that in the embedded feature representations of samples learned by the WCA method, which performs the most advanced among other methods, some categories are close to each other (such as F06 and F08) or even overlap (such as F00 and F02, F04 and F07). Compared with the representation of WCA, the performance of the embedded feature representation learned by RADB is better: speech samples of different categories can be easily distinguished in the feature representation space. This shows that the features of the forgery method learned by RADB are more distinguishable between classes and more compact within the class. This means that compared with WCA, in order to confuse the model's judgment of the feature representation learned by RADB and push the feature representation of one category learned by the model into the cluster of another category, the attacker generally needs to use a larger perturbation, so RADB has stronger adversarial robustness.

[0167] from Figure 3 It can be seen that compared with other methods, the RADB method significantly improves the upper limit of the model's adversarial robustness.

[0168] Embodiment 2

[0169] Based on the forged speech attribution method based on robust adversarial defense boundary described in Embodiment 1, this embodiment provides a forged speech attribution system based on robust adversarial defense boundary, including:

[0170] A data collection module, used for collecting voice samples;

[0171] An attribution module, used to input the voice sample into a trained forged voice attribution model, and output the probability that the voice sample belongs to various voice forgery methods as a recognition result of the forged voice attribution;

[0172] The forged voice attribution model includes:

[0173] A random transformation module is used to perform random combination transformation on speech samples and obtain their time-frequency features;

[0174] A feature extraction module, used to extract deep features of speech samples from the time-frequency features;

[0175] The feature classification module is used to classify the deep features and output the probability that the voice sample belongs to each type of voice forgery method.

[0176] Embodiment 3

[0177] Based on the forged speech attribution method based on robust adversarial defense boundary described in Example 1, this embodiment provides a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the forged speech attribution method based on robust adversarial defense boundary as described in Example 1 is implemented.

[0178] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A method for attributing fake speech, characterized in that: include: Collect voice samples; Inputting the voice sample into the trained forged voice attribution model, and outputting the probability that the voice sample belongs to each type of voice forgery method as the recognition result of the forged voice attribution; The forged voice attribution model includes: A random transformation module is used to perform random combination transformation on speech samples and obtain their time-frequency features; A feature extraction module, used to extract deep features of speech samples from the time-frequency features; The feature classification module is used to classify the deep features and output the probability that the voice sample belongs to each type of voice forgery method.

2. The method for attributing fake speech according to claim 1, characterized in that: The training method of the forged speech attribution model comprises: Input the speech samples in the training set into the random transformation module to obtain the time-frequency features of the speech samples after random combination transformation; Taking the time-frequency features as input, a feature extraction module is used to extract the deep features of the speech sample; the deep features are output to a feature classification module, and the predicted logical values ​​of the speech sample belonging to each category are output; Using a decision boundary distance regularization module, taking the predicted logical value as input, constructing an upper limit of the adversarial robustness of the model, and calculating a decision boundary distance regularization loss; According to the predicted logical value, the predicted label of the sample is obtained, and the classification loss is calculated; The linear combination of decision boundary distance regularization loss and classification loss is used as the robust adversarial defense boundary loss function to supervise the training of the feature extraction module and the feature classification module, and the weight parameters of the feature extraction module and the feature classification module are obtained.

3. The method for attributing fake speech according to claim 2, characterized in that: Perform random combination transformation on speech samples and obtain their time-frequency features, including: Based on the set n transformation types, randomly generate the number of quasi-sampling transformations , n represents the number of set transformation types; For the number of transformations obtained by sampling, Q, Secondary transformation , the type of randomly generated transformation , the probability of using this transformation and the intensity of the transformation ; Among them, for the type of transformation , different Corresponding to different transformation types, randomly select from the set n transformation types; use transformation Probability follows a Bernoulli distribution with parameter p, i.e. using the transformation The probability is p; using the transformation Strength follows a uniform distribution with given parameters; Apply Q transformations to the input speech sample in sequence to obtain a sample that has been randomly combined and transformed. : （1-1） in, represents an arrangement scheme of the Q transformations; each transformation is a three-element random parameter group, consisting of These three random variables are used to describe; Indicates that the parameter is The transformation function of Use short-time Fourier transform, Mel filtering and logarithmic operation to convert the randomly combined transformed speech samples into logarithmic Mel spectrum, that is, the time-frequency characteristics of the randomly combined transformed speech samples , expressed as ,in, represents the random transformation module, represents the short-time Fourier transform, Mel filtering and logarithmic operation.

4. The method for attributing fake speech according to claim 3, characterized in that: The set n types of transformations include: reverberation effect transformation, environmental noise superposition transformation, background music superposition transformation and noisy human voice superposition transformation.

5. The method for attributing fake speech according to claim 3, characterized in that: Constructing the upper bound of the model’s adversarial robustness, including: First, define the random neural network model The adversarial defense boundary is: (1) in, For the Class relative to The adversarial defense boundary of the class is: (2) in, It is a voice sample The true label of It is the model obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. The weight of It is obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. A randomly transformed version of ; The random transformation parameters sampled by the Monte Carlo algorithm are used when implementing adversarial sample attacks; It is obtained by sampling using the Monte Carlo algorithm when implementing adversarial sample attacks. The input is calculated as: (3) in, To implement adversarial sample attacks, add The adversarial perturbation on ; To solve the optimal disturbance when counteracting the disturbance, the calculation formula is: (4) in, and They represent the models obtained by sampling using the Monte Carlo algorithm when solving the optimal perturbation. The weights and transformation parameters of the input, Indicates the number of sampling times, Represents the loss function used by the Transformation Expectation Algorithm; the Transformation Expectation Algorithm is used in combination with the gradient-based attack method to solve the optimal perturbation for the random neural network; when the gradient-based attack method is used to generate adversarial samples, in each iteration of updating the adversarial sample, the Transformation Expectation Algorithm uses the Monte Carlo algorithm to sample multiple inputs or target model weights, and uses the average of the gradients calculated based on the sampled inputs or model weights as the final gradient; Next, based on the adversarial defense boundary described in formula (2), the model is obtained About Categories The adversarial robustness condition is: (5) That is, when An arbitrary Monte Carlo sampling transformation version of , Located at the border When the side is greater than 0, the model Still able to correctly identify The label of the model About Categories is robust against adversarial forces; Then use Taylor's formula at point Expand the left side of formula (5) to obtain: (6) in, represents the Euclidean norm operator, yes The higher-order infinitesimals of represents the gradient, yes and The angle between them; Formula (6) is obtained by identity transformation: (7) in, It is one of the factors that affect the model's adversarial robustness; It is a voice sample The true label, therefore, , the first term on the left side of formula (7) is greater than 0, and the numerator of the second term on the left side is is an infinitesimal term, much smaller than ; In the design of adversarial samples, Relative to is very small; therefore, the second term does not affect the overall sign of the left side, and the sign of the left side of formula (7) is determined by the sign of the first term on the left side; therefore, the left side of formula (7) is greater than 0; because ,distinguish Degrees, 0 degrees 90 degrees and 90 degrees 180 degrees, three situations, respectively analyzing the factors that affect the model's adversarial robustness; ① When 90 degrees At 180 degrees, ; At this time, the left side of formula (7) is greater than 0, because , so the left side of formula (7) is greater than , that is, formula (7) holds, then inequality (5) holds; that is, when 90 degrees At 180 degrees, Able to correctly predict the category of adversarial samples, the model is adversarially robust; ② When 0 degrees At 90 degrees, ; Strengthen the condition of formula (7) to: （8） As long as formula (8) holds, then formula (7) holds; formula (8) is transformed by identity to obtain: （9） ③ When Degrees, ,Right now and The direction is consistent, which includes the following special cases: is a special SNN, i.e. a fixed neural network; when When the neural network is fixed, the attacker targets the same model weight parameters when solving the optimal perturbation and using the perturbation to attack, and the transformation parameters of the input are equivalent to the identity transformation; at this time, the optimal perturbation obtained is in the gradient direction of the loss function with respect to the input, that is, and The direction is consistent. ; At this time, formula (7) is equivalent to formula (8), and after identity transformation, formula (9) is obtained; For the second and third cases, if , formula (9) holds, then formula (7) holds, and similarly formula (5) holds; the left side of formula (9) is defined as ,Right now: （10） because , In form it is equivalent to a sample With Class and Decision boundary of the class The distance between The model is defined as The upper bound of adversarial robustness in dimensionality; The larger the value, the greater the possibility that formula (9) is true. The stronger the adversarial robustness in the dimension, the stronger the It is a factor that affects the adversarial robustness of the model. Taking into account the adversarial robustness of the model in various dimensions, the upper limit of the adversarial robustness of the model is constructed as: （11） is the minimum upper limit of the model’s adversarial robustness in each dimension; therefore, increasing , will help increase each , thus increasing the possibility that formula (9) is true, which is beneficial to enhancing the adversarial robustness of the model; For a fixed neural network, the condition for the network to be robust against adversarial forces is that formula (8) holds true, that is, the left side of formula (8) is greater than 1; while for an SNN, the condition for the network to be robust against adversarial forces is that formula (7) holds true, that is, the left side of formula (7) is greater than ; Since the left sides of formula (7) and formula (8) are the same, and ,Therefore, under the condition that other factors are the same, the adversarial robustness condition of SNN is easier to be met, that is, SNN has stronger adversarial robustness.

6. The method for attributing fake speech according to claim 3, characterized in that: Compute the decision boundary distance regularization loss, including: For the voice sample , whose true label is , the time-frequency characteristics of the sample after random combination transformation are ,in, They are The width and height of extract The deep features ,in, They are deep features The number of channels, width and height; use the feature classification module , output A vector of predicted logical values ​​belonging to each class ,in, is the number of categories, ; Based on the adversarial robustness upper limit of the constructed model, calculate the decision boundary distance regularization loss for: （12） in, is the threshold parameter, represents the gradient; minimize Encourage the model to learn satisfy ;in, The formula is: （13） in, express Belong to The predicted logical value of the class, express Belong to The predicted logical value of the class, Represents the logical value vector right gradient.

7. The method for attributing fake speech according to claim 5, characterized in that: Calculate classification loss, including: using Softmax cross entropy loss function or AM-Softmax loss function to calculate classification loss; Consider a batch of samples , the number is , taking the logical value and label output by the feature classification module as input, When the Softmax cross entropy loss function is used to calculate the classification loss, the formula is: （14） in, Indicates samples, express Belong to the categorical logical value of the class, Representation sample The corresponding one-hot encoded vector, Represents logarithmic operation; When the AM-Softmax loss function is used to calculate the classification loss, the calculation formula is as follows: (14). The only difference is the calculation method of the logical value. The logical value used by the AM-Softmax loss function is: （14-1） in, The samples are The deep features and classification modules of Class classifier weights The intra-class and inter-class similarities of is the margin imposed on the intra-class similarity, where .

8. The method for attributing fake speech according to claim 6, characterized in that: The robust adversarial defense boundary loss function is: （15） in, is a parameter that balances the importance of the corresponding loss.

9. The method for attributing fake speech according to claim 7, characterized in that: Extract deep features of speech samples, including: Taking the time-frequency features as input, ResNet-50 is used to extract the deep features of the speech samples; ResNet-50 has four convolution blocks, and the output of the last convolution block is used as the deep features.

10. A forged speech attribution system, characterized in that: include: A data collection module, used for collecting voice samples; An attribution module, used to input the voice sample into a trained forged voice attribution model, and output the probability that the voice sample belongs to various voice forgery methods as a recognition result of the forged voice attribution; The forged voice attribution model includes: A random transformation module is used to perform random combination transformation on speech samples and obtain their time-frequency features; A feature extraction module, used to extract deep features of speech samples from the time-frequency features; The feature classification module is used to classify the deep features and output the probability that the voice sample belongs to each type of voice forgery method.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the forged speech attribution method as described in any one of claims 1 to 9 is implemented.

Images