Highly concealed adversarial speech sample generation method based on universal adversarial perturbation generation network

Abstract

The present invention discloses a highly concealed adversarial speech sample generation method based on a universal adversarial perturbation generation network, comprising the following steps: 1. processing audio data to obtain audio sample values ​​and setting label values ​​according to file names; 2. constructing a universal adversarial perturbation generation network and designing the network structure of a speaker recognition module, an adversarial sample detection module, and a universal adversarial perturbation generation module; 3. using the weighted sum of the loss function of the speaker recognition module, the loss function of the adversarial sample detection module, and the loss function of the universal adversarial perturbation generation module as the loss function; and 4. inputting the audio data into the universal adversarial perturbation generation network and training it, adjusting the network parameters, and obtaining an optimal model. The present invention can achieve a high attack success rate for both non-targeted and targeted attacks on the speaker recognition model, achieve real-time and rapid deployment while generating high-quality adversarial sample audio and high concealment, and improve the robustness of the speaker recognition model.

Description

Technical Field

[0001] The present invention relates to the technical field of universal adversarial audio sample generation, and in particular to a highly concealed adversarial speech sample generation method based on a universal adversarial perturbation generation network. Background Art

[0002] Speaker recognition models are powerful audio analysis technologies that extract unique speaker characteristics from speech signals and subsequently identify the speaker. This technology holds broad application prospects in security monitoring, identity verification, human-computer interaction, and other fields. However, as speaker recognition technology continues to advance, its security also faces increasingly severe challenges.

[0003] Adversarial attacks against speaker recognition models have become a key area of ​​security research. Adversarial attacks typically involve adding carefully crafted, subtle perturbations to the original speech signal, making the changes almost imperceptible to the auditory sense, yet capable of misleading the speaker recognition model into producing erroneous results. This attack method is crucial for protecting personal privacy and preventing identity forgery.

[0004] With the widespread adoption of deep learning-based speaker recognition models, they have demonstrated powerful capabilities in processing complex speech signals and extracting deep features. However, these deep learning models also face the threat of adversarial attacks. Some researchers have successfully attacked deep learning speaker recognition models using carefully designed adversarial examples, significantly reducing their recognition accuracy.

[0005] Traditional adversarial attacks differ significantly from general-purpose adversarial attacks, not only in their attack mechanisms but also in their application scenarios and effectiveness. Traditional adversarial attacks typically perturb specific input data. Deploying these attacks requires significant time and computing resources to generate perturbations, making them impractical for rapid deployment. However, their advantage lies in the high quality of the adversarial examples they generate.

[0006] In contrast, universal adversarial attacks have stronger generalization capabilities. They typically design a universal perturbation for all input data. Independent of specific data, they can be quickly deployed in real-world scenarios while maintaining a certain level of effectiveness. Therefore, universal adversarial attacks are more suitable for real-time deployment. However, their disadvantage is that the quality of the generated adversarial examples is inferior to that of traditional adversarial attack methods.

[0007] Therefore, research on adversarial attacks against speaker recognition models is crucial for improving the security and reliability of speaker recognition technology. However, most current technologies only guarantee the success rate of general adversarial attacks, without considering the concealment of perturbations and achieving higher adversarial audio quality. Summary of the Invention

[0008] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a method for generating highly concealed adversarial speech samples based on a universal adversarial perturbation generation network, in order to obtain high-quality and highly concealed universal adversarial audio, thereby realizing real-time deployment of adversarial attacks.

[0009] The present invention adopts the following scheme to solve the technical problem:

[0010] The invention provides a method for generating highly concealed adversarial speech samples based on a universal adversarial perturbation generation network, which comprises the following steps:

[0011] Step 1: Sample the first Frame Audio ={ };in, Indicates the Frame Audio Middle The audio value of the sampling point; for The total number of sampling points in ;

[0012] Generate audio using the UAP-GN generation method Adversarial Audio ={ };in, express The adversarial audio The speaker ID is , , where N is the total number of speakers; let The discriminant label is 1, The discriminant label is 0;

[0013] Step 2: Construct a universal adversarial perturbation generation network, including: speaker recognition module R, adversarial sample detection module D, and universal adversarial perturbation generation module G;

[0014] Step 2.1: The speaker recognition module R uses the Sinc-Net network to Process and output Frame Audio The probability vector of the speaker prediction label ;

[0015] based on and Construct a cross entropy loss function and use the RMSProp optimization algorithm to pre-train the speaker recognition module R to obtain the pre-trained speaker recognition module ;

[0016] Step 2.2: The adversarial sample detection module D uses the CNN network to and Process them separately to get a one-dimensional audio discriminant matrix and one-dimensional adversarial audio discriminant matrix ;

[0017] based on The discriminant labels and Construct binary cross entropy based on The discriminant labels and Construct binary cross entropy and use the Adam optimization algorithm to pre-train the adversarial sample detection module D to obtain the trained adversarial sample detection module ;

[0018] Step 2.3: Generate general adversarial perturbation module G The corresponding one-dimensional standard Gaussian noise Processing is performed to obtain a one-dimensional universal adversarial perturbation ;

[0019] Step 3: Establish the optimization objective of the general adversarial perturbation generation network;

[0020] Formula (1) and Formula (2) are used to establish the non-targeted attack optimization objective and the targeted attack optimization objective of the general adversarial perturbation generation module G respectively;

[0021] ) (1)

[0022] ) (2)

[0023] In formula (1) and formula (2), is the loss function of the universal adversarial perturbation generation module G; where U is the speaker label expected by the pre-set speaker recognition module R;

[0024] Step 4: Use the Adam method to train the general adversarial perturbation generation network and calculate the loss function The network parameters are updated until the maximum number of training times is reached, and then the training is stopped to obtain the optimal universal adversarial perturbation generation network for identifying the speaker.

[0025] The highly concealed adversarial speech sample generation method based on a universal adversarial perturbation generation network described in the present invention is also characterized in that the CNN network in step 2.2 includes: Layer 1D convolution and max pooling layers, and Layer fully connected layer;

[0026] Pass through After processing the 1D convolution layer and the maximum pooling layer, the audio feature map is obtained , and flattened into a one-dimensional audio vector, and then input The audio is processed in the fully connected layer and the activation function to output a one-dimensional audio discriminant matrix. ;in, The first eigenvalues, is the total number of features;

[0027] Pass through After processing the 1D convolution and maximum pooling layers, we get the adversarial audio feature map , and flattened into a one-dimensional adversarial audio vector, and then input in sequence The audio is processed in the fully connected layer and the activation function to output a one-dimensional adversarial audio discriminant matrix. ;in, To counter the audio feature map eigenvalues.

[0028] The adversarial perturbation generation module G in step 2.3 includes: a fully connected layer, an upsampling layer, and a 1D convolution;

[0029] The fully connected layer will After being converted into a two-dimensional tensor, the audio feature map with doubled dimension is obtained through the upsampling layer. ,in, The first The first channel The eigenvalue of dimension, is the total number of feature dimensions, , is the total number of channels;

[0030] like With the ideal output dimension If different, the audio feature map is adjusted using interpolation The dimension of the audio feature map after dimension adjustment is input into the 1D convolution layer for processing, and a one-dimensional universal adversarial perturbation is obtained. ;

[0031] like With the ideal output dimension If they are the same, then directly Input into the 1D convolution layer for processing and obtain a one-dimensional universal adversarial perturbation .

[0032] In step 3, the loss function of the general adversarial perturbation generation module G is obtained using formula (3): ;

[0033] (3)

[0034] In formula (3), is to limit the universal adversarial perturbation The loss function of size is obtained by formula (6); is the loss function that causes the speaker recognition model to produce misclassification. If the attack method is specified as non-target attack, then the formula (7) is used to obtain , if the attack method is targeted attack, then replace the formula (7) Set as Afterwards ; is the loss function for deceiving the adversarial sample detection module D, which is obtained by formula (8); 、 、 They are the trade-off loss functions The three weight coefficients of

[0035] (4)

[0036] (5)

[0037] (6)

[0038] (7)

[0039] (8)

[0040] In formula (4) to formula (8), Indicates that The probability vector output after inputting the speaker recognition module R can be obtained by formula (4), and we set The maximum probability value is , and the index corresponding to the maximum probability value is used as The predicted label , set the second largest probability value to ,make The probability value of the speaker ID dimension is ; Indicates that The probability vector output after inputting the adversarial sample detection module D is obtained by formula (5); let The probability value of index 0 is , The probability value of index 1 is ; It is a fixed value used to adjust the size of the loss function; are scaling factors; is the threshold that limits the size of the disturbance; Represents Iverson brackets. If Iverson meets the conditions in the brackets, then let The value of is 1, otherwise, The value of is 0.

[0041] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the highly concealed adversarial voice sample generation, and the processor is configured to execute the program stored in the memory.

[0042] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of generating highly concealed adversarial speech samples when executed by a processor.

[0043] Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0044] 1. The network structure designed by this invention consists of three interconnected network modules. By fixing two modules and training one of them, a more efficient training process is achieved. This training method helps generate more subtle universal perturbations because the adversarial sample detection module is incorporated into the network during training, which helps to better disguise the perturbations and make them more difficult to detect. Furthermore, by optimizing the training process, this invention improves the quality of adversarial samples, making the generated adversarial samples more stable and reliable, providing a new solution for improving the robustness of speaker recognition models.

[0045] 2. This invention generates universal adversarial perturbations through a universal adversarial perturbation generation network, significantly reducing the time required to deploy perturbations in traditional adversarial attacks. This instantaneous generation of universal adversarial perturbations not only improves attack efficiency but also increases the feasibility of implementing adversarial attacks in practical application scenarios.

[0046] 3. To further enhance the stealth and diversity of universal adversarial perturbations, this paper uses randomly sampled Gaussian noise as input to generate different universal perturbations. This design makes the generated universal perturbations more diverse in form and characteristics, making them difficult for the target model to identify and defend against. Furthermore, the randomness of Gaussian noise also ensures the stealth of universal perturbations, making attacks more difficult to track and defend against.

[0047] 4. The present invention successfully implemented two specific tasks: untargeted and targeted attacks, and both achieved good attack results in experiments. Untargeted attacks aim to reduce the classification accuracy of the target model, while targeted attacks attempt to cause the target model to produce specific misclassifications. Both attack methods are of great significance in practical applications: untargeted attacks can be used to reduce model performance, while targeted attacks can be used to achieve specific attack objectives. Through the implementation of these two attack methods, the present invention demonstrates its broad application potential and value in practical adversarial attack scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Flowchart of the present invention;

[0049] Figure 2 This is the overall framework diagram of the universal anti-perturbation generation network of the present invention;

[0050] Figure 3 This is a diagram of the adversarial sample detection network structure of the method of the present invention. DETAILED DESCRIPTION

[0051] In this embodiment, for the security-sensitive speaker recognition system, studying adversarial attacks can help us discover potential security vulnerabilities in advance and take corresponding measures to ensure the security of the system. A highly concealed adversarial speech sample generation method based on a universal adversarial perturbation generation network is proposed. The specific process is as follows: Figure 1 As shown, the following steps are included:

[0052] Step 1: Use the public dataset TIMIT and process this dataset. Each audio is sampled at a sampling rate of 16000. In this example, Set to 3200, according to To divide the sampled audio, thus obtaining audio frames, sampling the first Frame Audio ={ };in, Indicates the Frame Audio Middle The audio value of the sampling point; for The total number of sampling points in the dataset is used to generate adversarial audio using the UAP-GN generation method. After the same processing steps as the TIMIT dataset, adversarial audio frames are obtained.

[0053] Generate audio using the UAP-GN generation method Adversarial Audio ={ };in, express The adversarial audio The speaker ID is , , where N is the total number of speakers, which is 462 in this example; The discriminant label is 1, The discriminant label is 0, the discriminant label is 1, which means the sample is a real sample, and the discriminant label is 0, which means the sample is a generated adversarial sample, which is a fake sample;

[0054] Step 2: Construct a universal adversarial perturbation generation network, including: speaker recognition module R, adversarial sample detection module D and universal adversarial perturbation generation module G. The overall architecture is shown in the figure below. Figure 2 shown.

[0055] Step 2.1: Speaker recognition module R uses Sinc-Net network to Process and output Frame Audio The probability vector of the speaker prediction label ;

[0056] based on and Construct a cross entropy loss function and use the RMSProp optimization algorithm to pre-train the speaker recognition module R to obtain the pre-trained speaker recognition module .

[0057] Step 2.2: The network structure of module D is as follows Figure 3 As shown, the adversarial sample detection module D uses the CNN network to and Process them separately to get a one-dimensional audio discriminant matrix ;

[0058] In specific implementation, the CNN network includes: Layer 1D convolution and max pooling layers, and Layer fully connected layer;

[0059] Pass through After the processing of the 1D convolution layer and the maximum pooling layer, the pooling layer window size is 2, and the audio feature map is obtained , and flattened into a one-dimensional audio vector, and then input The fully connected layer and the activation function are processed. The final output of the fully connected layer is 2D. The Softmax activation function is used to output the probability distribution and output the one-dimensional audio discriminant matrix. , The value of index 0 in represents the probability that the sample is an adversarial sample, and the value of index 1 represents the probability that the sample is a real sample; The first eigenvalues, is the total number of features;

[0060] Pass through After processing the 1D convolution and maximum pooling layers, we get the adversarial audio feature map , and flattened into a one-dimensional adversarial audio vector, and then input in sequence The audio is processed in the fully connected layer and the activation function to output a one-dimensional adversarial audio discriminant matrix. , The value of index 0 in represents the probability that the sample is an adversarial sample, and the value of index 1 represents the probability that the sample is a real sample; To counter the audio feature map eigenvalues;

[0061] based on The discriminant labels and Construct binary cross entropy based on The discriminant labels and Construct the binary cross entropy and use the Adam optimization algorithm to pre-train the adversarial sample detection module D to obtain the trained adversarial sample detection module .

[0062] Step 2.3: Generate general adversarial perturbation module G The corresponding one-dimensional standard Gaussian noise To process, The dimension is 100, and we get a one-dimensional universal adversarial perturbation , The dimension is , the value is 3200, which is consistent with the input of the speaker recognition module R and the adversarial sample detection module D.

[0063] In the specific implementation, the adversarial perturbation generation module G includes: fully connected layers, upsampling layers and 1D convolution, and adopts the UAP-GN generation network;

[0064] The fully connected layer will After being converted into a two-dimensional tensor, the audio feature map with doubled dimension is obtained through the upsampling layer. ,in, The first The first channel The eigenvalue of dimension, is the total number of feature dimensions, , is the total number of channels. In this example, Set to 64.

[0065] like With the ideal output dimension If different, the audio feature map is adjusted using interpolation The dimension of the audio feature map after dimension adjustment is input into the 1D convolution layer for processing, and a one-dimensional universal adversarial perturbation is obtained. ,and ;

[0066] like With the ideal output dimension If they are the same, then directly Input into the 1D convolution layer for processing and obtain a one-dimensional universal adversarial perturbation .

[0067] Step 3: Establish the optimization objective of the general adversarial perturbation generation network;

[0068] Step 3.1: Use equations (1) and (2) to establish the non-targeted attack optimization objective and the targeted attack optimization objective of the general adversarial perturbation generation module G respectively;

[0069] ) (1)

[0070] ) (2)

[0071] In Equations (1) and (2), the universal adversarial perturbation generation network has three modules, so the loss function of the universal adversarial perturbation generation module G is the weighted sum of the losses of the three modules, is the loss function of the universal adversarial perturbation generation module G and is obtained by formula (3); where U is the speaker label expected by the preset speaker recognition module R, and U can be set to 0, 100, 200, etc.

[0072] (3)

[0073] In formula (3), is to limit the universal adversarial perturbation The loss function of size is obtained by formula (6); is the loss function that causes the speaker recognition model to produce misclassification. If the attack method is specified as non-target attack, then the formula (7) is used to obtain , if the attack method is targeted attack, then replace the formula (7) Set as Afterwards ; is the loss function of the deceptive adversarial sample detection module D, which is obtained by formula (8); 、 、 They are the trade-off loss functions The three weight coefficients of The value is usually set to 2000 or 3000. Generally, it is set to 1;

[0074] (4)

[0075] (5)

[0076] (6)

[0077] (7)

[0078] (8)

[0079] In formula (4) to formula (8), Indicates that The probability vector output after inputting the speaker recognition module R can be obtained by formula (4), and we set The maximum probability value is , and the index corresponding to the maximum probability value is used as The predicted label , set the second largest probability value to ,make The probability value of the speaker ID dimension is ; Indicates that The probability vector output after inputting the adversarial sample detection module D is obtained by formula (5); let The probability value of index 0 is , The probability value of index 1 is ; It is a fixed value used to adjust the size of the loss function; are scaling factors; Is the threshold that limits the size of the disturbance. In this example, Set to 0.05, Set to 0.01, Set to 0.1; Represents Iverson brackets. If Iverson meets the conditions in the brackets, then let The value of is 1, otherwise, The value of is 0.

[0080] Step 4: Use the Adam method to train the general adversarial perturbation generation network and calculate the loss function The network parameters are updated until the maximum number of training times is reached, and then the training is stopped to obtain the optimal universal adversarial perturbation generation network for identifying the speaker.

[0081] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0082] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

Claims

1. A highly concealed adversarial speech sample generation method based on a universal adversarial perturbation generation network, characterized in that: The steps include: Step 1: Sample the first Frame Audio ={ };in, Indicates the Frame Audio Middle The audio value of the sampling point; for The total number of sampling points in ; Generate audio using the UAP-GN generation method Adversarial Audio ={ };in, express The adversarial audio The speaker ID is , , where N is the total number of speakers; let The discriminant label is 1, The discriminant label is 0; Step 2: Construct a universal adversarial perturbation generation network, including: speaker recognition module R, adversarial sample detection module D, and universal adversarial perturbation generation module G; Step 2.1: The speaker recognition module R uses the Sinc-Net network to Process and output Frame Audio The probability vector of the speaker prediction label ; based on and Construct a cross entropy loss function and use the RMSProp optimization algorithm to pre-train the speaker recognition module R to obtain the pre-trained speaker recognition module ; Step 2.2: The adversarial sample detection module D uses the CNN network to and Process them separately to get a one-dimensional audio discriminant matrix and one-dimensional adversarial audio discriminant matrix ; based on The discriminant labels and Construct binary cross entropy based on The discriminant labels and Construct binary cross entropy and use the Adam optimization algorithm to pre-train the adversarial sample detection module D to obtain the trained adversarial sample detection module ; Step 2.3: Generate general adversarial perturbation module G The corresponding one-dimensional standard Gaussian noise Processing is performed to obtain a one-dimensional universal adversarial perturbation ; Step 3: Establish the optimization objective of the general adversarial perturbation generation network; Formula (1) and Formula (2) are used to establish the non-targeted attack optimization objective and the targeted attack optimization objective of the general adversarial perturbation generation module G respectively; ) (1) ) (2) In formula (1) and formula (2), is the loss function of the universal adversarial perturbation generation module G; where U is the speaker label expected by the pre-set speaker recognition module R; Step 4: Use the Adam method to train the general adversarial perturbation generation network and calculate the loss function The network parameters are updated until the maximum number of training times is reached, and then the training is stopped to obtain the optimal universal adversarial perturbation generation network for identifying the speaker.

2. The method for generating highly concealed adversarial speech samples based on a general adversarial perturbation generation network according to claim 1, characterized in that: The CNN network in step 2.2 includes: Layer 1D convolution and max pooling layers, and Layer fully connected layer; Pass through After processing the 1D convolution layer and the maximum pooling layer, the audio feature map is obtained , and flattened into a one-dimensional audio vector, and then input The audio is processed in the fully connected layer and the activation function to output a one-dimensional audio discriminant matrix. ;in, The first eigenvalues, is the total number of features; Pass through After processing the 1D convolution and maximum pooling layers, we get the adversarial audio feature map , and flattened into a one-dimensional adversarial audio vector, and then input in sequence The audio is processed in the fully connected layer and the activation function to output a one-dimensional adversarial audio discriminant matrix. ;in, To counter the audio feature map eigenvalues.

3. The method for generating highly concealed adversarial speech samples based on a general adversarial perturbation generation network according to claim 2, characterized in that: The adversarial perturbation generation module G in step 2.3 includes: a fully connected layer, an upsampling layer, and a 1D convolution; The fully connected layer will After being converted into a two-dimensional tensor, the audio feature map with doubled dimension is obtained through the upsampling layer. ,in, The first The first channel The eigenvalue of dimension, is the total number of feature dimensions, , is the total number of channels; like With the ideal output dimension If different, the audio feature map is adjusted using interpolation The dimension of the audio feature map after dimension adjustment is input into the 1D convolution layer for processing, and a one-dimensional universal adversarial perturbation is obtained. ; like With the ideal output dimension If they are the same, then directly Input into the 1D convolution layer for processing and obtain a one-dimensional universal adversarial perturbation .

4. The method for generating highly concealed adversarial speech samples based on a general adversarial perturbation generation network according to claim 3, wherein: In step 3, the loss function of the general adversarial perturbation generation module G is obtained using formula (3): ; (3) In formula (3), is to limit the universal adversarial perturbation The loss function of size is obtained by formula (6); is the loss function that causes the speaker recognition model to produce misclassification. If the attack method is specified as non-target attack, then the formula (7) is used to obtain , if the attack method is targeted attack, then replace the formula (7) Set as Afterwards ; is the loss function for deceiving the adversarial sample detection module D, which is obtained by formula (8); 、 、 They are the trade-off loss functions The three weight coefficients of (4) (5) (6) (7) (8) In formula (4) to formula (8), Indicates that The probability vector output after inputting the speaker recognition module R can be obtained by formula (4), and we set The maximum probability value is , and the index corresponding to the maximum probability value is used as The predicted label , set the second largest probability value to ,make The probability value of the speaker ID dimension is ; Indicates that The probability vector output after inputting the adversarial sample detection module D is obtained by formula (5); let The probability value of index 0 is , The probability value of index 1 is ; It is a fixed value used to adjust the size of the loss function; are scaling factors; is the threshold that limits the size of the disturbance; Represents Iverson brackets. If Iverson meets the conditions in the brackets, then let The value of is 1, otherwise, The value of is 0.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the highly concealed adversarial voice sample generation method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a highly concealed adversarial speech sample according to any one of claims 1 to 4 are executed.

Images