An adversarial sample restoration method based on deep speech waveform prior

Abstract

The application discloses a kind of dynamic confrontation sample defense methods of plug and play-LowDDAWP-Net, for the transcription of confrontation sample is recovered into benign transcription.Firstly, the confrontation voice is segmented into mute segment and speech segment by voice activity detection algorithm, then the distribution of confrontation disturbance is extracted by mute segment, the confrontation disturbance of speech segment is filtered out by reconstruction separation, so as to recover confrontation audio into benign audio, and the structure information of speech segment is extracted by AMR compression algorithm and log minimum mean square error algorithm based on speech uncertainty (Logmmse_SPU-cohen), improve the fitting speed of defense algorithm.By the application, aggressive confrontation sample can be restored into benign sample, and the parameters of defense module are dynamically updated, which improves the security performance of the speech recognition system, and provides an effective restoration method for the defense of confrontation sample.

Description

Technical Field

[0001] This invention relates to an adversarial sample reconstruction method based on deep speech waveform priors, belonging to the field of computer speech recognition technology. Background Technology

[0002] With the development of deep learning technology, researchers have begun to apply it to speech recognition. Deep learning can automatically learn the features of speech signals, significantly improving the accuracy and robustness of speech recognition. Therefore, automatic speech recognition technology has begun to be integrated into various fields, such as smart homes, smart cars, and smart robots, making people's production and lives more efficient and convenient. However, recent research shows that by adding carefully crafted adversarial perturbations to benign samples, adversarial examples can be generated that cause ASR systems to make incorrect judgments, and these examples are virtually indistinguishable to human perception. These adversarial speech examples can not only cause the victim model to misclassify, but can even cause it to transcribe the data into the attacker's desired outcome, thus posing security risks to related applications (such as voice assistants and voice verification codes).

[0003] Current research on defense against adversarial examples in speech processing mainly focuses on detection defense, with only a few studies targeting the reconstruction of adversarial examples. However, a robust ASR system should aim to correctly transcribe any sample, not just detect adversarial examples and deny service. Furthermore, existing reconstruction defenses have low performance against adversarial examples and have a significant impact on benign examples, requiring modifications to the target model. Additionally, the parameters of the defense modules proposed in existing research are basically fixed, making them vulnerable to attack once fixed parameters are obtained through gradient estimation or other methods.

[0004] This invention addresses existing problems by proposing the LowDDAWP-Net algorithm, achieving dynamic defense against adversarial examples. This method reconstructs adversarial examples based on deep speech waveform prior theory and incorporates a low-resolution information extraction module to accelerate the fitting speed of the LowDDAWP-Net network. A dual DAWP network is designed to reconstruct and separate adversarial perturbations and speech segments, improving the adversarial example reconstruction performance of the LowDDAWP-Net algorithm. This method requires no pre-training and its gradient information changes in real time, effectively increasing the difficulty of attacks. Summary of the Invention

[0005] The purpose of this invention is to provide a method for reconstructing adversarial samples based on deep speech waveform priors, which is used to restore the speech recognition content of adversarial samples into the transcription of benign samples.

[0006] To solve the above problems, the present invention adopts the following technical solution:

[0007] This invention provides a method for dynamic reconstruction of adversarial examples based on deep speech waveform priors. This method can achieve high-precision reconstruction of adversarial examples with minimal impact on the transcription results of benign examples. Taking adversarial examples as an example, the specific steps are as follows:

[0008] Step 1: Randomly read an adversarial sample and segment the sample into silent segments and speech segments using the VAD algorithm;

[0009] Step 2: The resulting silent segment, containing a large amount of noise information and adversarial perturbation information, is used as the adversarial perturbation fitting network (DAWP). noise The network uses randomly generated Gaussian noise as input to obtain fitted adversarial perturbation audio.

[0010] Specifically, DAWP noise The network structure is a U-shaped encoder and decoder. The decoder consists of six convolutional coding blocks and one single-layer bidirectional LSTM block; the encoder consists of six convolutional decoding blocks. Each convolutional coding block consists of two one-dimensional convolutional layers. The first convolutional layer has a kernel size of 8 and a stride of 4, while the second convolutional layer has a kernel size of 1 and a stride of 1. The output of each convolutional layer is activated using the ReLU function. The deepest convolutional coding block feeds its output into a single-layer bidirectional LSTM layer, and then a linear layer maps the fused high-dimensional features back to the input size. Each decoding block consists of a cascaded one-dimensional convolutional layer with a kernel size of 3 and a stride of 1, and a transposed convolutional layer with a kernel size of 8 and a stride of 4. The output of each convolutional layer is also activated by the ReLU function before being input into the next layer. The last convolutional decoding block synthesizes the final waveform signal.

[0011] DAWP noise The input is random Gaussian noise with a mean of 0 and a variance of 0.1, and the output is an estimate of the silent audio. The loss function is calculated as follows:

[0012]

[0013] Step 3: For the obtained utterance fragments, it is used as a utterance information fitting network (DAWP). speec h) The input to the tag and low-resolution information extraction module; low-resolution audio and DAWP speec The output audio of h is used to fit the speech fragment.

[0014] Specifically, DAWP speec h's network structure and input with DAWP noise The network is exactly the same, DAWP speec The loss function for h is:

[0015]

[0016] Here, `base` represents the output of the low-resolution information extraction module. This module consists of a cascade of AMR-V and Logmmse_SPU-cohen algorithms. The adversarial example is first processed by the AMR-V algorithm to filter out redundant signals and disrupt the integrity of the adversarial perturbation; then, the Logmmse_SPU-cohen algorithm is used to enhance the decompressed audio, aiming to further filter out noise and adversarial perturbations while ensuring the preservation of low-frequency structural information.

[0017] Step 4: Use the obtained estimates of the spoken audio and the adversarial perturbation audio to fit the given adversarial sample together, aiming to ensure that the fitting audio of the adversarial perturbation and the fitting of the spoken audio are both performed in the correct direction.

[0018] Compared to existing technologies, this invention leverages the ability of convolutional neural networks to automatically extract prior speech information without pre-training, applying it to the reconstruction and defense against adversarial examples. Furthermore, it employs a dual DAWP network to reconstruct and separate adversarial perturbations and speech information, while a low-resolution information extraction module is designed to accelerate the fitting speed of the DAWP network.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] 1. The method of this invention first introduces the deep speech waveform prior theory into the reconstruction defense of speech adversarial examples. This method does not require pre-training, but instead obtains an independent denoiser for each individual input adversarial example by performing targeted training. This means that for any input, a different gradient will be obtained from other samples, which effectively increases the difficulty of attack.

[0021] 2. The method of this invention is an adversarial example reconstruction method based on deep speech waveform priors. This method does not require modification of the target model and can be used independently as a preprocessing or postprocessing module of a speech recognition system, and can be plugged and played on any ASR system.

[0022] 3. The method of this invention designs a low-resolution information extraction module, which accelerates the fitting speed of the LowDDAWP-Net network; it designs a dual DAWP network to reconstruct and separate adversarial perturbations and speech segments, thereby improving the adversarial sample reconstruction performance of the LowDDAWP-Net algorithm.

[0023] 4. The method of the present invention does not require adjustment of other parameters after setting the initial value, and is an end-to-end, plug-and-play speech adversarial sample reconstruction algorithm. Attached Figure Description

[0024] Figure 1This is a schematic diagram of the entire process of the present invention.

[0025] Figure 2 This is a schematic diagram of the network structure of the present invention.

[0026] Figure 3 This is a schematic diagram of the structure of the DAWP neural network.

[0027] Figure 4 This is a comparison diagram between the results of the low-resolution information extraction module of the present invention and the original audio.

[0028] Figure 5 This is a comparison of the spectrograms of the defensive sample and the original sample of this invention. Detailed Implementation

[0029] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0030] Example 1: See Figure 1 An adversarial example reconstruction method based on deep speech waveform priors is proposed, with the following specific steps:

[0031] Step 1: Randomly read an adversarial sample and segment the sample into silent segments and speech segments using the VAD algorithm;

[0032] Step 2: The resulting silent segment, containing a large amount of noise information and adversarial perturbation information, is used as the adversarial perturbation fitting network (DAWP). noise The network uses randomly generated Gaussian noise as input to obtain fitted adversarial perturbation audio.

[0033] Specifically, DAWP noise The network structure is a U-shaped encoder and decoder. The decoder consists of six convolutional coding blocks and one single-layer bidirectional LSTM block; the encoder consists of six convolutional decoding blocks. Each convolutional coding block consists of two one-dimensional convolutional layers. The first convolutional layer has a kernel size of 8 and a stride of 4, while the second convolutional layer has a kernel size of 1 and a stride of 1. The output of each convolutional layer is activated using the ReLU function. The deepest convolutional coding block feeds its output into a single-layer bidirectional LSTM layer, and then a linear layer maps the fused high-dimensional features back to the input size. Each decoding block consists of a cascaded one-dimensional convolutional layer with a kernel size of 3 and a stride of 1, and a transposed convolutional layer with a kernel size of 8 and a stride of 4. The output of each convolutional layer is also activated by the ReLU function before being input into the next layer. The last convolutional decoding block synthesizes the final waveform signal.

[0034] DAWP noise The input is random Gaussian noise with a mean of 0 and a variance of 0.1, and the output is an estimate of the silent audio. The loss function is calculated as follows:

[0035]

[0036] Step 3: For the obtained utterance fragments, it is used as a utterance information fitting network (DAWP). speec h) The input to the tag and low-resolution information extraction module; low-resolution audio and DAWP speec The output audio of h is used to fit the speech fragment.

[0037] Specifically, DAWP speec h's network structure and input with DAWP noise The network is exactly the same, DAWP speec The loss function for h is:

[0038]

[0039] Here, `base` represents the output of the low-resolution information extraction module. This module consists of a cascade of AMR-V and Logmmse_SPU-cohen algorithms. The adversarial example is first processed by the AMR-V algorithm to filter out redundant signals and disrupt the integrity of the adversarial perturbation; then, the Logmmse_SPU-cohen algorithm is used to enhance the decompressed audio, aiming to further filter out noise and adversarial perturbations while ensuring the preservation of low-frequency structural information.

[0040] Step 4: Use the obtained estimates of the spoken audio and the adversarial perturbation audio to fit the given adversarial sample together, aiming to ensure that the fitting audio of the adversarial perturbation and the fitting of the spoken audio are both performed in the correct direction.

[0041] Effectiveness evaluation:

[0042] This invention presents a method for reconstructing adversarial examples based on deep speech waveform priors, achieving dynamic defense against adversarial examples. Compared with existing methods, this invention exhibits significantly higher performance in reconstructing adversarial examples, while having a lower impact on benign examples.

[0043] Table 1. Defense effectiveness of the present invention on a CW adversarial sample dataset generated based on SpeechCommand data.

[0044]

[0045] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or replacements made based on the above technical solutions are all within the scope of protection of the present invention.

Claims

1. A method for reconstructing adversarial examples based on deep speech waveform priors, characterized in that, The method includes the following steps: Step 1: Obtain the input speech sample x̂ to be restored, and use a speech activity detection algorithm to segment the input speech sample into speech segments and silence segments, and obtain the corresponding speech segment mask. speech and silence segment mask noise ; Step 2: Use the silent segment as the adversarial perturbation fitting network DAWP noise The training labels are given by using the first random Gaussian noise z2 as the training label for the adversarial perturbation fitting network DAWP. noise The input is used to train and obtain DAWP. noise The output y noise The y noise For fitting the adversarial perturbation audio; Step 3: Fit the utterance fragments as utterance information to the DAWP network. speech The training labels are generated, and the utterance fragments are input into the low-resolution information extraction module; the low-resolution information extraction module consists of the AMR-V algorithm and Logmmse. SPU The -cohen enhancement algorithm is cascaded to process the speech fragments and output a low-resolution audio base that preserves the low-frequency structural information of the speech fragments; the second random Gaussian noise z1 is used as the speech information fitting network DAWP. speech The input is used to train and obtain DAWP. speech The output y speech and make y speech Together with base, the utterance segment is fitted to obtain the fitted utterance audio y. speech +base; Step 4: Apply the total loss function of the LowDDAWP-Net network to DAWP noise and DAWP speech Real-time training is performed, and the total loss function is as follows: Among them, L noise L speech and L rec They are as follows: Where L1 represents the mean absolute error function of the two waveform samples, N(0, σ) 2 ) represents Gaussian noise with a mean of 0 and a variance of σ; Step 5: Output the fitted speech audio y speech +base is used as an estimate of the benign sample after the input speech sample has been defended.

2. The adversarial example reconstruction method based on deep speech waveform priors according to claim 1, characterized in that, During the defense phase of the algorithm, the DAWP noise and DAWP speech Both require real-time training and have the same network structure and input format, with the input being random Gaussian noise with a mean of 0 and a variance of 0.

1. The network structure is a U-shaped encoder and decoder structure. The encoder consists of 6 convolutional coding blocks and one single-layer bidirectional LSTM block, and the decoder consists of 6 convolutional decoding blocks. Each convolutional coding block consists of two one-dimensional convolutional layers. The first convolutional layer has a kernel size of 8 and a stride of 4, while the second convolutional layer has a kernel size of 1 and a stride of 1. The output of each convolutional layer is activated using the ReLU function. The deepest convolutional coding block feeds the output into a single-layer bidirectional LSTM layer, and then a linear layer is used to map the fused high-dimensional features back to the input size. Each convolutional decoding block consists of a cascaded one-dimensional convolutional layer with a kernel size of 3 and a stride of 1 and a transposed convolutional layer with a kernel size of 8 and a stride of 4. The output of each convolutional layer is also activated by the ReLU function before being input into the next layer. The last convolutional decoding block synthesizes the final waveform signal.

3. The adversarial example reconstruction method based on deep speech waveform priors according to claim 1, characterized in that, The low-resolution information extraction module in step 3 includes the variable bit-rate AMR compression AMR-V algorithm and the Log-mean-square error enhancement algorithm with speech uncertainty based on Cohen decision. SPU -cohen, the two are cascaded to obtain low-resolution audio that preserves the low-frequency structural information of speech fragments.

Images