An adversarial sample restoration method based on multi-version pre-processing sequence fusion

Abstract

The application discloses a plug-and-play voice confrontation sample defense method, first, the input voice is smoothed by Gaussian noise to generate a smooth voice sequence with subtle differences, then the smooth voice sequence is input into a multi-version preprocessing algorithm module, different implementation compression algorithms and enhancement algorithms are used to process the smooth voice to generate a multi-version voice sequence; then the generated multi-version voice sequence is input into a voice recognition system to obtain a corresponding voice transcription text sequence; finally, a mode voting weight allocation (MVWA) method is used to allocate weights to each text sequence, and then a recognizer output voting error reduction (ROVER) algorithm is used to estimate the benign transcription of the input voice according to the weights. Through the application, the aggressive confrontation sample can be restored to a benign sample, the safety performance of the voice recognition system is improved, and an effective restoration method is provided for the defense of the confrontation sample.

Description

Technical Field

[0001] This invention relates to an adversarial example reconstruction method based on multi-version preprocessing sequence fusion, belonging to the field of computer speech recognition technology. Background Technology

[0002] With the development of deep learning technology, researchers have begun to apply it to speech recognition. Deep learning can automatically learn the features of speech signals, significantly improving the accuracy and robustness of speech recognition. Therefore, automatic speech recognition technology has begun to be integrated into various fields, such as smart homes, smart cars, and smart robots, making people's production and lives more efficient and convenient. However, recent research shows that by adding carefully crafted adversarial perturbations to benign samples, adversarial examples can be generated that cause ASR systems to make incorrect judgments, and these examples are virtually indistinguishable to human perception. These adversarial speech examples can not only cause the victim model to misclassify, but can even cause it to transcribe the data into the attacker's desired outcome, thus posing security risks to related applications (such as voice assistants and voice verification codes).

[0003] Current research on defense against adversarial examples in speech processing mainly focuses on detection defense, with only a few studies addressing adversarial example reconstruction. However, a robust ASR system should aim to correctly transcribe any sample, not just detect adversarial examples and deny service. Furthermore, existing reconstruction defenses have low performance against adversarial examples and significantly impact benign examples, requiring modifications to the target model. Additionally, existing defense modules often simply discuss reconstruction using a single preprocessing algorithm or defense methods with new adversarial vulnerabilities, resulting in low robustness and poor reconstruction accuracy.

[0004] This invention addresses existing problems by proposing a multi-version preprocessing sequence fusion algorithm to achieve defense against adversarial examples. This method reconstructs adversarial examples based on multi-version programming, effectively breaking the overall integrity of adversarial perturbations through a random smoothing module; it designs a multi-version preprocessing module to reduce the impact of introduced Gaussian noise and further weaken the aggressiveness of adversarial perturbations; it introduces a ROVER voting mechanism to extract benign transcripts from the generated multi-version transcripts, and proposes a mode weight allocation method for dynamic weight calculation of preprocessed transcripts. The proposed method requires no pre-training and integrates the advantages of multiple preprocessing algorithms, effectively improving the algorithm's reproducibility and robustness. Summary of the Invention

[0005] The purpose of this invention is to provide an adversarial example restoration method based on multi-version preprocessing sequence fusion, namely the Multi-Version Preprocessing Sequence Fusion (MVPSF) algorithm, for restoring the transcription of adversarial examples into benign transcriptions. Specifically, it is used to restore the speech recognition content of adversarial speech examples into benign transcripts. This invention enables the restoration of aggressive adversarial examples into benign ones, and it integrates the advantages of multiple speech preprocessing algorithms.

[0006] To solve the above problems, the present invention adopts the following technical solution:

[0007] This invention provides an adversarial sample restoration method based on multi-version preprocessing sequence fusion, which can achieve high-precision restoration of adversarial samples with minimal impact on the transcription results of benign samples.

[0008] An adversarial example reconstruction method based on multi-version preprocessed sequence fusion, characterized by the following specific steps:

[0009] Step 1 involves inputting either benign or adversarial samples into the MVPSF algorithm. The aim is to ensure that processing benign samples does not affect their transcription, while reverting adversarial sample transcription to benign transcription. This is achieved by generating n Gaussian noise sequences of the same length as the input, with a mean of 0 and a variance of σ, using a Gaussian noise algorithm (such as the numpy.random.normal function). These n noise sequences are then added to the input. First, the Gaussian noise algorithm is used to smooth the input speech; that is, n Gaussian noise sequences of the same length as the input audio, with a mean of 0 and a variance of σ, are randomly generated. These n Gaussian noise sequences are then added to the input audio, resulting in n slightly different smooth speech sequences. This process initially disrupts the adversarial perturbation of the input adversarial samples to a certain extent.

[0010] Step 2: Input the obtained smoothed speech sequence into the multi-version preprocessing algorithm module for processing. This module consists of n compression and enhancement algorithms with different implementations. Input the obtained n smoothed sequences with slight differences into this module. Each smoothed audio corresponds to a preprocessing algorithm (compression or enhancement) for processing and output. Finally, the multi-version preprocessed speech sequence can be obtained through this module.

[0011] Step 3: The original input speech and multiple versions of the speech sequence are simultaneously input into the target speech recognition system. The speech recognition system converts each audio file into transcribed text, thereby obtaining the original transcript p and the multiple versions of the speech transcript p. i (i = 1, ..., n);

[0012] Step 4: Combine the original transcription p with multiple versions of the speech transcription sequence p. i (i = 1, ..., n) are input into the mode voting weight allocation module for weight calculation;

[0013] Step 5, convert the multiple versions of the speech transcription sequence p i The weight sequences of each transcription (i = 1, ..., n) are input into the ROVER module for voting to extract and output the transcription of benign samples, which is the benign transcription estimate of a given adversarial sample.

[0014] As an improvement of the present invention, the method implements defense in both the preprocessing and postprocessing stages of the speech recognition system. In the preprocessing stage, the input sample is first processed by Gaussian smoothing, and then processed by a multi-version preprocessing algorithm, so that each processing will remove adversarial perturbations to varying degrees. In the postprocessing stage, the speech recognition system estimates benign transcription from the output multi-version preprocessed transcription sequence, dynamically allocates weights based on the probability of the input sample being an adversarial sample, and performs voting based on the weights to finally obtain the transcription estimate of the sample.

[0015] As an improvement of the present invention, in step 4, the input to the mode voting weight allocation module consists of two variables and one constant. The input variables are the original audio transcription p and the multi-version speech transcription sequence p. i (i = 1, ..., n); the input constant is the character error rate threshold array for adversarial sample binary classification corresponding to each preprocessing algorithm in the multi-version preprocessing algorithm module. In the mode weight allocation module, it is first based on the original audio transcription p and the multi-version speech transcription sequence p. i (i = 1, ..., n) Calculate the character error rate of the text processed by the multi-version speech preprocessing algorithm; then classify adversarial examples based on the threshold and the calculated character error rate; finally, classify the samples according to the number of preprocessing algorithms that classify them as adversarial examples. Specifically, if the number of preprocessing algorithms that classify a sample as an adversarial example is not less than half of the total number of preprocessing algorithms, then the sample is considered more likely to be an adversarial example; otherwise, it is more likely to be a benign example. Finally, calculate the weights based on the classification of the samples. The formula for calculating the weights is:

[0016]

[0017] In the formula, num represents the number of preprocessing algorithms that classify the input samples as adversarial examples, and threshold i This represents the character error rate classification threshold set by the i-th (i = 1, ..., n) preprocessing algorithm, since cer i The value could be 0, but to avoid programming errors, 0.1 was added to the denominator cer.i superior.

[0018] As an improvement to this invention, the multi-version preprocessing algorithm module in step 3 is configured as a parallel combination of eight preprocessing algorithms, including the Variable Bit Rate Adaptive Multi-Rate Speech Coding (AMR-V) algorithm, the Fixed Bit Rate Adaptive Multi-Rate Speech Coding (AMR-C) algorithm, the Dolby AC-3 algorithm, the Variable Bit Rate SPEEX algorithm (SPEEX-V), the Log-Mean Square Error Enhancement Algorithm with Speech Uncertainty Based on Soft Decision (Logmmse_SPU-soft), the Log-Mean Square Error Enhancement Algorithm with Speech Uncertainty Based on Hard Decision (Logmmse_SPU-hard), the Log-Mean Square Error Enhancement Algorithm with Speech Uncertainty Based on Malah Decision (Logmmse_SPU-malah), and the Log-Mean Square Error Enhancement Algorithm with Speech Uncertainty Based on Cohen Decision (Logmmse_SPU-cohen).

[0019] Compared with existing technologies, this invention utilizes the characteristic that different implementations of preprocessing algorithms have different degrees and ranges of filtering out adversarial perturbations. It proposes to use a multi-version preprocessing algorithm to process adversarial samples, and then use the ROVER algorithm and the mode weighting method to extract benign transcripts from the multi-version transcripts, thus achieving a highly reproducible and robust reproducible defense against adversarial samples.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] 1. The method of this invention first introduces the concept of multi-version programming into the reconstruction defense of adversarial speech examples. This method does not require training and effectively integrates the advantages of multiple preprocessing methods, thereby effectively improving the robustness of the reconstruction module.

[0022] 2. The method of this invention is an adversarial example reconstruction method based on multi-version preprocessed sequence fusion. This method does not require modification of the target model, can be used independently as a processing module of a speech recognition system, and can be plugged and played on any ASR system.

[0023] 3. The method of this invention designs a mode weight allocation method, which classifies the input by the results of multiple versions of preprocessing algorithms, and then assigns different weight calculation methods to each preprocessed transcription according to different classifications, thereby realizing dynamic weight allocation of preprocessed transcription and achieving effective estimation of benign transcriptions of adversarial samples.

[0024] 4. The method of the present invention does not require adjustment of other parameters after setting the initial value, and is an end-to-end, plug-and-play speech adversarial sample reconstruction algorithm. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the entire process of the present invention.

[0026] Figure 2 This is a schematic diagram of the structure of the present invention, wherein the target system is taken as an example of the DeepSpeech model. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0028] Example 1: See Figure 1 , Figure 2 An adversarial example reconstruction method based on multi-version preprocessed sequence fusion is described below. The specific steps are as follows: Taking an adversarial example as an example, the specific steps are as follows:

[0029] Step 1: Randomly read an adversarial sample. First, pass the sample through a random smoothing module to amplify a single input speech into n smooth speech sequences with subtle differences, while at the same time destroying the offensiveness of the adversarial perturbation to a certain extent.

[0030] Step 2: Input the obtained n smooth speech sequences with subtle differences into a multi-version speech preprocessing module with n different implementations. While eliminating the influence of Gaussian noise introduced by the smoothing module, further filter out interference that opposes perturbation, and output a multi-version speech sequence containing n speech sequences.

[0031] Step 3: Input the obtained multi-version speech sequences and the original audio into the target speech recognition system to obtain multi-version speech transcription p. i (i = 1, ..., n) and the original transcription p;

[0032] Step 4: Transcribe the obtained multi-version speech into p i The original transcription p is input into the mode weight allocation module to assign weights to each preprocessed transcription;

[0033] Specifically, in the mode voting weight allocation module, the input consists of two variables and one constant. The input variables are the original audio transcription p and the multiple versions of the speech transcription sequence p. i (i = 1, ..., n); the input constant is an array of character error rate thresholds for adversarial binary classification corresponding to each preprocessing algorithm in the multi-version preprocessing algorithm module. In the mode weight allocation module, the original audio transcription p and the multi-version speech transcription sequence p are first used as the basis for the weight allocation. i(i = 1, ..., n) Calculate the character error rate of the text processed by the multi-version speech preprocessing algorithm; then classify adversarial examples based on the threshold and the calculated character error rate; finally, classify the samples according to the number of preprocessing algorithms that classify them as adversarial examples. Specifically, if the number of preprocessing algorithms that classify a sample as an adversarial example is not less than half of the total number of preprocessing algorithms, then the sample is considered more likely to be an adversarial example; otherwise, it is more likely to be a benign example. Finally, calculate the weights based on the classification of the samples. The formula for calculating the weights is:

[0034]

[0035] In the formula, num represents the number of preprocessing algorithms used to classify the input sample as adversarial examples. i This represents the character error rate classification threshold set by the i-th (i = 1, ..., n) preprocessing algorithm. Since cer i The value could be 0, but to avoid programming errors, 0.1 was added to the denominator cer. i superior.

[0036] Step 5, convert the multiple versions of the speech transcription sequence p i The weight sequences of each transcription (i = 1, ..., n) are input into the ROVER module for voting to extract and output the transcription of benign samples, which is the benign transcription estimate of a given adversarial sample.

[0037] Effectiveness evaluation:

[0038] This invention presents a method for reconstructing adversarial examples based on deep speech waveform priors, achieving dynamic defense against adversarial examples. Compared with existing methods, this invention exhibits significantly higher performance in reconstructing adversarial examples, while having a lower impact on benign examples.

[0039] Table 1. Defense effectiveness of the present invention on a CW adversarial sample dataset generated based on SpeechCommand data.

[0040]

[0041] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or replacements made based on the above technical solutions are all within the scope of protection of the present invention.

Claims

1. A method for reconstructing adversarial examples based on multi-version preprocessed sequence fusion, characterized in that, The specific steps of the method are as follows: Step 1: Input benign or adversarial samples into the MVPSF algorithm. The aim is to ensure that benign samples are processed without affecting their transcription, while the transcription of adversarial samples is restored to benign transcription. Use a Gaussian noise algorithm to smooth the input speech. Specifically, use the Gaussian noise algorithm to generate n Gaussian noise sequences with the same length as the input, mean 0, variance σ, and then add the n noise sequences to the input. This results in a smooth speech sequence with subtle differences and can destroy adversarial perturbations to a certain extent. Step 2: The obtained smoothed speech sequence is input into the multi-version preprocessing algorithm module for processing. This module consists of n different implementations of compression and enhancement algorithms. Specifically, the n smoothed speech sequences in Step 1 are processed by a preprocessing algorithm and output, thus obtaining the multi-version preprocessed speech sequence through this module. Step 3: The original input speech and multiple versions of the speech sequence are simultaneously input into the target speech recognition system. The target speech recognition system transcribes each speech and outputs the result, ultimately obtaining the original transcribed p and the multiple version speech transcribed sequences. ; Step 4: Combine the original transcribed p and multiple versions of the speech transcription sequence. The input is fed into the mode voting weight allocation module for weight calculation; Step 5: Transcribe multiple versions of the speech sequence. The weighted sequences of each transcription are input into the ROVER module for voting to extract and output the transcription of benign samples, which is the benign transcription estimate of a given adversarial sample. In step 4, the mode voting weight allocation module receives two variables and one constant as input. The input variable is the transcription of the original audio. and multiple versions of speech transcription sequences The input constant is an array of character error rate thresholds for adversarial binary classification corresponding to each preprocessing algorithm in the multi-version preprocessing algorithm module. In the mode weight allocation module, it is first determined based on the transcription of the original audio. and multiple versions of speech transcription sequences The process involves calculating the character error rate (CR) of the text processed by multiple versions of speech preprocessing algorithms; then, classifying adversarial examples based on a threshold and the calculated CR; finally, classifying samples according to the number of preprocessing algorithms that classify them as adversarial examples, specifically as follows: if the number of preprocessing algorithms that classify a sample as adversarial is not less than half of the total number of preprocessing algorithms, the sample is considered more likely to be an adversarial example; otherwise, it is more likely to be a benign example. Finally, weights are calculated based on the classification of the samples, using the following formula: In the formula, num represents the number of preprocessing algorithms that classify the input samples as adversarial examples. This represents the character error rate classification threshold set by the i-th (i=1,…,n) preprocessing algorithm.

2. The adversarial example reconstruction method based on multi-version preprocessed sequence fusion according to claim 1, characterized in that, This method defends against perturbations in both the preprocessing and postprocessing stages of the speech recognition system. In the preprocessing stage, the input samples are first processed by Gaussian smoothing and then by multiple versions of the preprocessing algorithm, so that each processing will remove adversarial perturbations to varying degrees. In the post-processing stage, the speech recognition system estimates benign transcription from the output multi-version preprocessed transcribed sequences, dynamically allocates weights based on the probability that the input sample is judged as an adversarial sample, and performs voting based on the weights to finally obtain the transcription estimate of the sample.

3. The adversarial example reconstruction method based on multi-version preprocessed sequence fusion according to claim 1, characterized in that, In this method, the multi-version preprocessing algorithm module in step 3 is configured as a parallel combination of eight preprocessing algorithms, including a variable bit rate adaptive multi-rate speech coding algorithm, a fixed bit rate adaptive multi-rate speech coding algorithm, a Dolby AC-3 algorithm, a variable bit rate SPEEX algorithm, a log-mean square error enhancement algorithm with speech uncertainty based on soft decision, a log-mean square error enhancement algorithm with speech uncertainty based on hard decision, a log-mean square error enhancement algorithm with speech uncertainty based on Malah decision, and a log-mean square error enhancement algorithm with speech uncertainty based on Cohen decision.

Images