Speech recognition adversarial sample defense method and system based on spectral features

Abstract

The application discloses a speech recognition adversarial sample defense method and system based on spectral features. White-box attacks and black-box attacks are used to attack known clean samples to generate adversarial samples, and the clean samples and the generated adversarial samples are subjected to a variety of speech de-disturbance methods to obtain simulated clean samples. The samples are simultaneously converted into a frequency domain, a feature vector is obtained by using the similarity of each frame of a spectrum diagram, a classifier is trained by marking a label on the similarity feature vector. When unknown adversarial samples from the outside world arrive, the samples are first subjected to a variety of speech de-disturbance methods to generate simulated clean samples, and the adversarial samples and the generated clean samples are also subjected to pretreatment and converted into a frequency domain. The spectrum diagram similarity feature vector is input into the classifier. If the classifier detects the adversarial samples, the adversarial samples are directly discarded. If the classifier detects normal samples, the normal samples enter a speech recognition system to complete the defense. The application can be applied to various types of speech adversarial sample attack modes and improve the defense success rate.

Description

Technical Field

[0001] This invention relates to a method and system for defending against adversarial examples in speech recognition based on spectral features, belonging to the field of artificial intelligence testing. Background Technology

[0002] With the rapid development of computers and the internet, speech recognition has gradually become a key intelligent technology for efficient human-computer interaction. Speech recognition technology, also known as Automatic Speech Recognition (ASR), aims to convert the lexical content of human speech into computer-readable input. The application of speech recognition has greatly improved the efficiency of human-computer interaction and is permeating all aspects of people's daily lives. Speech recognition is mainly based on deep learning technology, but adversarial examples can attack deep learning models, causing them to make incorrect judgments when recognizing speech. Hackers only need to add some carefully designed perturbations that are difficult for humans to detect to a speech segment, and the speech recognition model will recognize the speech as gibberish or empty speech segments. Therefore, adversarial example attacks against speech recognition pose a security threat to related applications. Most speech recognition models are affected by adversarial example attacks, making research on adversarial example defense particularly important in response to the emergence of such adversarial attack methods.

[0003] Adversarial example defense has been extensively studied in image recognition and text processing, while speech adversarial example defense is a relatively new field, and its methods differ somewhat from those used for image and text adversarial examples. Currently, the defense schemes developed by scholars for speech adversarial examples are largely similar. However, due to the ever-evolving nature of adversarial example attack patterns, the effectiveness of many defense methods drops drastically with slight variations in the attack. Recently, Qiang Zeng et al. proposed a novel adversarial example detection method based on the established fact that different ASR systems using different architectures, parameters, and training datasets result in differences in their transcription of the same audio. This method, combined with the concept of multi-version programming, still has some drawbacks: the success rate of defense is not high, and the classifier's defense effectiveness varies significantly across different samples, leaving room for improvement. Summary of the Invention

[0004] Purpose of the invention: Considering the proliferation of adversarial example attack patterns and the different characteristics of various speech and audio, and the lack of universality in defense solutions, most of which are specifically designed to defend against single attacks, and the limitations of some defense solutions in improving the accuracy of speech recognition, the purpose of this invention is to provide a speech recognition adversarial example defense method and system based on spectral features, which can be applied to various types of speech adversarial example attack methods and improve the success rate of defense.

[0005] Technical Solution: To achieve the above-mentioned objectives, the present invention provides a speech recognition adversarial example defense method based on spectral features, comprising the following steps:

[0006] Step 1: Denoise the unknown original sample from the outside world using speech denoising methods to obtain simulated clean sample 1; the speech denoising methods include multiple methods such as Wiener filtering, spectral subtraction, ACC compression, and Speex compression.

[0007] Step 2: Perform preprocessing operations on the original sample and the obtained simulated clean sample 1, and convert them to the frequency domain to obtain their spectrogram features;

[0008] Step 3: Train the classifier: Use white-box attack and black-box attack methods to obtain adversarial samples of known clean speech samples respectively, and use speech denoising method to denoise the clean speech samples and the generated adversarial samples to obtain simulated clean sample 2;

[0009] The clean speech samples, adversarial samples, and the simulated clean sample 2 obtained after their processing are preprocessed and transformed into the frequency domain to obtain their spectrogram features.

[0010] By comparing the similarity of the spectrograms of the two types of samples in each frame, their feature vectors are obtained. Labels are then added to the vectors to determine whether they are adversarial samples, and this is used to train a classifier.

[0011] Step 4: Input the similarity feature vector corresponding to the spectrogram in Step 2 into the trained classifier to detect whether the original sample is an adversarial sample, thus completing the overall defense framework.

[0012] Preferably, in step 1, several excellent speech denoising methods are selected and appropriately improved. Based on the denoising characteristics, four speech denoising methods are selected: Wiener filtering, spectral subtraction, AACC compression, and Speex compression. Spectral subtraction is further optimized to overcome the problem of "musical noise."

[0013] Improved Spectral Subtraction: The original sample is perturbed by improved spectral subtraction to obtain the corresponding simulated clean sample 1. The noise is assumed to be locally stable, i.e., consistent with the speech signal, and is additive noise, uncorrelated with the clean speech signal. Based on these assumptions, the following definition is made: The noise in speech is only additive noise; by subtracting the noise spectrum from the noisy speech spectrum, clean speech can be obtained. To prevent the occurrence of "musical noise," a speech activity detection method is used. First, silent segments are removed by energy thresholding. Then, the zero-crossing rate feature of speech activity detection distinguishes noise frames from speech frames, identifying the noise segment. Finally, based on the perturbation distribution location, since noise is concentrated in the high-frequency speech segment to be ignored by the ear, several frames of noise are selected in the high-frequency speech segment, and their average is taken to estimate the overall noise. The original sample is then framed, and each frame undergoes Fourier transform to the frequency domain, spectral subtraction, and inverse Fourier transform to obtain simulated clean sample 1.

[0014] Preferably, the specific steps in step 2 are further described as follows:

[0015] Step 21: Perform a series of preprocessing operations on the original sample and the obtained simulated clean sample 1, such as pre-emphasis, frame segmentation, and windowing.

[0016] Step 22: Convert the speech signals of the original sample and the obtained simulated clean sample 1 from the time domain to the frequency domain to obtain the spectrum.

[0017] Step 23: Match the features of the simulated clean sample 1 obtained by each perturbation method with the original sample spectrogram features, and calculate the similarity between the corresponding spectrograms to prepare for input into the classifier;

[0018] Preferably, the specific steps in step 3 are further described as follows:

[0019] Step 31: Use white-box attack methods and black-box attack methods to attack clean samples to form corresponding adversarial samples;

[0020] Step 32: Denoise the clean samples and the generated adversarial samples using the same speech de-perturbation method as in Step 1 to obtain the corresponding simulated clean sample 2;

[0021] Step 33: Perform a series of preprocessing operations on the clean samples, adversarial samples, and the simulated clean samples 2 obtained after processing them, such as pre-emphasis, frame segmentation, and windowing.

[0022] Step 34: Convert the speech signals of the clean sample, the adversarial sample, and the simulated clean sample 2 obtained after their processing from the time domain to the frequency domain to obtain the spectrogram.

[0023] Step 35: Match the spectrogram features of the simulated clean sample 2, the clean sample, and the generated adversarial sample obtained by each perturbation method one by one, and prepare to compare their similarity.

[0024] Step 36: Calculate the average of the four correlations between each frame of the clean sample dataset or adversarial sample dataset and each simulated clean sample dataset. Label the resulting multiple pairs of correlation coefficient vectors. Label the vectors obtained from clean samples with clean labels and the vectors obtained from adversarial samples with adversarial labels. Train the classifier in this way.

[0025] Step 37: First, use a convolutional neural network to extract features from the correlation coefficient vector, and then use the cross-entropy loss function to train the parameters of the convolutional neural network.

[0026] Preferably, in step 7

[0027] If the classifier identifies it as an adversarial example, it discards it directly; if it identifies it as a clean example, it inputs it into the speech recognition system for the next step, completing the entire defense system.

[0028] Based on the same inventive concept, this invention provides a speech recognition adversarial example defense system based on spectral features, comprising:

[0029] The original sample denoising module is used to denoise unknown original samples from the outside world using speech denoising methods to obtain simulated clean sample 1; the speech denoising methods include multiple methods such as Wiener filtering, spectral subtraction, ACC compression and Speex compression.

[0030] The original sample frequency domain conversion module is used to preprocess the original sample and the obtained simulated clean sample 1, and convert them to the frequency domain to obtain their spectrum characteristics.

[0031] The classifier training module uses white-box and black-box attacks to obtain adversarial samples from known clean speech samples. It then uses speech denoising methods to denoise both the clean speech samples and the generated adversarial samples, resulting in simulated clean sample 2. The clean speech samples, adversarial samples, and the simulated clean sample 2 are preprocessed and converted to the frequency domain to obtain their spectrogram features. The similarity of the spectrograms of the two types of samples in each frame is compared to obtain their feature vectors. Labels are then added to these vectors to determine whether they are adversarial samples, and this process is used to train the classifier.

[0032] It also includes a defense detection module, which inputs the similarity feature vector corresponding to the spectrogram obtained by the original sample frequency domain conversion module into the trained classifier to detect whether the original sample is an adversarial sample, thus completing the overall defense framework.

[0033] Based on the same inventive concept, the present invention provides a computer system including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the steps of the speech recognition adversarial example defense method based on spectral features.

[0034] Based on the same inventive concept, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the speech recognition adversarial sample defense method based on spectral features.

[0035] Beneficial Effects: This invention provides a speech recognition adversarial example defense method based on spectral features. It utilizes feature similarity in the frequency domain to train a classifier, and then uses this classifier to complete the adversarial example defense. When externally interfered audio samples are input, various de-perturbation methods are used to generate corresponding simulated clean samples. Both the original audio samples and the simulated clean samples are first initialized, then converted to the frequency domain, and the similarity between the frequency domain features of each frame is obtained and input into the classifier. The classifier identifies the two types of samples as adversarial or clean samples by calculating the similarity between their frequency domains, thus completing the overall defense. To verify the effectiveness of this method on real speech datasets, the white-box attack method (C&W attack) and the black-box attack method (GAGE attack) are selected as typical attack methods to generate adversarial example datasets. In the white-box attack, 1000 samples from the Mozilla Common Voice dataset are selected; in the black-box attack, 1000 samples from the Google Speech Command dataset are selected. Both attacks target deep speech, resulting in the adversarial example datasets. The dataset is divided into training and test sets in an 8:2 ratio. 800 pairs of samples are used to train the classifier, and the remaining 200 adversarial examples are used to test the effectiveness of the defense method. Experiments show that this invention has the following advantages over other speech adversarial example defense schemes: first, it utilizes methods unique to speech, making it applicable in many real-world speech scenarios; second, it has a certain degree of universality, providing some defense against most speech attack methods on the market; and third, compared to other detection-based defense schemes in the field of speech adversarial defense, this scheme shows improvement in both accuracy and recall. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the overall steps of an embodiment of the present invention;

[0037] Figure 2 This is a flowchart illustrating a specific example of the method of the present invention. Detailed Implementation

[0038] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0039] like Figure 1 As shown in the figure, an adversarial example defense method for speech recognition based on spectral features disclosed in this invention mainly includes the following steps:

[0040] Step 1: Use the speech denoising method to denoise the unknown original samples from the outside world to obtain the simulated clean sample 1;

[0041] Step 2: Perform preprocessing operations on the original sample and the obtained simulated clean sample 1, and convert them to the frequency domain to obtain their spectrogram features;

[0042] Step 3: Classifier Training: Attack clean speech samples in the open-source library using white-box and black-box attacks respectively to obtain adversarial samples. Then, use speech denoising methods to denoise the clean samples and the generated adversarial samples to obtain simulated clean sample 2. Perform preprocessing operations on the clean samples, adversarial samples, and the simulated clean sample 2 obtained after their processing, and convert them to the frequency domain to obtain their spectrogram features. Compare the similarity of the spectrograms of the two types of samples in each frame to obtain their feature vectors. Label the vectors to determine whether they are adversarial samples, and use this to train the classifier.

[0043] Step 4: Input the similarity feature vector corresponding to the spectrogram in Step 2 into the trained classifier to detect whether the original sample is an adversarial sample, thus completing the overall defense framework.

[0044] like Figure 2 As shown, this embodiment of the invention uses the Mozilla Common Voice dataset and the Google Speech Command dataset for speech recognition as examples to illustrate the detailed steps of a speech recognition adversarial example defense method based on spectral features disclosed in this embodiment of the invention, as follows:

[0045] Step 1: Use the speech denoising method to denoise the unknown original sample from the outside world to obtain the simulated clean sample 1.

[0046] This step selects Wiener filtering, AACC compression, Speex compression, and spectral subtraction. To address the "musical noise" problem inherent in spectral subtraction, certain improvements are made. A speech activity detection method is used. Since the original spectral subtraction estimates the noise (perturbation) in the entire speech segment by estimating the noise in the initial silence segment, and assumes the noise in the entire speech segment is smooth, there are always discrepancies. The speech activity detection method first removes silence segments using an energy threshold, then estimates the subsequent noise segments, distinguishes noise frames from speech frames using zero-crossing rate features, and then, based on the perturbation distribution location, selects several frames in the high-frequency speech segment and averages them to estimate the overall noise. Specifically, this step includes:

[0047] Step 11: De-perturb the original sample using Wiener filtering to obtain the corresponding simulated clean sample 1. Let y(m) represent the noisy speech signal in the original sample and satisfy: y(m)=s(m)+n(m), where s(m) represents the clean signal without noise and n(m) is the additive noise in the original signal. When s(m) and n(m) are uncorrelated and the random process is stationary, perform a discrete Fourier transform on y(m)=s(m)+n(m) to obtain equation (1): Y(m,k)=S(m,k)+N(m,k). After obtaining equation (1), let the frequency domain response function of Wiener filtering be H(m,k), and obtain the Fourier transform of the best signal estimate s′(m) as S′(m,k), as shown in equation (2): S′(m,k)=H(m,k)·Y(m,k). Finally, following the principle of minimum mean square error, the output signal s′(m) is made as close as possible to the original signal; where m is the frame number and k is the frequency point.

[0048] Step 12: Perform improved spectral subtraction on the original sample to remove perturbation, obtaining the corresponding simulated clean sample 1. Assume the noise is locally stable, i.e., consistent with the speech signal, and that the noise is additive, uncorrelated with the clean speech signal. Based on the above assumptions, we have the following definition: Assuming the noise in speech is only additive, we can obtain clean speech by subtracting the noise spectrum from the noisy speech spectrum. To prevent the occurrence of "musical noise," we use a speech activity detection method. First, we remove silent segments by dividing the speech by an energy threshold. Then, we distinguish noise frames from speech frames by the zero-crossing rate feature of speech activity detection to find the segment where the noise is located. Finally, based on the perturbation distribution location, since the noise is concentrated in the high-frequency speech segment to be ignored by the ear, we select several frames of noise in the high-frequency speech segment and average them to estimate the noise of the entire segment. Then, we divide the original sample into frames, perform Fourier transform on each frame to the frequency domain, perform spectral subtraction and inverse Fourier transform to obtain simulated clean sample 1.

[0049] Step 13: Use the ACC compression method to remove perturbations from the original sample to obtain the corresponding simulated clean sample 1. ACC compresses the original sample for audio, thereby eliminating certain perturbations. AAC can support up to 48 audio tracks, 15 low-frequency tracks, 5.1 multi-channel support, higher sampling rates, higher sampling precision, and has compatibility capabilities in multiple languages, as well as higher decoding efficiency. The simulated clean sample 1 obtained after AAC not only reduces the size by 30% but also provides better sound quality.

[0050] Step 14: Use the speex compression method to remove perturbations from the original sample to obtain the corresponding simulated clean sample 1. Speex compresses the original sample for audio, thereby eliminating certain perturbations. Speex is targeted at Internet Protocol Telephony (VoIP) and file-based compression. The Speex encoder uses multiple bitrates and supports ultra-wideband, wideband (16 kHz sampling rate), and narrowband (8 kHz sampling rate). The original sample after Speex forms a high-quality simulated clean sample in a way of reducing the bitrate.

[0051] Step 2: Perform preprocessing operations on the original sample and the obtained simulated clean sample 1, and convert them to the frequency domain to obtain their spectrogram features; specifically including:

[0052] Step 21: Perform a series of preprocessing operations on the original sample and the obtained simulated clean sample 1, including pre-emphasis, framing, and windowing, etc. Among them, use a filter to implement pre-emphasis on the audio sample. Let a be the pre-emphasis coefficient, 0.9 < a < 1.0. The speech sampling value at time n is x(n), and the result after pre-emphasis processing is y(n) = x(n) - ax(n - 1). In this example, a = 0.97 is taken.

[0053] Perform a framing operation on the audio sample. The Fourier transform requires that the input signal be stationary, but the speech signal is generally non-stationary as a whole. To make the frames smoothly transition and maintain their continuity, the frame length is taken as 200, and the frame shift is taken as 80.

[0054] Window the audio sample with a Hamming window, that is, multiply it by a window function. Windowing is for Fourier expansion to make the global more continuous and avoid the Gibbs effect; when windowing, the originally non-periodic speech signal shows some characteristics of a periodic function. The cost of windowing is that the two ends of a frame of the signal are weakened, so there needs to be an overlap between frames during framing, and the window length is set to N = 200.

[0055] Step 22: Convert the speech signals of the original samples and the obtained simulated clean sample 1 from the time domain to the frequency domain to obtain the spectrogram. Transform the audio samples from the time domain to the frequency domain using Fourier transform to obtain their spectrograms. Set the spectrogram vectors of each set of original samples and the obtained simulated clean sample 1, and calculate the similarity between the corresponding spectrograms.

[0056] Step 3: Train the classifier. First, generate the required samples. Use white-box and black-box attacks to attack clean speech samples in the open-source library to obtain adversarial samples. Then, use speech denoising methods to denoise both the clean samples and the generated adversarial samples to obtain simulated clean samples 2.

[0057] Step 31: Train the classifier. First, use white-box and black-box attack methods to attack clean samples to form corresponding adversarial examples. Select the white-box attack method C&W attack and the black-box attack method GAGE ​​attack as typical attack methods to generate the adversarial example dataset. In the white-box attack, select 1000 samples from the Mozilla Common Voice dataset; in the black-box attack, select 1000 samples from the Google Speech Command dataset. Both attacks target deep speech, resulting in the adversarial example dataset.

[0058] Step 32: Denoise the clean samples and the generated adversarial samples using the speech de-perturbation method in Step 1 to obtain the corresponding simulated clean samples 2; specifically, denoise the adversarial samples using Wiener filtering, ACC compression, Speex compression and spectral subtraction respectively to generate the corresponding simulated clean samples 2.

[0059] Step 4: Train the classifier and convert to the frequency domain. Preprocess the clean samples, adversarial samples, and the simulated clean sample 2 obtained after their processing, and convert them to the frequency domain to obtain their spectrogram features.

[0060] Step 41: Perform preprocessing operations on the clean samples, adversarial samples, and the simulated clean samples 2 obtained after their processing using the same pre-emphasis, framing, and windowing methods as in Step 2.

[0061] Step 42: The clean samples, adversarial samples, and the simulated clean sample 2 obtained after their processing are all transformed from the time domain to the frequency domain by Fourier transform to obtain their spectrum. The spectrum vectors of each group of clean samples or adversarial samples and the obtained simulated clean sample 2 are set to correspond one-to-one.

[0062] Step 5: Train the classifier using similarity scoring. Compare the similarity of the spectrograms of the two classes of samples in each frame to obtain their feature vectors. Label the vectors to determine whether they are adversarial examples, and use this to train the classifier.

[0063] Step 51: The Pearson Correlation Coefficient is a classic correlation analysis method used to measure the degree of linear correlation between two variables. Linear correlation refers to a linear relationship between two variables, where the value of one variable increases or decreases as the value of the other increases. In this embodiment, the Pearson Correlation Coefficient is used to calculate the similarity of the spectrograms of the two classes of samples in each frame. The Pearson Correlation Coefficient between two sample variables is defined as the quotient of the covariance and standard deviation of the two variables. The specific steps for calculating the Pearson Correlation Coefficient are as follows: 1. Standardize the two variables by subtracting their mean from each variable and then dividing by their standard deviation. 2. For each data point, multiply the standardized values ​​of the two variables and sum them over all data points. 3. Divide this sum by the number of sample data points to obtain a value r, which is between -1 and 1. 4. The closer r is to 1, the stronger the positive correlation between the two variables; the closer r is to -1, the stronger the negative correlation between the two variables; and an r value close to 0 indicates almost no linear correlation between the two variables. In this embodiment, we calculate the covariance and standard deviation of each frame for the clean sample or adversarial sample and the obtained simulated clean sample. Then, we calculate the similarity between each frame according to the Pearson correlation coefficient formula. Since this scheme uses four perturbation methods, we add up the four similarity values ​​of each frame to obtain the average, form the similarity vector of this frame, and label it according to the initial sample, ready to input it into the classifier.

[0064] Step 52: Use the correlation coefficient vector of each frame to train the classifier. Label the similarity vectors between clean samples and their corresponding simulated samples with a clean label, and label the similarity vectors between adversarial samples and their corresponding simulated samples with an adversarial label, allowing the classifier to learn. Use a convolutional neural network for feature extraction, and then use the cross-entropy loss function to model the classifier network. A correlation coefficient between spectrograms less than a certain threshold is considered a clean sample, and a correlation coefficient greater than a certain threshold is considered an adversarial sample. This process is used to train a classifier network.

[0065] Step 6: Input the similarity feature vector corresponding to the spectrogram from Step 2 into the classifier to detect whether the original sample is an adversarial sample, thus completing the overall defense framework. Specifically, input the spectrogram similarity feature vector extracted from each group of original samples and simulated clean sample 1 in Step 2 into the classifier. If the classifier identifies it as an adversarial sample, it is discarded directly; if the classifier identifies it as a clean sample, it is input into the speech recognition system to complete the next step of speech-to-text conversion.

[0066] In summary, this invention provides a novel classifier training method and completes overall defense against adversarial examples in speech detection. It addresses, to some extent, the problem of severe overfitting in many current adversarial defense methods, which are ineffective against unfamiliar adversarial examples. Furthermore, the application of acoustic features in speech detection tasks and the use of diversity ensemble methods to detect adversarial examples require further research; most current speech adversarial example defense methods still do not consider real-world applications. This invention uses acoustic spectrogram feature extraction to train a classifier, attempting to detect whether the test sample contains adversarial examples. If adversarial examples are found, the sample is rejected, thereby reducing adversarial attacks. First, this invention uses four speech denoising methods, conducting denoising experiments to observe changes in signal-to-noise ratio, verifying the effectiveness of each method. Then, to verify the effectiveness of this method, white-box and black-box attacks were used to generate adversarial examples, enabling the trained classifier to resist most attacks. Experiments were also conducted on the classifier's performance, observing the accuracy and recall. Both white-box and black-box attacks showed improvements in accuracy and recall, validating the overall defensive effect of the method.

[0067] The purpose of the detection-based defense experiment was to verify the effectiveness of the classifier. Precision was used to calculate the probability that a sample predicted as positive was actually positive, and recall was the proportion of true positive samples out of all true positive samples. An F1 score was also defined, which considers both precision and recall, aiming to maximize both while achieving a balance. The results of five trials are shown in the table below.

[0068] Table 1 Results of White-Box Adversarial Sample Experiments

[0069] White-box adversarial examples 1 2 3 4 5 Precision (%) 97.3 96.1 96.3 97.0 97.1 Recall (%) 91.8 90.3 90.5 90.8 90.1 F1-Score 0.945 0.931 0.933 0.938 0.935

[0070] Table 2 Results of Black-Box Adversarial Sample Experiments

[0071] Black-box adversarial samples 1 2 3 4 5 Precision (%) 93.8 92.2 95.2 94.7 94.1 Recall (%) 91.1 90.3 91.2 90.8 90.1 F1-Score 0.924 0.912 0.932 0.927 0.921

[0072] Based on the same inventive concept, this invention discloses a speech recognition adversarial sample defense system based on spectral features, comprising: an original sample denoising module, used to denoise unknown original samples from the outside world using a speech denoising method to obtain simulated clean sample 1; an original sample frequency domain conversion module, used to preprocess the original samples and the obtained simulated clean sample 1, and convert them to the frequency domain to obtain their spectrogram features; a classifier training module, using white-box attack and black-box attack methods to obtain adversarial samples of known clean speech samples respectively, and using a speech denoising method to denoise the clean speech samples and the generated adversarial samples to obtain simulated clean sample 2; preprocessing the clean speech samples, adversarial samples, and the simulated clean sample 2 obtained after their processing, and converting them to the frequency domain to obtain their spectrogram features; comparing the similarity of the spectrograms of the two types of samples in each frame to obtain their feature vectors, labeling the vectors to determine whether they are adversarial samples, and using this to train the classifier; and a defense detection module, used to input the similarity feature vector corresponding to the spectrogram obtained by the original sample frequency domain conversion module into the trained classifier to detect whether the original sample is an adversarial sample, thus completing the overall defense framework.

[0073] Based on the same inventive concept, an embodiment of the present invention discloses a computer system including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the steps of the speech recognition adversarial sample defense method based on spectral features.

[0074] Based on the same inventive concept, embodiments of the present invention disclose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the speech recognition adversarial sample defense method based on spectral features.

Claims

1. A speech recognition adversarial example defense method based on spectral features, characterized in that, Includes the following steps: Step 1: Denoise the unknown original sample from the outside world using speech denoising methods to obtain simulated clean sample 1; the speech denoising methods include multiple methods such as Wiener filtering, spectral subtraction, ACC compression, and Speex compression. Step 2: Perform preprocessing operations on the original sample and the obtained simulated clean sample 1, and convert them to the frequency domain to obtain their spectrogram features; Step 3: Classifier Training: Adversarial samples of known clean speech samples are obtained using white-box and black-box attacks, respectively. Speech denoising methods are then applied to both the clean speech samples and the generated adversarial samples to obtain simulated clean sample 2. The clean speech samples, adversarial samples, and the simulated clean sample 2 obtained after their processing are preprocessed and converted to the frequency domain to obtain their spectrogram features. The similarity of the spectrograms of the two types of samples in each frame is compared to obtain their feature vectors. Labels are then added to these vectors to determine whether they are adversarial samples, and this is used to train the classifier. Step 4: Input the similarity feature vector corresponding to the spectrogram in Step 2 into the trained classifier to detect whether the original sample is an adversarial sample, thus completing the overall defense framework.

2. The method for adversarial example defense in speech recognition based on spectral features according to claim 1, characterized in that, Four speech denoising methods were selected: Wiener filtering, spectral subtraction, ACC compression, and Speex compression. When calculating the similarity of each frame of the spectrogram, the average similarity between the four simulated clean samples obtained by the four speech denoising methods and the samples before denoising was taken.

3. The method for defending against adversarial examples in speech recognition based on spectral features according to claim 1, characterized in that, The spectral subtraction method used in step 1 is an improved spectral subtraction method, and its noise estimation includes the following steps: Using a speech activity detection method, silence segments are first removed by dividing the speech into energy thresholds. Then, the noise segments in the sample are estimated, and the noise frames and speech frames are distinguished by the zero-crossing rate feature of speech activity detection to find the noise segments. Finally, based on the location of the disturbance distribution, several frames of noise are selected in the high-frequency speech segment, and their average is taken to estimate the noise of the entire segment.

4. The method for adversarial example defense in speech recognition based on spectral features according to claim 1, characterized in that, Step 2 includes the following steps: Step 21: Perform a series of preprocessing operations on the original sample and the obtained simulated clean sample 1, including pre-emphasis, framing, and windowing; Step 22: Convert the speech signals of the original sample and the obtained simulated clean sample 1 from the time domain to the frequency domain to obtain the spectrum. Step 23: Match the features of the simulated clean sample 1 obtained by each perturbation method with the original sample spectrogram features, and calculate the similarity between the corresponding spectrograms to prepare for input into the classifier.

5. The method for defending against adversarial examples in speech recognition based on spectral features according to claim 1, characterized in that, Step 3 includes the following steps: Step 31: Use white-box attack methods and black-box attack methods to attack clean samples to form corresponding adversarial samples; Step 32: Denoise the clean samples and the generated adversarial samples using the same speech de-perturbation method as in Step 1 to obtain the corresponding simulated clean sample 2; Step 33: Perform a series of preprocessing operations on the clean samples, adversarial samples, and the simulated clean samples 2 obtained after processing them, including pre-emphasis, frame segmentation, and windowing. Step 34: Convert the speech signals of the clean sample, the adversarial sample, and the simulated clean sample 2 obtained after their processing from the time domain to the frequency domain to obtain the spectrogram; Step 35: Match the features of the simulated clean sample 2, the clean sample, and the generated adversarial sample spectrum obtained by each perturbation method to one-to-one correspondence. Step 36: Calculate the average of the four correlations between each frame of the clean sample dataset or adversarial sample dataset and each simulated clean sample dataset. Label the resulting multiple pairs of correlation coefficient vectors. Label the vectors obtained from clean samples with clean labels and the vectors obtained from adversarial samples with adversarial labels. Train the classifier in this way. Step 37: First, use a convolutional neural network to extract features from the correlation coefficient vector, and then use the cross-entropy loss function to train the parameters of the convolutional neural network.

6. The method for adversarial example defense in speech recognition based on spectral features according to claim 1, characterized in that, The Pearson correlation coefficient was used to calculate the similarity of the spectrograms.

7. The method for defending against adversarial examples in speech recognition based on spectral features according to claim 1, characterized in that, In step 4, if the classifier identifies an adversarial sample, it is discarded directly; if it identifies a clean sample, it is input into the speech recognition system for the next step, thus completing the entire defense system.

8. A speech recognition adversarial example defense system based on spectral features, characterized in that, include: The original sample denoising module is used to denoise unknown original samples from the outside world using speech denoising methods to obtain simulated clean sample 1; The speech de-perturbation methods include multiple methods such as Wiener filtering, spectral subtraction, ACC compression, and Speex compression. The original sample frequency domain conversion module is used to preprocess the original sample and the obtained simulated clean sample 1, and convert them to the frequency domain to obtain their spectrum characteristics. The classifier training module uses white-box and black-box attacks to obtain adversarial samples from known clean speech samples. It then uses speech denoising methods to denoise both the clean speech samples and the generated adversarial samples, resulting in simulated clean sample 2. The clean speech samples, adversarial samples, and the simulated clean sample 2 are preprocessed and converted to the frequency domain to obtain their spectrogram features. The similarity of the spectrograms of the two types of samples in each frame is compared to obtain their feature vectors. Labels are then added to these vectors to determine whether they are adversarial samples, and this process is used to train the classifier. It also includes a defense detection module, which inputs the similarity feature vector corresponding to the spectrogram obtained by the original sample frequency domain conversion module into the trained classifier to detect whether the original sample is an adversarial sample, thus completing the overall defense framework.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the steps of the speech recognition adversarial example defense method based on spectral features according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the speech recognition adversarial sample defense method based on spectral features according to any one of claims 1-7.

Images