A speech recognition adversarial sample defense method and system based on local distribution smoothing

Abstract

The application discloses a speech recognition adversarial sample defense method and system based on local distribution smoothing. The application obtains original clean samples and generates adversarial samples, and screens out disturbance points with a prediction confidence lower than a preset threshold. A local neighborhood is constructed based on a standard deviation self-adaption with each disturbance point as a center. A slight disturbance is added to sample points in the neighborhood to generate new adversarial samples. The average value of the KL divergence between the disturbance points and the prediction probability distribution of the new adversarial samples is taken as a local distribution smoothing index. A total loss function including an original recognition task loss and a local smoothing loss term is constructed, and a hybrid training set is used to perform adversarial training on a speech recognition model. The application can effectively reduce the word error rate and attack success rate of the model to various attack types, significantly improve the recognition accuracy and robustness of the speech recognition model in an adversarial environment, has good generalization ability and training efficiency, and can be widely applied to a safe and sensitive speech interaction system.

Description

Technical Field

[0001] This invention relates to the fields of speech recognition and artificial intelligence technology, specifically to a speech recognition adversarial example defense method and system based on local distribution smoothing. Background Technology

[0002] Adversarial example defense methods in the field of speech recognition can be mainly divided into two categories: one is to start from the input data end, and the other is to improve the samples or neural networks from the model end.

[0003] In data-driven defense, existing technologies focus on preprocessing or detecting input speech samples to remove perturbations or discard some adversarial examples at the input stage. Common techniques include speech denoising, acoustic feature transformation, and adding additional detection networks. For example, some researchers use convolutional neural networks to treat adversarial example detection as a binary classification problem, using Mel-frequency cepstral coefficients (MFCCs) of speech as input to train a detector capable of distinguishing between benign and adversarial examples. Other methods add reasonable noise before the input speech and compare the consistency of the two decoding results to determine the presence of an attack. Furthermore, there are schemes that combine multiple audio preprocessing methods (such as MP3 compression, bandpass filtering, and audio shifting) with ensemble learning (such as Boosting and random forests) for joint detection. Other researchers utilize the temporal dependence of speech, using the ratio of the longest common prefix length of partial transcription to all transcriptions as a detection metric; or treat adversarial example detection as anomaly pattern detection in the ASR model space, quantifying anomaly scores through statistical methods; or utilize the transcriptional differences of the same audio from different ASR systems, combining multi-version programming ideas and classifiers for adversarial example judgment. In addition, input transformation (such as optimizing sampling and quantization methods) can also defend against disturbances to some extent.

[0004] In terms of model-oriented defense, existing technologies focus on improving the internal structure or training strategies of speech recognition models, including adversarial training, regularization, and model layer enhancement. For example, adversarial training of speaker recognition models is conducted using adversarial examples generated by attack methods such as Projective Gradient Descent (PGD) and Concurrent Wave (CW); the concept of flooding score is proposed to identify adversarial examples by calculating the degree of noise perturbation required to cause the model to misclassify; adversarial training is carried out using Generative Adversarial Networks (GANs) to generate clean pseudo-labels and mix them with noisy labeled data to enhance model robustness; and general adversarial training methods are used to improve the performance of automatic speech recognition models on multiple datasets.

[0005] However, existing technologies still have significant shortcomings. On the one hand, adversarial training methods typically require a large number of adversarial samples, resulting in long training times and high costs. Furthermore, adversarial samples generated for fixed attack methods can only defend against specific attacks. To defend against multiple attacks, multiple methods must be used to generate samples for training, which can easily lead to model overfitting. On the other hand, both data-oriented and model-oriented methods are mostly effective only for specific attack types in the training set. Data-oriented detection methods perform poorly against unseen attack methods; model-oriented adversarial training is mostly focused on single attack types, and its performance drops significantly under multiple attacks. With the continuous emergence of attack methods, the universality of existing defense schemes needs to be improved. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for defending against adversarial examples in speech recognition based on local distribution smoothing, so as to solve the problems of high adversarial training cost, poor generalization ability, and difficulty in effectively dealing with multiple attack types in existing speech recognition adversarial example defense methods.

[0007] To achieve the above objectives, the technical solution provided by this invention is: a method for defending against adversarial examples in speech recognition based on local distribution smoothing, comprising the following steps: S1: Obtain the original clean sample, and generate the original adversarial sample for the target speech recognition model using at least one attack method; calculate the average prediction confidence of the original clean sample and the original adversarial sample misclassified by the target speech recognition model, and analyze their prediction probability distribution characteristics; select sample points with average prediction confidence lower than the preset threshold and conforming to the preset probability distribution characteristics as perturbation points. S2: For each perturbation point, take the position of the perturbation point in the feature space as the center, calculate the distance between the center and other sample points based on the distance metric; determine the neighborhood radius according to a preset multiple of the standard deviation of the data distribution around the perturbation point, and classify the sample points within the neighborhood radius into the local neighborhood of the perturbation point; S3: Add a small perturbation to the sample points in the local neighborhood to generate new adversarial examples; S4: Input the perturbation point and its corresponding new adversarial sample into the target speech recognition model to obtain the original predicted probability distribution of the perturbation point and the perturbation-post-prediction probability distribution of the new adversarial sample; calculate the KL divergence between the original predicted probability distribution and the perturbation-post-prediction probability distribution, and take the average value of the KL divergence to construct a local smoothing loss term. S5: Construct a total loss function, which includes the original speech recognition task loss term and a local smoothing loss term multiplied by a local distribution smoothing index and a weight coefficient; using a mixed training set containing original clean samples, perturbation points, and newly added adversarial samples, perform adversarial training on the target speech recognition model by minimizing the total loss function, and update the parameters of the target speech recognition model.

[0008] To optimize the above technical solution, the specific measures also include: In step S1, the generation of original adversarial samples against the target speech recognition model using at least one attack method includes white-box attack methods and black-box attack methods; The white-box attack method is based on the complete structure and parameter information of the target speech recognition model. It constructs a target loss function that includes adversarial loss terms and correction terms, minimizes the target loss function through an iterative optimization algorithm, adjusts the features of the original clean samples, and generates original adversarial samples. The black-box attack method, without obtaining the gradient and parameter information of the target speech recognition model, uses a surrogate model or zero-order optimization method to estimate the gradient information. Through an evolutionary iterative process of population initialization, fitness evaluation, individual selection and crossover mutation, it gradually modifies the features of the original clean sample to generate the original adversarial sample.

[0009] In step S1, the calculation of the average prediction confidence of the original clean sample and the original adversarial sample being misclassified by the target speech recognition model is specifically as follows: The original clean sample and the original adversarial sample are repeatedly input into the target speech recognition model several times. The maximum value of the Softmax probability corresponding to all the wrong categories when the model misclassifies the sample is taken. The arithmetic mean of all the maximum values ​​of the Softmax probability is taken to obtain the average prediction confidence of the original clean sample and the original adversarial sample.

[0010] In step S2, determining the neighborhood radius based on a preset multiple of the standard deviation of the data distribution around the disturbance point specifically involves multiplying the standard deviation of the sample points around the disturbance point by a preset multiple to obtain the neighborhood radius, where the preset multiple is 1 to 2 times; the distance metric is calculated as Manhattan distance or Euclidean distance.

[0011] In step S3, the methods for adding small perturbations to sample points in the local neighborhood include: adding random noise that follows a Gaussian or uniform distribution; adjusting the feature weights of the sample points; and projecting the features of the sample points into a new feature space through a mapping function.

[0012] In step S4, calculating the KL divergence between the original predicted probability distribution and the perturbated predicted probability distribution specifically involves:

[0013] in, Represents the original predicted probability distribution; This represents the predicted probability distribution after the perturbation; This represents the probability density function corresponding to the original predicted probability distribution; This represents the probability density function corresponding to the predicted probability distribution after the perturbation; The probability density weight represents the original predicted probability distribution, and the differential weight represents the probability of the original distribution at the value x.

[0014] In step S5, the total loss function includes the original speech recognition task loss term and the local smoothing loss term, which is the local distribution smoothing index multiplied by a weighting coefficient. The expression is as follows:

[0015] in, The term employs connectionist temporal classification (CTC) loss; This represents the local distribution smoothing regularization weight coefficient, with a value range of 0.01 to 1; k The number of perturbation samples; This represents the local distribution smoothing index, i.e., the predicted probability distribution of the original clean speech samples. The predicted probability distribution of the speech samples after local perturbation smoothing. The KL divergence between them.

[0016] As another important technical solution, the present invention also provides a speech recognition adversarial example defense system based on local distribution smoothing, comprising: The perturbation point screening module is used to obtain original clean samples and generate original adversarial samples against the target speech recognition model using at least one attack method; calculate the average prediction confidence of the original clean samples and original adversarial samples misclassified by the target speech recognition model, and analyze their prediction probability distribution characteristics; select sample points with average prediction confidence below a preset threshold and conforming to preset probability distribution characteristics as perturbation points. The local neighborhood construction module is used to calculate the distance between each perturbation point and other sample points based on the distance metric, with the position of the perturbation point in the feature space as the center; determine the neighborhood radius according to a preset multiple of the standard deviation of the data distribution around the perturbation point; and classify the sample points within the neighborhood radius into the local neighborhood of the perturbation point. The neighborhood augmentation adversarial example generation module is used to add small perturbations to sample points in a local neighborhood to generate new adversarial examples. The local distribution smoothing index calculation module is used to input the perturbation point and its corresponding new adversarial sample into the target speech recognition model to obtain the original predicted probability distribution of the perturbation point and the perturbation-post-prediction probability distribution of the new adversarial sample; calculate the KL divergence between the original predicted probability distribution and the perturbation-post-prediction probability distribution, and take the average value of the KL divergence to construct the local smoothing loss term. The hybrid adversarial training optimization module is used to construct the total loss function, which includes the original speech recognition task loss term and the local smoothing loss term multiplied by the local distribution smoothing index and the weight coefficient. Using a hybrid training set containing the original clean samples, perturbation points and newly added adversarial samples, the module performs adversarial training on the target speech recognition model by minimizing the total loss function and updates the parameters of the target speech recognition model.

[0017] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a speech recognition adversarial example defense method based on local distribution smoothing as described above.

[0018] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute a speech recognition adversarial example defense method based on local distribution smoothing as described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention effectively reduces the recognition error rate caused by adversarial examples by applying distribution smoothing constraints to the local neighborhood of the perturbation point and combining it with a hybrid adversarial training strategy. Speech recognition models trained by this method can maintain a high recognition accuracy, and the word error rate is significantly reduced compared with traditional adversarial training methods, greatly enhancing the robustness of the model.

[0020] This invention is not only applicable to single attacks (such as white-box or black-box attacks), but can also defend against combined attacks of multiple attack methods. It exhibits stable defense performance under different attack methods (including CW, FGSM, PGD, ATN, GAGE, etc.), and when multiple attacks are combined, the decrease in recognition accuracy is much smaller than that of existing methods, demonstrating excellent generalization ability.

[0021] After adversarial training according to the present invention, the success rate of attackers is greatly suppressed when facing various adversarial samples, the distortion of the model output from the original input is significantly reduced, and the model has stronger anti-interference ability while maintaining normal function, effectively mitigating the damage of adversarial disturbances to the decision boundary.

[0022] This invention constructs local neighborhoods and applies smoothing constraints only for the selected perturbation points, without requiring extensive perturbation calculations on the entire dataset. This significantly reduces the computational overhead of adversarial training, effectively prevents model overfitting, ensures stable training convergence, and ultimately enables the model to be efficiently deployed in real-world speech recognition systems. Attached Figure Description

[0023] Figure 1 : Flowchart from an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0025] The following is an explanation of some of the terms used in this invention: Adversarial examples: Special samples that add subtle perturbations to the original speech samples, which are difficult for humans to detect, causing speech recognition models to misjudge.

[0026] Speech recognition: A technique that converts human speech signals into text, based on deep learning models.

[0027] Spectral features: The characteristics of a speech signal in the frequency domain after Fourier transform, including the spectrogram, Mel-frequency cepstral coefficients (MFCC), etc.

[0028] Local Distribution Smoothing (LDS): A technique that optimizes the decision boundary of a model by adjusting the distribution characteristics of data in the neighborhood of a perturbation point.

[0029] White-box attack: An attack method in which the attacker knows the model structure and parameters (such as CW attack, FGSM attack).

[0030] Black-box attack: An attack method in which the attacker infers the model only from the input and output (such as GAGE ​​attack, ATN attack).

[0031] Speech Deperturbation: A processing method that reduces adversarial perturbations in speech through techniques such as filtering and compression (e.g., Wiener filtering, VAD spectral subtraction).

[0032] The technical solution of the present invention will be further described in detail below with reference to specific embodiments: In some implementations, such as Figure 1 As shown, this invention provides a speech recognition adversarial example defense method based on local distribution smoothing, comprising the following steps: S1: Obtain the original clean sample, and generate the original adversarial sample for the target speech recognition model using at least one attack method; calculate the average prediction confidence of the original clean sample and the original adversarial sample misclassified by the target speech recognition model, and analyze their prediction probability distribution characteristics; select sample points with average prediction confidence lower than the preset threshold and conforming to the preset probability distribution characteristics as perturbation points. In some implementations, a dataset is acquired, original clean samples are selected, and various white-box and black-box attack methods such as CW, FGSM, PGD, ATN, and GAGE ​​are used to generate audio adversarial samples for the Deep Speech speech model.

[0033] The Deep Speech model is an excellent deep learning-based engine for handling speech-to-text problems. It can transcribe speech recordings into written text and obtain the best results from cleanly recorded speech under optimal conditions.

[0034] Attack methods include white-box attack methods and black-box attack methods.

[0035] In some implementations, white-box attacks, taking CW as an example, rely on model structure and parameter information to construct a target loss function consisting of adversarial loss terms and correction terms. :

[0036] Among them, the counter-loss item The correction term forces the model to misclassify adversarial examples as the target category. The adversarial loss term measures the difference between the adversarial example and the original speech, limits the perturbation amplitude, and ensures the concealment of the adversarial example. It measures how the target model misclassifies the data and real data, minimizing the distance between the input and the target output to cause the target model to misclassify the adversarial example. The correction term maintains perceptual consistency between the adversarial example and the original audio by limiting the difference between them.

[0037] In some implementations, the target loss function is minimized through iterative optimization algorithms, such as optimizers or gradient descent: original adversarial examples are generated by adjusting the features of the input audio while maintaining the aforementioned distance constraints, and by adjusting attack parameters, such as perturbation size, attack step size, and number of iterations, to balance the robustness and identifiability of the adversarial examples.

[0038] In some implementations, black-box attacks, using GAGE ​​as an example, involve selecting one or more other models (e.g., surrogate models) as black-box models when the gradients and parameters of the target model are unavailable. First, the target DeepSpeech speech model is determined, whose parameters or gradient information are inaccessible to the attacker. An original audio sample is selected as the starting point and modified into an adversarial example. A genetic algorithm is used to randomly generate a set of initial individuals, each representing a modification scheme to the initial sample, such as changing some spectral features of the audio. Gradient estimation methods, such as ZOO (Zeroth Order), are employed. The optimization algorithm estimates the gradient information of each individual with respect to the objective function; using the estimated gradient information, it calculates the degree to which each individual deceives the target model, and uses the output of the target model to evaluate the fitness of each individual; based on the fitness of the individuals, it selects the better individuals as the basis for the next generation population, and performs crossover and mutation on the selected excellent individuals to generate the next generation population. Finally, it repeats the above process iteratively. In each generation, gradient estimation, individual fitness evaluation, selection of excellent individuals, crossover and mutation are performed until the attacker's stopping condition is met, such as reaching the maximum number of iterations or generating original adversarial samples that meet the attack target.

[0039] Both white-box and black-box attacks induce Deep Speech models to make misjudgments by superimposing tiny audio perturbations, which is the prerequisite for the robust defense of the model in this study.

[0040] In some implementations, after obtaining the original adversarial examples, the model's misclassification is evaluated. The generated original adversarial examples are then input into the target model or other models, and the prediction results are recorded. The labels of the previous original adversarial examples or clean original examples are retained, ensuring that the model's paired results for each sample are recorded. The model's prediction results are compared with the true labels to determine which samples are misclassified. For each Deep Speech model experiment, samples are labeled based on the mismatch between the predicted and true labels to determine the misclassification situation. Through multiple rounds of repeated experiments, the misclassification rate is statistically analyzed to eliminate experimental bias interference and identify high-frequency, error-prone sample points that are consistently misclassified multiple times.

[0041] In some implementations, the original clean sample and the original adversarial sample are repeatedly input into the target speech recognition model at least three times to eliminate random errors in a single inference. In each inference, the maximum value of the Softmax probability corresponding to all misclassified categories when the model misclassifies the sample is taken, and the arithmetic mean of all the maximum values ​​of the Softmax probability is taken to obtain the average prediction confidence of the original clean sample and the original adversarial sample, respectively.

[0042] The probability distribution characteristics of the adversarial perturbation point are that the model output probability distribution is highly flat, with only the probability of the target misclass being slightly higher than that of other classes. The overall confidence is extremely low, indicating that the model is extremely hesitant in classifying the sample, which is the decision boundary that the model is most likely to breach.

[0043] In some implementations, the average false positive confidence level is used as the core indicator, and sample points that meet the probability of less than 0.1 and conform to the above distribution characteristics are retained as perturbation points. As shown in Table 1, taking the CW attack results as an example, the average false positive confidence level of the selected 5 groups of perturbation points is less than 0.1, which meets the selection criteria for perturbation points that the model is prone to misjudgment.

[0044] Table 1. Confidence Scoring Table for Perturbation Points

[0045] S2: For each perturbation point, take the position of the perturbation point in the feature space as the center, calculate the distance between the center and other sample points based on the distance metric; determine the neighborhood radius according to a preset multiple of the standard deviation of the data distribution around the perturbation point, and classify the sample points within the neighborhood radius into the local neighborhood of the perturbation point; Defining a local neighborhood for each selected perturbation point avoids perturbation calculations on the entire dataset and effectively reduces the computational complexity of model training. Furthermore, since the perturbation points are different each time, the neighborhood range needs to be recalculated adaptively.

[0046] In some implementations, the upper and lower bounds of the feature space are determined by taking the location of the perturbation point in the feature space as the center and selecting a fixed multiple of the standard deviation as the neighborhood radius. The upper bound is obtained by adding a multiple of the standard deviation to the feature value of the perturbation point and multiplying it by the standard deviation. The lower bound is obtained by subtracting a multiple of the standard deviation from the feature value of the perturbation point and multiplying it by the standard deviation.

[0047] The formula for calculating standard deviation is:

[0048] Where Σ represents summation, Represents each sample point, This represents the average value of the data. The standard deviation indicates the number of sample points. A larger standard deviation indicates a higher degree of dispersion of the sample points relative to the mean; a smaller standard deviation indicates a lower degree of dispersion of the sample points relative to the mean.

[0049] After determining the range or distance of the neighborhood, it is determined which sample points will be included in this local neighborhood. For the perturbation point, the Manhattan distance between it and other sample points in the dataset is calculated. Based on the selected neighborhood range, all sample points within a certain range of the calculated distance around the perturbation point are selected as members of the local neighborhood.

[0050] In some implementations, the determined local neighborhood is fine-tuned to ensure that the number of samples in the neighborhood is sufficient and the feature distribution is reasonable, taking into account both the integrity of local features and the convergence of the range, and avoiding the loss of local distribution characteristics due to an excessively large neighborhood.

[0051] S3: Add a small perturbation to the sample points in the local neighborhood to generate new adversarial examples; In some implementations, in order to unify the characteristics of neighborhood perturbation, expand data diversity, and improve the model's generalization and anti-interference capabilities, controllable small perturbations are uniformly introduced into the samples within the neighborhood. Specifically, this includes: adding random noise to the features of the sample points to increase the diversity and robustness of the data; selecting noise with different distributions, such as Gaussian noise or uniform noise, and controlling the magnitude of the noise. Change the feature weights on the sample points to make them have a greater impact on the model's decisions; increase or decrease the weights of certain features to adjust their relative importance in the subsequent training and reclassification process; By employing appropriate mapping transformations, the positions of sample points in the feature space can be altered, thereby increasing the diversity of the feature space. Linear or nonlinear transformation methods can be used for feature space mapping.

[0052] In some implementations, after uniformly applying perturbation to the perturbation point and its neighboring samples, new adversarial samples are generated. The new adversarial samples and the original clean samples are used together as the basis data for subsequent local distribution smoothing and adversarial training, ensuring that the Deep Speech model can effectively adapt to slight audio distortions after training, thereby improving the robustness of speech transcription.

[0053] S4: Input the perturbation point and its corresponding new adversarial sample into the target speech recognition model to obtain the original predicted probability distribution of the perturbation point and the perturbation-post-prediction probability distribution of the new adversarial sample; calculate the KL divergence between the original predicted probability distribution and the perturbation-post-prediction probability distribution, and take the average value of the KL divergence to construct a local smoothing loss term. The original clean sample data yields preliminary decision results and probability distributions based on the predictions of the Deep Speech model. Combined with the corresponding labels, some perturbation points and their neighboring sample points are perturbed, generating perturbed sample points. These perturbed sample points will serve as new inputs to calculate the perturbed prediction results and probability distributions, resulting in entirely new decision results and probability distributions.

[0054] In some implementations, for each set of original sample points and corresponding perturbed sample points, this paper uses the KL divergence to quantify the deviation between the two sets of predicted probability distributions. When the two distributions are completely identical, the calculated KL divergence is 0. The original predicted probability distribution is denoted as... The predicted probability distribution after perturbation is denoted as Given continuous random variables, their probability density functions are respectively and If based on distribution To approximate the distribution The corresponding KL divergence calculation formula is:

[0055] in, This represents the probability density function corresponding to the original predicted probability distribution; This represents the probability density function corresponding to the predicted probability distribution after the perturbation; The probability density weights represent the original predicted probability distribution, and the differential weights of the original distribution at the value x are also represented. The integral term represents the operation performed in the entire value space of the random variable, and the logarithmic term is used to characterize the ratio of the two probability density functions. The overall numerical value intuitively reflects the magnitude of the difference between the distributions.

[0056] After calculating the KL divergence between the original distribution and the neighborhood perturbed sample distribution of each sample point, the average of all KL divergences within the neighborhood is used to construct a local smoothing loss term, specifically: For each sample, generate within its local neighborhood k For each perturbed sample, calculate the KL divergence between the original distribution and the distribution corresponding to each perturbed sample; average the KL divergences within the neighborhood to obtain the local smoothing index for that sample. During the training of the model against adversarial examples, this loss term is included in the total loss function and participates in iterative optimization, thereby constraining the output distribution of the model and improving the stability and robustness of the prediction results.

[0057] S5: Construct a total loss function, which includes the original speech recognition task loss term and a local smoothing loss term multiplied by a local distribution smoothing index and a weight coefficient; using a mixed training set containing original clean samples, perturbation points, and newly added adversarial samples, perform adversarial training on the target speech recognition model by minimizing the total loss function, and update the parameters of the target speech recognition model.

[0058] In some implementations, a targeted local smoothing loss term is introduced to effectively improve the model's generalization ability and robustness in complex speech scenarios. The training process no longer relies solely on the basic loss of the original speech recognition task, but instead adopts a weighted fusion approach, combining the original speech recognition task loss term with the local smoothing loss term to construct a novel total loss function adapted for adversarial training of the model.

[0059] The specific weighted fusion method involves setting learnable balance weight coefficients to assign weights to the loss term of the original speech recognition task. These weights guide the model to learn speech sequence features and fit real labels, ensuring that the accuracy of basic speech recognition and the local smoothing loss term are weighted and linearly superimposed to obtain the total loss function. This dynamically balances the accuracy of the main task with the strength of the regularization constraint, achieving adaptive adjustment of the two losses and thus adaptive fusion adjustment of the two losses.

[0060] The total loss function comprises the original speech recognition task loss term and a regularization term consisting of a local distribution smoothing metric multiplied by weight coefficients, and is expressed as:

[0061] in, The term employs connectionist temporal classification (CTC) loss; This represents the local distribution smoothing regularization weight coefficient, with a value range of 0.01 to 1.

[0062] In some implementations, the total loss function plays a dual optimization role. On the one hand, it inherits the core function of the original speech recognition loss, guiding the model to learn speech sequence features and fit real labels, ensuring the model's basic speech recognition accuracy and task adaptability. On the other hand, relying on the regularization constraint effect of the local smoothing loss term, it standardizes the model's prediction behavior in the neighborhood of the input data, making up for the deficiency of single-task loss in resisting constraints, and guiding the model to weaken the prediction fluctuations caused by small input perturbations from the perspective of loss optimization. Adjustable weight coefficients can accurately balance the training proportion and constraint strength of the two losses in the model iteration optimization process. This can ensure the model's basic fitting ability for speech and text recognition, avoid excessive interference of smoothing loss in the learning of the main task, and give full play to the regularization effect of LDS loss, constraining the stability of the prediction distribution in the model's input neighborhood, and solving the problems of traditional speech models being easily disturbed by small perturbations and sensitive to decision boundaries.

[0063] In the specific training iteration process, this paper adopts a hybrid training method of clean normal speech samples and constructed adversarial speech samples to alternately update the network parameters. This allows the model to continuously perceive feature changes caused by small input perturbations while learning the features of real speech data and completing the recognition task. By continuously minimizing the fused total loss function, the model's classification decision boundary is continuously corrected and optimized, making it smoother and more robust. This effectively reduces the model's sensitivity to input noise and adversarial perturbations, and suppresses overfitting. To ensure the model's training effect and efficiency, when the fused total loss function no longer shows a significant downward trend after several rounds of continuous training, tends to stabilize, and enters the loss plateau period, it proves that the model parameters have fully converged, and the network's feature learning and regularization optimization have reached their optimal state. At this point, the training iteration is immediately stopped to avoid model performance degradation and resource waste caused by overtraining, ultimately obtaining a speech recognition model with both excellent recognition accuracy and strong adversarial robustness.

[0064] In some implementations, the best way to verify applications in speech recognition is through speech-to-text conversion. This is also the direct result of adversarial examples affecting the model. Therefore, the word error rate (BER) will be used first to evaluate the defensive effectiveness of the method. The BER represents the degree of difference between the recognized result and the reference text, and is a commonly used evaluation metric. It indicates the degree of difference between the recognized result and the reference text, usually expressed as a percentage. The better the accuracy of the speech recognition system, the lower the BER.

[0065] For adversarial training using a single attack method, a comparison of single methods was conducted for multiple adversarial training methods. CW attack, FGSM attack, PGD attack, ATN attack, and GAGE ​​attack were used as benchmark methods. The proposed method was then used to train against these five attacks for comparison. Five sets of experiments were conducted, and the average value was taken. The word error rate after adversarial attack, the word error rate after adversarial training, the reduction in word error rate, and the model accuracy were used as evaluation criteria, as shown in Table 2. Table 2. Single Method Accuracy Table

[0066] Experimental results show that this invention significantly outperforms traditional adversarial training under various attack types. Under white-box attacks (CW, FGSM, PGD), the word error rate is reduced to approximately 5%, and the model accuracy exceeds 94%, representing an improvement of approximately 4-6% compared to existing methods. Under black-box attacks (ATN, GAGE), the word error rate is reduced to approximately 9%-10%, and the accuracy reaches approximately 90%, representing an improvement of approximately 4-5% compared to existing methods. Overall, the adversarial training method of this invention effectively improves model robustness and recognition accuracy under different attack types, demonstrating significant defensive capabilities.

[0067] The method of this invention is compared with the General Adversarial Training (GPAT) method. The comparison is made from the perspective of two datasets. Since GPAT has its own way of generating perturbations, the best result of this method in the two datasets is selected for comparison. That is, the white-box attack method is combined with the LibriSpeech corpus for further experimentation. For the Mozilla Common Voice dataset, it is assumed that the word error rate of the white-box attack method after training is better than that of the black-box attack method, so the experiment is not repeated, as shown in Table 3.

[0068] Table 3 Comparison of Adversarial Training Methods

[0069] Experimental results show that, compared with the current mainstream adversarial training method GPAT, on the Mozilla CommonVoice dataset, the error rate of this invention is about 1.1% lower than that of GPAT, and the number of word errors is reduced by about 2.2%, thus improving the defense effect to a certain extent.

[0070] For adversarial training using two attack methods, the success rate of reproducing complex adversarial training using both attack methods is significantly lower than that of training individual adversarial methods, and the success rate is still relatively low. This invention utilizes perturbation points from different attack methods to achieve local distribution smoothing, which can also train multiple attack methods. The invention combines three white-box attack methods in pairs and compares them separately, as shown in Table 4.

[0071] Table 4. Accuracy of Multiple Methods

[0072] Experimental results show that for adversarial training combining CW and PGD attacks, the word error rate of this invention can be reduced by more than 15%, and the accuracy of the model after training reaches nearly 85%; for adversarial training combining CW and FGSM attacks, the word error rate of this method can be reduced by more than 17%, and the accuracy of the model after training reaches nearly 83%; for adversarial training combining PGD and FGSM attacks, the word error rate of this invention can be reduced by more than 22%, and the accuracy of the model after training reaches nearly 83%, all of which have achieved very good results.

[0073] Through single-method and multi-method word error rate experiments, it can be seen that adversarial training reduces the word error rate and improves the model's accuracy, indicating that the adversarial training method based on local distribution smoothing can effectively enable the trained model to have better speech recognition accuracy. The performance of the model before and after training will be compared below to demonstrate its robustness. The success rate and distortion rate of the adversarial attack method will be used to verify the effectiveness of the defense.

[0074] The success rate metric of adversarial attack methods reflects the attacker's ability to successfully generate adversarial samples and deceive the model. A higher success rate indicates a poorer robustness of the model against adversarial samples. A high success rate may mean that the model is vulnerable to adversarial attacks and lacks robustness, as shown in Table 5. Table 5 Comparison of Attack Success Rates Before and After Adversarial Training

[0075] Experimental results show that the success rate of the model after adversarial training in this invention is significantly reduced. The success rate of white-box attacks is reduced to below 10%, a decrease of about 85%, which is nearly 10% lower than adversarial training with a specified attack method. The success rate of black-box attacks is reduced to about 13%, a decrease of about 75%, which is nearly 12% lower than adversarial training with a specified attack method.

[0076] The distortion rate reflects the degree of difference between adversarial samples and original samples. A lower distortion rate indicates that the model can maintain good classification performance under adversarial attacks, identify the original features of adversarial samples and classify them accurately, as shown in Table 6. Table 6. Comparison of Distortion Rates Before and After Adversarial Training

[0077] Experimental results show that the model distortion rate is significantly reduced after adversarial training according to the present invention. The distortion rate of the white-box method is reduced to about 5%, while the original distortion rate of the model used in the experiment is also 5%-10%, indicating excellent performance. The distortion rate is also reduced by about 3% compared to the adversarial training of the specified attack method. The distortion rate of the black-box method is reduced to about 10%, while the original distortion rate of the model used in the experiment is around 12%, which is also reduced by about 2% compared to the adversarial training of the specified attack method.

[0078] The above experiments, by comparing the changes in attack success rate and model distortion rate before and after model training, demonstrate that the defense achieved good results. Measuring the model distortion rate, i.e., the degree to which the model output is perturbed, allows us to assess whether adversarial training affected the model's normal function. The success rate of model attacks reflects the model's performance in the face of specific attacks. This indicates that the proposed method ensures both accuracy in speech recognition and robustness of the model, demonstrating good defensive effectiveness.

[0079] In another embodiment of the present invention, a speech recognition adversarial example defense system based on local distribution smoothing is proposed, comprising: The perturbation point screening module is used to obtain original clean samples and generate original adversarial samples against the target speech recognition model using at least one attack method; calculate the average prediction confidence of the original clean samples and original adversarial samples misclassified by the target speech recognition model, and analyze their prediction probability distribution characteristics; select sample points with average prediction confidence below a preset threshold and conforming to preset probability distribution characteristics as perturbation points. The local neighborhood construction module is used to calculate the distance between each perturbation point and other sample points based on the distance metric, with the position of the perturbation point in the feature space as the center; determine the neighborhood radius according to a preset multiple of the standard deviation of the data distribution around the perturbation point; and classify the sample points within the neighborhood radius into the local neighborhood of the perturbation point. The neighborhood augmentation adversarial example generation module is used to add small perturbations to sample points in a local neighborhood to generate new adversarial examples. The local distribution smoothing index calculation module is used to input the perturbation point and its corresponding new adversarial sample into the target speech recognition model to obtain the original predicted probability distribution of the perturbation point and the perturbation-post-prediction probability distribution of the new adversarial sample; calculate the KL divergence between the original predicted probability distribution and the perturbation-post-prediction probability distribution, and take the average value of the KL divergence to construct the local smoothing loss term. The hybrid adversarial training optimization module is used to construct the total loss function, which includes the original speech recognition task loss term and the regularization term of the local distribution smoothing index multiplied by the weight coefficient. Using a hybrid training set containing original clean samples, perturbation points and newly added adversarial samples, the module performs adversarial training on the target speech recognition model by minimizing the total loss function, and updates the parameters of the target speech recognition model.

[0080] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a speech recognition adversarial example defense method based on local distribution smoothing as described above.

[0081] In another embodiment of the present invention, a computer-readable storage medium is provided storing a computer program that causes a computer to execute a speech recognition adversarial example defense method based on local distribution smoothing as described above.

[0082] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A speech recognition adversarial example defense method based on local distribution smoothing, characterized in that, Includes the following steps: S1: Obtain the original clean sample, and generate the original adversarial sample against the target speech recognition model using at least one attack method; calculate the average prediction confidence of the original clean sample and the original adversarial sample being misclassified by the target speech recognition model, and analyze their prediction probability distribution characteristics. Sample points with an average prediction confidence level lower than a preset threshold and conforming to a preset probability distribution characteristic are selected as perturbation points; S2: For each perturbation point, take the position of the perturbation point in the feature space as the center, calculate the distance between the center and other sample points based on the distance metric; determine the neighborhood radius according to a preset multiple of the standard deviation of the data distribution around the perturbation point, and classify the sample points within the neighborhood radius into the local neighborhood of the perturbation point; S3: Add a small perturbation to the sample points in the local neighborhood to generate new adversarial examples; S4: Input the perturbation point and its corresponding new adversarial sample into the target speech recognition model to obtain the original predicted probability distribution of the perturbation point and the perturbation-post-prediction probability distribution of the new adversarial sample; calculate the KL divergence between the original predicted probability distribution and the perturbation-post-prediction probability distribution, and take the average value of the KL divergence to construct a local smoothing loss term. S5: Construct a total loss function, which includes the original speech recognition task loss term and a local smoothing loss term multiplied by a local distribution smoothing index and a weight coefficient; using a mixed training set containing original clean samples, perturbation points, and newly added adversarial samples, perform adversarial training on the target speech recognition model by minimizing the total loss function, and update the parameters of the target speech recognition model.

2. The method for defending against adversarial examples in speech recognition based on local distribution smoothing according to claim 1, characterized in that: In step S1, the generation of original adversarial samples against the target speech recognition model using at least one attack method includes white-box attack methods and black-box attack methods; The white-box attack method is based on the complete structure and parameter information of the target speech recognition model. It constructs a target loss function that includes adversarial loss terms and correction terms, minimizes the target loss function through an iterative optimization algorithm, adjusts the features of the original clean samples, and generates original adversarial samples. The black-box attack method, without obtaining the gradient and parameter information of the target speech recognition model, uses a surrogate model or zero-order optimization method to estimate the gradient information. Through an evolutionary iterative process of population initialization, fitness evaluation, individual selection and crossover mutation, it gradually modifies the features of the original clean sample to generate the original adversarial sample.

3. The method for defending against adversarial examples in speech recognition based on local distribution smoothing according to claim 1, characterized in that: In step S1, the calculation of the average prediction confidence of the original clean sample and the original adversarial sample being misclassified by the target speech recognition model is specifically as follows: The original clean sample and the original adversarial sample are repeatedly input into the target speech recognition model several times. The maximum value of the Softmax probability corresponding to all the wrong categories when the model misclassifies the sample is taken. The arithmetic mean of all the maximum values ​​of the Softmax probability is taken to obtain the average prediction confidence of the original clean sample and the original adversarial sample.

4. The speech recognition adversarial example defense method based on local distribution smoothing according to claim 1, characterized in that: In step S3, the methods for adding small perturbations to sample points in the local neighborhood include: adding random noise that follows a Gaussian or uniform distribution; adjusting the feature weights of the sample points; and projecting the features of the sample points into a new feature space through a mapping function.

5. The speech recognition adversarial example defense method based on local distribution smoothing according to claim 1, characterized in that: In step S4, calculating the KL divergence between the original predicted probability distribution and the perturbated predicted probability distribution specifically involves: in, Represents the original predicted probability distribution; This represents the predicted probability distribution after the perturbation; This represents the probability density function corresponding to the original predicted probability distribution; This represents the probability density function corresponding to the predicted probability distribution after the perturbation; The probability density weight represents the original predicted probability distribution, and the differential weight represents the probability of the original distribution at the value x.

6. The method for defending against adversarial examples in speech recognition based on local distribution smoothing according to claim 1, characterized in that: In step S5, the total loss function includes the original speech recognition task loss term and the local smoothing loss term, which is the local distribution smoothing index multiplied by a weighting coefficient. The expression is as follows: in, Represents the total loss function; The loss term for the original speech recognition task is represented by connectionist temporal classification loss. This represents the weighting coefficients for local distribution smoothing regularization; k The number of perturbation samples; Indicates a local distribution smoothing index; Represents the original predicted probability distribution; This represents the predicted probability distribution after the perturbation.

7. The method for defending against adversarial examples in speech recognition based on local distribution smoothing according to claim 1, characterized in that: In step S2, determining the neighborhood radius based on a preset multiple of the standard deviation of the data distribution around the disturbance point specifically involves multiplying the standard deviation of the sample points around the disturbance point by a preset multiple to obtain the neighborhood radius, where the preset multiple is 1 to 2 times; the distance metric is calculated as Manhattan distance or Euclidean distance.

8. A speech recognition adversarial example defense system based on local distribution smoothing, characterized in that, include: The perturbation point filtering module is used to obtain original clean samples and generate original adversarial samples against the target speech recognition model using at least one attack method. The average prediction confidence of the original clean sample and the original adversarial sample misclassified by the target speech recognition model were calculated respectively, and the prediction probability distribution characteristics were analyzed. Sample points with an average prediction confidence level lower than a preset threshold and conforming to a preset probability distribution characteristic are selected as perturbation points; The local neighborhood construction module is used to calculate the distance between each perturbation point and other sample points based on the distance metric, with the position of the perturbation point in the feature space as the center; determine the neighborhood radius according to a preset multiple of the standard deviation of the data distribution around the perturbation point; and classify the sample points within the neighborhood radius into the local neighborhood of the perturbation point. The neighborhood augmentation adversarial example generation module is used to add small perturbations to sample points in a local neighborhood to generate new adversarial examples. The local distribution smoothing index calculation module is used to input the perturbation point and its corresponding new adversarial sample into the target speech recognition model to obtain the original predicted probability distribution of the perturbation point and the perturbation-post-prediction probability distribution of the new adversarial sample; calculate the KL divergence between the original predicted probability distribution and the perturbation-post-prediction probability distribution, and take the average value of the KL divergence to construct the local smoothing loss term. The hybrid adversarial training optimization module is used to construct the total loss function, which includes the original speech recognition task loss term and the local smoothing loss term multiplied by the local distribution smoothing index and the weight coefficient. Using a hybrid training set containing the original clean samples, perturbation points and newly added adversarial samples, the module performs adversarial training on the target speech recognition model by minimizing the total loss function and updates the parameters of the target speech recognition model.

9. An electronic device, characterized in that, include: The present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a speech recognition adversarial example defense method based on local distribution smoothing as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute a speech recognition adversarial example defense method based on local distribution smoothing as described in any one of claims 1 to 7.

Images