Speech recognition adversarial sample generation method and device based on phoneme perturbation

Abstract

The present application relates to a phoneme perturbation-based speech recognition adversarial sample generation method, and also relates to a phoneme perturbation-based speech recognition adversarial sample generation device, and belongs to the technical field of deep learning. The method comprises the following steps: obtaining a sampling sample from an audio data set; injecting phoneme noise into different audio segments of the sampling sample in a sliding manner to generate an adversarial sample. The present application can not only improve the security and stability of a speech recognition algorithm model, but also protect user speech privacy.

Description

Technical Field

[0001] This invention relates to a method for generating adversarial examples for speech recognition based on phoneme perturbation, and also to a device for generating adversarial examples for speech recognition based on phoneme perturbation, belonging to the field of deep learning technology. Background Technology

[0002] Deep learning-based speech recognition is a crucial research area in artificial intelligence, encompassing fields such as automatic speech recognition (ASR) and voiceprint recognition (VPR). Currently, almost all ASR systems utilize deep neural network architectures. However, despite significant achievements and positive results, research has demonstrated that deep neural networks are highly susceptible to artificially designed adversarial examples, posing security challenges to their application in scenarios with high reliability requirements.

[0003] Automatic speech recognition systems are used in high-security scenarios such as voiceprint identification in banks, voice control in various smart scenarios, and autonomous driving. Therefore, the security of speech recognition systems is of paramount importance. By adding imperceptible adversarial perturbations to the voice input, the model can be made to output incorrect classification predictions, leading to errors in model inference.

[0004] Furthermore, real-world audio recognition applications can be misused for malicious social purposes; for example, mobile apps may steal users' audio information to analyze their behavior. Therefore, the need to protect user privacy has been increasing and gaining widespread attention in recent years. As an attack method that influences model decisions, adversarial examples offer a new option for user privacy protection. By conducting research on adversarial attacks, adversarial noise with general offensive capabilities can be generated, thereby misleading the intelligent models used by eavesdroppers to analyze speech. This can provide a new and feasible solution for privacy protection in the era of deep learning, attracting widespread attention from researchers.

[0005] Against this backdrop, researching and developing adversarial examples for speech recognition is of great significance for applications in intelligent speech recognition. Besides testing the security and stability of existing deep learning speech recognition models, it can also evaluate the security and stability of existing models in more demanding scenarios (such as the physical world), facilitating the optimization of deep learning-based speech recognition models and improving their security and stability. Simultaneously, this perturbation can also protect speech data from eavesdropping in security scenarios. It can interfere with the eavesdropper's transcription of audio, making the information they obtain only discernible by human hearing, increasing the difficulty for the eavesdropper, and thus representing a feasible speech privacy protection solution. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a method and device for generating adversarial examples for speech recognition based on phoneme perturbation, which can not only improve the security and stability of speech recognition algorithm models, but also protect users' voice privacy.

[0007] To achieve the above technical objectives, on the one hand, the present invention provides a method for generating adversarial examples for speech recognition based on phoneme perturbation, comprising:

[0008] Obtain sampled data from the audio dataset;

[0009] Adversarial examples are generated by injecting phoneme noise into different audio segments of the sampled data in a sliding manner.

[0010] Preferably, obtaining sampling samples from the audio dataset specifically includes:

[0011] Audio samples whose phoneme density meets the preset conditions are selected from the audio dataset as sampling samples; the phoneme density is the number of phonemes in the audio sample per unit length.

[0012] Preferably, the step of selecting audio samples whose phoneme density meets preset conditions from the audio dataset as sampling samples specifically includes:

[0013] Calculate the phoneme density of each audio sample in the audio dataset;

[0014] The average phoneme density of the audio dataset is calculated using the phoneme density of each audio sample.

[0015] Audio samples whose phoneme density differs from the average phoneme density by no more than the phoneme density threshold are selected as the selected samples.

[0016] Select samples whose length and amplitude are both greater than the corresponding preset threshold are used as sampling samples.

[0017] Preferably, the step of injecting phoneme noise into different audio segments of the sampled samples in a sliding manner to generate adversarial examples specifically includes:

[0018] Based on deep learning, phoneme noise is optimized by injecting it into different audio segments of the sampled samples in a sliding manner, and the output phonemes are used to combat noise.

[0019] Adversarial samples are obtained by combining adversarial noise with ordinary audio samples.

[0020] Preferably, the method of optimizing phoneme noise based on deep learning by injecting phoneme noise into different audio segments of the sampled samples in a sliding manner specifically includes:

[0021] For each optimization iteration, phoneme noise is injected into different audio segments of the sampled samples with a variable sliding step size;

[0022] For the sampled samples with injected phoneme noise, the phoneme noise is optimized using a loss function.

[0023] Preferably, the phoneme noise is injected into the sampling samples with a variable sliding step size, which can be expressed as:

[0024]

[0025] In formula (7), This represents the phoneme noise injection module; clip(·) represents the clipping function, which clips the input sample x into a segment of length ba based on the clipping interval [a, b], where a is the clipping start time point and b is the clipping end time point; It is a concatenation function; n is the iteration number, l x It is the length of sample x during training. It is phoneme noise δ p The length of the sliding window is β, where β is the step size of the sliding window.

[0026] Preferably, the optimization process can be expressed as follows:

[0027]

[0028] In formula (8), For CTC loss function, This represents a speech recognition model. For RIR functions, ||·|| ∞ It is the infinite norm, min means minimization, X represents the sampled dataset, and ∈ represents the constraint value.

[0029] Preferably, the expression for selecting samples is:

[0030]

[0031] In formula (5), D represents the average phoneme density of the audio dataset. The phoneme density of the audio sample is represented by counter(·), ψ(·) represents the phoneme recognizer function, α represents the phoneme density threshold, and x represents the audio sample. x Indicates the length of the audio sample.

[0032] On the other hand, the speech recognition adversarial example generation device based on phoneme perturbation provided by the present invention includes:

[0033] Selecting units is used to obtain sampled data from the audio dataset;

[0034] The generation unit is used to inject phoneme noise into different audio segments of the sampled sample in a sliding manner to generate adversarial examples.

[0035] Thirdly, the speech recognition adversarial example generation system based on phoneme perturbation provided by the present invention includes: a processor and a memory, wherein the processor reads a computer program in the memory and performs the following operations:

[0036] Obtain sampled data from the audio dataset;

[0037] Adversarial examples are generated by injecting phoneme noise into different audio segments of the sampled data in a sliding manner.

[0038] In this invention, adversarial examples are generated by injecting phoneme noise into different audio segments of the sampled data in a sliding manner, rather than simply aligning the noise repeatedly input into the audio samples. The adversarial examples generated by this invention can better represent the key characteristics of samples with different factors, thereby activating the uncertainty of the recognition model and achieving stronger general attack capabilities. The adversarial examples generated by this invention can effectively evaluate the adversarial robustness and application security of speech recognition models. Furthermore, the phoneme adversarial attacks performed by the adversarial examples generated by this invention can also highlight the crucial role of phoneme features in speech recognition, providing guidance for subsequent speech recognition model design and feature extraction. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the method flow of an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of the device in the embodiments of this application;

[0042] Figure 3 This is a schematic diagram of the system structure in the embodiments of this application;

[0043] Figure 4 This is a schematic flowchart of the method for optimizing phoneme noise in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the method for generating adversarial examples in the embodiments of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Adversarial examples were first proposed in the field of computer vision to mislead deep learning models into making incorrect predictions. To date, there has been a wealth of research on adversarial examples across various tasks.

[0047] Given an audio dataset For a clean audio x (where... An audio adversarial noise δ can be perturbed to a given depth model. This leads to incorrect predictions, and the noise δ is ε-constrained. Furthermore, a general audio adversarial noise, possessing a stronger ability to deceive and mislead the model, is applicable to almost all... Both are valid, and can be expressed as:

[0048]

[0049] In formula (1), ||·|| is a distance metric for measuring noise intensity using the ∞-norm, and ε is a constraint value.

[0050] However, the Phoneme Adversarial Noise (PAN) method proposed in this embodiment, which is a speech recognition adversarial example generation method based on phoneme perturbation, differs somewhat from the above definition. More precisely, since the PAN proposed in this embodiment is based on phoneme-level feature information, it can be expressed as follows within the framework of the above definition:

[0051]

[0052] In formula (2), It is a concatenation function that can concatenate K phonemes against noise δ. p To splice them together. Furthermore, It can be represented as:

[0053]

[0054] In formula (3), Concat(·) means concatenation.

[0055] In the method described in this embodiment, the duration of the noise suppression is at the phoneme level, which requires cyclically splicing dozens or even hundreds of phoneme noises. For example, for a duration of l... x The audio x in this embodiment needs to be spliced. There are noise segments, among which It is a phoneme-based noise countermeasure δ p The duration.

[0056] like Figure 1 As shown, this application provides a method for generating adversarial examples for speech recognition based on phoneme perturbation, including:

[0057] 101. Obtain sampled data from the audio dataset;

[0058] 102. Inject phoneme noise into different audio segments of the sampled samples in a sliding manner to generate adversarial examples.

[0059] In this embodiment, adversarial examples are generated by injecting phoneme noise into different audio segments of the sampled samples in a sliding manner, rather than simply aligning the noise repeatedly input into the audio samples. The adversarial examples generated by this invention can better obtain key representations of different factor samples, thereby activating the uncertainty of the recognition model and achieving stronger general attack capabilities. The adversarial examples generated by this invention can effectively evaluate the adversarial robustness and application security of speech recognition models.

[0060] like Figure 5 As shown, in this embodiment, step 101 represents equalized phoneme density sampling. The equalized phoneme density sampling in this embodiment can obtain representative instances (i.e., fewer in number but rich in phoneme features), thereby effectively reducing training costs and achieving a significant acceleration effect in generation while maintaining attack performance.

[0061] In step 101, obtaining sampled data from the audio dataset specifically includes:

[0062] 1011. Select audio samples whose phoneme density meets the preset conditions from the audio dataset as sampling samples; the phoneme density is the number of phonemes in the audio sample per unit length.

[0063] In 1011, the step of selecting audio samples whose phoneme density meets preset conditions from the audio dataset as sampling samples specifically includes:

[0064] 10111. Calculate the phoneme density of each audio sample in the audio dataset; the expression is:

[0065]

[0066] In formula (5), x represents an audio sample, ψ(·) represents the phoneme recognizer function, and l x is the length of the audio sample x, and counter(·) represents the counter function.

[0067] In this embodiment, in order to select more representative sampling samples, phoneme density is defined as the number of phonemes per unit length. Phoneme density describes the richness of phonemes within a certain data point.

[0068] 10112. Calculate the average phoneme density of the audio dataset using the phoneme density of each audio sample; the expression is:

[0069]

[0070] In formula (5), Let x represent the i-th audio sample. i Length, Let represent the audio dataset, and ψ(·) represent the phoneme recognizer function.

[0071] 10113. Audio samples whose phoneme density differs from the average phoneme density by no more than a phoneme density threshold are selected as the selected samples; the expression for the selected samples is:

[0072]

[0073] In formula (5), D represents the average phoneme density of the audio dataset. Let x represent the phoneme density of the audio sample, counter(·) represent the counter function, ψ(·) represent the phoneme recognizer function, α represent the phoneme density threshold, which is the threshold controlling the density interval; x represents the audio sample, l x Indicates the length of the audio sample.

[0074] Considering the correlation between the sampled data and the audio dataset, excessively high or low phoneme densities may lead to overfitting or underfitting. This embodiment assumes that the appropriate phoneme density for the sampled data points should approximate the average phoneme density of the dataset; that is, the phoneme density should be as consistent as possible with the overall phoneme distribution at the dataset level. Specifically, for each sample... This embodiment determines whether a sample should be included in the sampling by estimating its phoneme density. This indicates that the sample x is judged based on the selected function. k The result of whether or not the sample should be selected is 1, which means that the sample should be selected; otherwise, it should not be selected.

[0075] 10114. Select samples whose length and amplitude are both greater than the corresponding preset threshold as sampling samples.

[0076] In this embodiment, all acceptable samples will be selected for training data using a simple selection function top(·).

[0077]

[0078] In formula (6), the function top(·) represents selecting the n (n≥1) samples with the longest length and the largest amplitude, and x represents the sampled sample.

[0079] General adversarial noise generation methods heavily rely on the size of the training dataset, consuming significant time. Existing adversarial attack methods mostly generate adversarial noise at the audio instance level, requiring a large number of audio data instances for training, resulting in substantial time overhead. However, the method described in this embodiment, targeting phoneme-level features, allows for the optimization of phoneme adversarial noise with fewer training instances. Specifically, since phoneme features are extremely short compared to sample-level data (typically tens of times shorter), a single sample may contain dozens of different phoneme features, making it highly feasible to optimize phoneme adversarial noise with fewer samples. Therefore, based on this viewpoint, this embodiment, considering phoneme density, extracts more representative samples from the given dataset, i.e., equalized phoneme density sampling.

[0080] In this embodiment, because the sampled data has relatively rich phoneme features, it can serve as effective and targeted training samples for phoneme-based noise suppression. Furthermore, the sampled audio data is much smaller in scale compared to the normal optimization process, thereby reducing the computational cost of training.

[0081] like Figure 4 and 5 As shown, in the PAN method proposed in this embodiment, the most critical factor determining attack capability is the impact of generated noise on phoneme patterns in sample instances. Considering that feature diversity may be crucial for model generalization, this embodiment increases the diversity of phoneme features used during optimization, which can also enable the phoneme noise to be optimized to acquire a more considerable attack capability against phonemes. Therefore, this embodiment proposes a diversified phoneme pattern learning strategy, namely step 102. In each step of the phoneme adversarial noise generation process, it is forced to asynchronously experience multiple audio segments, rather than simply repeating and aligning them. In this way, the generated adversarial noise can better learn the key representations in different phoneme patterns, thereby activating model uncertainty and obtaining a stronger general attack capability.

[0082] like Figure 4 As shown in 102, the step of injecting phoneme noise into different audio segments of the sampled samples in a sliding manner to generate adversarial examples specifically includes:

[0083] 1021. Based on deep learning, phoneme noise is optimized by injecting it into different audio segments of the sampled samples in a sliding manner, and the output phonemes are used to combat noise.

[0084] 1022. Combine adversarial noise with ordinary audio samples to obtain adversarial samples.

[0085] In 1021, the method based on deep learning optimizes phoneme noise by injecting it into different audio segments of the sampled samples in a sliding manner. Specifically, this includes:

[0086] 10211. For each optimization iteration, phoneme noise is injected into different audio segments of the sampled samples with a variable sliding step size; the variable sliding step size injection of phoneme noise into the sampled samples can be expressed as:

[0087]

[0088] In formula (7), This represents the phoneme noise injection module; clip(·) represents the clipping function, which clips the input sample x into a segment of length ba based on the clipping interval [a,b], where a is the clipping start time point and b is the clipping end time point; It is a concatenation function; n is the iteration number, l x It is the length of sample x during training. It is phoneme noise δ p The length of the sliding window is β, where β is the step size of the sliding window.

[0089] like Figure 5 As shown, in the method described in this embodiment, the adversarial noise optimization process is not simply a repetition of injecting noise δ into audio instance x at each step. Simple repetition would result in weak attack adaptability to different audio sample instances. Therefore, this embodiment introduces a sliding window to inject phoneme adversarial noise into sample instances in a diversified manner during the optimization process. For each iteration, the phoneme noise will be variably injected into different audio segments with a sliding step size β. For example, for a phoneme noise δ... p If it is injected into the audio segment [M, N] in the first iteration, then in the next iteration, it will slide for a duration of 1×β, that is, be injected into a segment with a duration interval of [M+β, N+β]. Intuitively, the phoneme noise to be optimized will face audio segments with more diverse phoneme patterns in different iteration rounds, thus enabling it to efficiently learn adversarial representations and better activate model uncertainty on different samples.

[0090] By leveraging diversified phoneme pattern learning, the phoneme noise synthesis module spans multiple phoneme segments within the sampled instances in each iteration. This allows for more effective capture of key cross-phoneme patterns during optimization, learning more targeted adversarial representations, and thus better stimulating the uncertainty of the original model's output, resulting in stronger general attack capabilities. The Phoneme Adversarial Noise (PAN) method in this embodiment performs fast and general adversarial attacks targeting phoneme features in speech. To accelerate generation, PAN first obtains representative audio instances through the proposed equalized phoneme density sampling. To improve general attack capabilities, PAN enhances the diversity of phoneme patterns during training by employing a diversified phoneme pattern learning strategy. In this way, the generated phoneme adversarial noise fully utilizes phoneme-level optimization, thereby achieving better adversarial attack performance.

[0091] In addition, this embodiment also needs to consider the actual effects in complex physical environments. Existing research has achieved certain results in audio analysis, and this paper introduces a representative method: Room Impulse Response (RIR). The RIR function, or Room Impulse Response Function, ensures the usability of the PAN in the real world by encoding Acoustic Channel State Information (CSI).

[0092] 10212. For the sampled samples with injected phoneme noise, optimize the phoneme noise using a loss function. The optimization process can be expressed as:

[0093]

[0094] In formula (8), For CTC loss function, This represents a speech recognition model. For RIR functions, ||·|| ∞ It is the infinite norm, min means minimization, X represents the sampled dataset, and ∈ represents the constraint value.

[0095] In this embodiment, CTCloss stands for Connectionist Temporal Classification Loss, which is a loss function used to solve time series data classification problems.

[0096] In this embodiment, in order to improve the general attack capability, a diversified phoneme pattern learning method is proposed to asynchronously complete the optimization process of generated noise, so that the adversarial noise to be optimized faces more different audio segments containing different phoneme features in a limited number of training samples, thereby better stimulating the uncertainty of the attacked model.

[0097] like Figure 4As shown, before the optimization iteration, the phoneme noise δ is first randomly initialized. p Then use Perform sliding phoneme noise injection; then minimize the loss function using formula (8) to optimize the phoneme noise δ. p The optimized phoneme noise δ is output after the number of optimization iterations is reached. p The above process is illustrated in Table 1:

[0098] Table 1

[0099]

[0100] During the training process, it is necessary to determine the end of the iteration. If the iteration ends, the training of the input phoneme noise is stopped; otherwise, the iteration will continue.

[0101] This embodiment can be applied in practice in the following ways: For example... Figure 5 As shown, the training dataset is obtained by balanced sampling based on phoneme density from the original dataset, and phoneme-level adversarial noise δ is initialized. p Subsequently, training and optimization are performed using a sliding phoneme noise injection method, with the number of optimization rounds determined by the specified number of optimization iterations. During each iteration, adversarial noise δ is addressed. p Sliding noise injection is performed, primarily through a sliding noise injection module. Additionally, before each optimization, the adversarial noise δ needs to be adjusted. p Through the room pulse response module After processing, the loss function is calculated for optimization. The obtained adversarial noise can be injected into the original speech in the digital world to form adversarial examples, or it can be played in the physical world to affect the speech data collected by the acquisition device, thereby attacking the speech recognition module and achieving the effects of model evaluation or speech privacy protection.

[0102] The effectiveness of the method described in this embodiment will be demonstrated through experimental results below:

[0103] For datasets, the experiments primarily used the well-known LibriSpeech dataset for speech recognition tasks. To verify the generality of the generated adversarial noise, the experiments also used the TED-LIUM dataset and the Mozilla Common Voice dataset. For baseline models, this section selected several state-of-the-art models with different architectures and parameter sizes to perform different tasks. Specifically, for the ASR task, the DeepSpeech2, Wav2Vec, and SEW models were chosen. For the VPR task, the DeepSpeaker model was selected. All models were pre-trained on open-source datasets.

[0104] To demonstrate the effectiveness of the PAN method proposed in this embodiment, experiments were first conducted in both the digital and physical worlds. For digital world attacks, the experiments used data points sampled from the LibriSpeech dataset to train adversarial noise, and DeepSpeech2 (a speech recognition model) was used as the attacked model. In addition to the baseline model, the experiments also introduced some simulated noise from Noise92 (a noise library) for comparison.

[0105] Table 2 shows the experimental results on different datasets in the digital world. In Table 2, the up arrow indicates that a larger value is better, and the down arrow indicates the opposite. x The strength of the adversarial noise is represented. Raw represents the model's accuracy in recognizing clean samples. t is the time consumption of the attack process (in minutes). Speedup represents the speedup factor of the proposed method compared to the time consumption of the selected baseline algorithm. Since UAP is one of the most typical methods of general audio attacks, the experiment uses UAP as the speed baseline.

[0106] The adversarial noise generated by the method presented in this paper achieves significantly superior attack capabilities, and the following conclusions are drawn:

[0107] 1. The Phoneme Adversarial Noise (PAN) proposed in this embodiment possesses powerful general attack capabilities due to its unique design. Specifically, on the LibriSpeech dataset, the PAN attack success rate (SR) is significantly improved to 0.87, while the character error rate (CER) also reaches a high value of 0.83. Compared with the baseline method, the noise generated by the PAN method shows an average improvement of 46% in SR and an average improvement of 38% in CER, demonstrating strong general attack capabilities. Furthermore, compared with the noise simulated by Noise92, our SR metric is significantly superior, indicating the factual effectiveness of adversarial noise.

[0108] Table 2

[0109]

[0110] 2. The PAN method proposed in this embodiment can generate general audio adversarial noise at a lower time cost, i.e., a faster generation speed. For example, the typical "TUAP" attack on the Librispeech dataset can only achieve a CER value of 0.18, but its time cost is 187 minutes (13 times slower than the method in this paper). The current best general attack model, "CUAP," has powerful attack capabilities, but requires more computational cost, i.e., more than 41 times the additional time overhead. The same trend is observed on other datasets. Overall, the average time cost of this paper is 10.67 minutes, while the time costs of UAP, TUAP, CUAP, and APUP are 308, 186.67, 512.33, and 311.33 minutes, respectively. Compared with the baseline method, the PAN method achieves an average speed improvement of 32.46 times.

[0111] 3. Apart from CUAP, the other baseline methods exhibit relatively weak attack capabilities; specifically, on LibriSpeech, the SR metrics are 0.03 for UAP, 0.06 for TUAP, and 0.02 for APUP. For UAP, despite the model being fully trained according to the original paper's method, the results appear unsatisfactory, indicating that its design is not practical. For TUAP and APUP, this paper argues that they are designed for individual-targeted attacks, thus limiting their ability to execute general attacks. Since there is a scarcity of work on non-targeted general audio attacks, these targeted attacks were also included in the comparison.

[0112] For attacks in the physical world, this embodiment constructs a simulated scenario in the physical world and reports the experimental results. Specifically, three devices (two iPads and one mobile phone) were used as the main player and receiver in the experiment. One iPad was used to play noisy audio, and the other was used to play normal audio; the mobile phone was used to receive mixed physical audio. For accurate evaluation, 50 instances were randomly selected from LibriSpeech for testing, and the experimental results are shown in Table 3. The results show that compared with the baseline model, the PAN method proposed in this embodiment has the strongest attack capability. That is, the SR and CER indices of the method proposed in this embodiment are 0.68 and 0.74, respectively, which are higher than the 0.32 and 0.48 of the CUAP model. In addition, referring to Table 3, it can be found that the performance loss of the method proposed in this embodiment in cross-digital domain-physical domain attacks is less than that of other baselines, which reflects the important significance of basic phoneme features in improving the performance of adversarial attacks.

[0113] Table 3

[0114]

[0115] Since no additional time overhead is required under physical world conditions, time consumption metrics are not displayed.

[0116] In summary, this embodiment proposes a method that utilizes fundamental phoneme features in audio data to achieve faster and more universal adversarial attacks. Specifically, this embodiment proposes equalized phoneme density sampling to obtain fewer but more representative sample instances for phoneme adversarial noise optimization, thereby accelerating the generation speed. Furthermore, this embodiment proposes a diversified phoneme pattern learning strategy, using sliding noise injection during the optimization process to complete diversified phoneme pattern learning, which helps enhance the universal attack capability against adversarial noise. Benefiting from the utilization of phoneme-level features, the phoneme adversarial noise generated by the method described in this embodiment achieves excellent performance. Experimental results show that the PAN method described in this embodiment outperforms other state-of-the-art methods in both speed and attack success rate, achieving an average speedup of over 24 times and a 38% improvement in attack success rate compared to baseline methods.

[0117] like Figure 2 As shown in the figure, this application embodiment also provides a speech recognition adversarial example generation device based on phoneme perturbation, including:

[0118] Select Unit 1 to obtain sampled data from the audio dataset;

[0119] Generation unit 2 is used to inject phoneme noise into different audio segments of the sampled sample in a sliding manner to generate adversarial examples.

[0120] like Figure 3 As shown, this application embodiment also provides a speech recognition adversarial example generation system based on phoneme perturbation, characterized in that it includes: a processor 3 and a memory 4, wherein the processor 3 reads a computer program in the memory 4 and performs the following operations:

[0121] Obtain sampled data from the audio dataset;

[0122] Adversarial examples are generated by injecting phoneme noise into different audio segments of the sampled data in a sliding manner.

[0123] The functions and roles of the device and system described in this embodiment are as described in the method above, and will not be repeated here. For any content that is not fully explained, please refer to the content described above.

[0124] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0125] To make the description of this disclosure more detailed and complete, illustrative descriptions of the embodiments and specific examples of the present invention have been provided above; however, this is not the only form of implementing or utilizing the specific examples of the present invention. The embodiments cover the features of multiple specific examples and the method steps and their order for constructing and operating these specific examples. However, other specific examples may also be used to achieve the same or equivalent functions and order of steps.

[0126] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0127] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0128] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0129] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0130] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0131] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0132] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0133] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating adversarial examples for speech recognition based on phoneme perturbation, characterized in that, include: Obtain sampled data from the audio dataset; Adversarial examples are generated by injecting phoneme noise into different audio segments of the sampled samples in a sliding manner. The step of injecting phoneme noise into different audio segments of the sampled samples in a sliding manner to generate adversarial examples specifically includes: Based on deep learning, phoneme noise is optimized by injecting it into different audio segments of the sampled samples in a sliding manner, and the output phonemes are used to combat noise. Combining adversarial noise with ordinary audio samples yields adversarial samples; The method based on deep learning optimizes phoneme noise by injecting it into different audio segments of the sampled data in a sliding manner. Specifically, this includes: For each optimization iteration, phoneme noise is injected into different audio segments of the sampled samples with a variable sliding step size; For the sampled samples with injected phoneme noise, the phoneme noise is optimized using a loss function; The phoneme noise is injected into the sampled samples with a variable sliding step size, and can be expressed as: （7） In formula (7), This indicates the phoneme noise injection module; This represents the shearing function, which cuts the input sampled data according to the shearing interval [a, b]. The cut is a segment of length ba, where a is the start time point of the cut and b is the end time point of the cut; It is a concatenation function; For the number of iterations, Samples collected during training Length, It is phoneme noise Length, It is the step size of the sliding window.

2. The method for generating adversarial examples for speech recognition based on phoneme perturbation according to claim 1, characterized in that, The step of obtaining sampling samples from the audio dataset specifically includes: Audio samples whose phoneme density meets the preset conditions are selected from the audio dataset as sampling samples; the phoneme density is the number of phonemes in the audio sample per unit length.

3. The method for generating adversarial examples for speech recognition based on phoneme perturbation according to claim 2, characterized in that, The step of selecting audio samples whose phoneme density meets preset conditions from the audio dataset as sampling samples specifically includes: Calculate the phoneme density of each audio sample in the audio dataset; The average phoneme density of the audio dataset is calculated using the phoneme density of each audio sample. Audio samples whose phoneme density differs from the average phoneme density by no more than the phoneme density threshold are selected as the selected samples. Select samples whose length and amplitude are both greater than the corresponding preset threshold are used as sampling samples.

4. The method for generating adversarial examples for speech recognition based on phoneme perturbation according to claim 1, characterized in that, The optimization process can be expressed as: （8） In formula (8), For CTC loss function, This represents a speech recognition model. For RIR functions, It is an infinite norm. Indicates minimization. Represents the sampled dataset. This represents the constraint value.

5. The method for generating adversarial examples for speech recognition based on phoneme perturbation according to claim 3, characterized in that, The expression for selecting the sample is: （5） In formula (5), This represents the average phoneme density of the audio dataset. Represents the phoneme density of an audio sample. Represents the counter function. This represents the phoneme recognizer function. Indicates the phoneme density threshold. Represents audio samples, Indicates the length of the audio sample.

6. A speech recognition adversarial example generation device based on phoneme perturbation, characterized in that, include: Selecting units is used to obtain sampled data from the audio dataset; The generation unit is used to inject phoneme noise into different audio segments of the sampled sample in a sliding manner to generate adversarial examples; The generation unit is specifically used for: For each optimization iteration, phoneme noise is injected into different audio segments of the sampled samples with a variable sliding step size; For the sampled samples with injected phoneme noise, the phoneme noise is optimized using a loss function; The phoneme noise is injected into the sampled samples with a variable sliding step size, and can be expressed as: （7） In formula (7), This indicates the phoneme noise injection module; This represents the shearing function, which cuts the input sampled data according to the shearing interval [a, b]. The cut is a segment of length ba, where a is the start time point of the cut and b is the end time point of the cut; It is a concatenation function; For the number of iterations, Samples collected during training Length, It is phoneme noise Length, It is the step size of the sliding window; Adversarial samples are obtained by combining adversarial noise with ordinary audio samples.

7. A speech recognition adversarial example generation system based on phoneme perturbation, characterized in that, include: A processor and memory, wherein the processor reads a computer program from the memory and performs the following operations: Obtain sampled data from the audio dataset; Adversarial examples are generated by injecting phoneme noise into different audio segments of the sampled samples in a sliding manner. The step of injecting phoneme noise into different audio segments of the sampled samples in a sliding manner to generate adversarial examples specifically includes: Based on deep learning, phoneme noise is optimized by injecting it into different audio segments of the sampled samples in a sliding manner, and the output phonemes are used to combat noise. Combining adversarial noise with ordinary audio samples yields adversarial samples; The method based on deep learning optimizes phoneme noise by injecting it into different audio segments of the sampled data in a sliding manner. Specifically, this includes: For each optimization iteration, phoneme noise is injected into different audio segments of the sampled samples with a variable sliding step size; For the sampled samples with injected phoneme noise, the phoneme noise is optimized using a loss function; The phoneme noise is injected into the sampled samples with a variable sliding step size, and can be expressed as: （7） In formula (7), This indicates the phoneme noise injection module; This represents the shearing function, which cuts the input sampled data according to the shearing interval [a, b]. The cut is a segment of length ba, where a is the start time point of the cut and b is the end time point of the cut; It is a concatenation function; For the number of iterations, Samples collected during training Length, It is phoneme noise Length, It is the step size of the sliding window.

Images