Human ear imperceptible physical voice confrontation sample generation method

Abstract

The invention discloses a human-ear-imperceptible physical voice confrontation sample generation method, which comprises the following steps of: reading waveform data of an original voice sample, calculating voice confrontation disturbance loss, calculating room impulse response of a target environment, and generating a human-ear-imperceptible physical voice confrontation sample. And iteratively optimizing the voice confrontation disturbance loss and the reasoning loss of the voice confrontation sample on the target voice model by using a gradient descent algorithm until the voice confrontation disturbance meeting the requirement is generated, so that the optimization target in the voice confrontation sample generation process is consistent with the generation target, and the convergence of the optimization target in the confrontation sample generation process is accelerated. In the forward and reasoning process during voice model training, psychoacoustic loss of voice confrontation disturbance is added on the basis of reasoning loss on a target voice model, and the problem that an optimization target and a generation target are inconsistent due to the decoding process is solved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition security technology, and in particular to a method for generating physical speech adversarial samples that are imperceptible to human ears. Background Art

[0002] Since AlexNet won the ImageNet competition with a classification accuracy that was 10% higher than the second place, deep learning has quickly attracted the attention of academia and industry. It has been applied to many pattern recognition tasks and is constantly surpassing the upper limit of artificial intelligence. In some tasks, its performance even exceeds that of humans. As an important bridge for human-computer interaction, speech recognition can be seen everywhere in life, such as voice assistants on mobile phones, smart speakers, service robots, etc. Therefore, industry and academia have also applied deep learning technology to speech recognition to improve the performance of human-computer interaction devices, such as DeepSpeech and DeepSpeech2 proposed by Baidu, Paraformer and DFSMN proposed by Alibaba Damo Academy, Wav2letter++ and Wav2vec proposed by Facebook, LAS proposed by Google, and Whisper proposed by OpenAI.

[0003] Adversarial samples refer to samples that contain adversarial perturbations that are imperceptible to humans. Adversarial samples will not affect human perception or cognitive results, but can cause the target model to infer incorrect results. The threat model of adversarial samples can be divided into: white-box samples that know the internal information of the model, black-box samples that can only know the output of the model, directional samples that can make the model output a specified result, and non-directional samples that can only make the model wrong. Adversarial samples first appeared in the field of computer vision, but with the continuous research of the academic community, adversarial samples have expanded to text, speech, point cloud and other fields. Typical speech adversarial sample generation algorithms include: the speech version of the C&W algorithm proposed by Carlini and Wagner, Qin uses psychoacoustics and room impulse response to generate speech adversarial samples, and Alzantot uses genetic algorithms to generate speech adversarial samples.

[0004] Although there are some speech adversarial sample generation algorithms, some generation algorithms have the problem that the optimization target is inconsistent with the generation target. The goal of speech adversarial samples is to reduce or increase the word error rate of the target speech model, while the optimization target of the speech version C&W and Qin algorithms is to reduce or increase the loss function value, such as the CTC loss function value, RNN-T loss function value, etc. CTC, RNN-T and other loss functions solve the alignment problem by label expansion. Therefore, using CTC, RNN-T and other loss functions as optimization targets will lead to problems such as difficulty in convergence and slow convergence in the optimization process. In the speech version C&W and Qin algorithms, they solve this problem by fixing the labels in the CTC, RNN-T and other loss functions. The forward process of speech model training includes: forward propagation and calculation of loss function values. The reasoning process of speech model includes: forward propagation and decoding. The word error rate is calculated after decoding and is not differentiable. Therefore, the decoding process will also lead to inconsistency between the optimization target and the generation target. Summary of the invention

[0005] The purpose of the present invention is to provide a method for generating physical speech adversarial samples that are imperceptible to the human ear, so as to overcome the problem in the prior art that the optimization target and the generation target are inconsistent.

[0006] A method for generating physical speech adversarial samples that are imperceptible to human ears comprises the following steps:

[0007] S1, read the waveform data of the original speech sample, and generate the initial speech adversarial perturbation using a function whose domain is the set of all real numbers and whose range is 0 to 1;

[0008] S2, uses the loudness of the speech to quantify the size of the initial speech adversarial perturbation and calculates the speech adversarial perturbation loss;

[0009] S3, playing and recording speech in the target environment, and calculating the room impulse response of the target environment;

[0010] S4, convolution operation is performed on the speech adversarial sample and the room impulse response of the target environment to calculate the inference loss of the speech adversarial sample on the target speech model;

[0011] S5, using the gradient descent algorithm to iteratively optimize the speech adversarial perturbation loss and the inference loss of the speech adversarial sample on the target speech model until a speech adversarial perturbation that meets the requirements is generated;

[0012] S6, the psychoacoustic loss of the generated speech against the perturbation is calculated using the psychoacoustic criterion;

[0013] S7, add the psychoacoustic loss of speech adversarial perturbation to the inference loss of the speech adversarial sample on the target speech model, and use the gradient descent algorithm to iteratively optimize the three losses until the speech adversarial perturbation that meets the requirements is generated;

[0014] S8, using the obtained speech adversarial perturbation that meets the requirements to generate speech adversarial samples.

[0015] Preferably, in S1, the numerical value of the waveform data of the original speech sample is normalized to [0, 1], as shown below:

[0016]

[0017] Where x is the original speech sample of length T.

[0018] Preferably, in S1, the expression of the function whose domain is the set of all real numbers and whose range is 0 to 1 is:

[0019]

[0020] Among them, δ is the speech adversarial perturbation, δ∈[0,1], z∈(-∞,+∞), z is arctan(x), and this function is used to generate the speech adversarial perturbation δ.

[0021] Preferably, in S2, the loudness calculation formula of the speech is:

[0022] dB(x)=20log 10 (x)

[0023] The calculation formula for speech adversarial perturbation loss is:

[0024]

[0025] Preferably, in S3, the room impulse response calculation formula of the target environment is:

[0026]

[0027] in, r is the room impulse response, w is the speech after speech x is played in the target environment, * is the convolution operation, is the reverse order of the sound x.

[0028] Preferably, in S4, the sample after the speech adversarial sample is convolved with the room impulse response of the target environment is:

[0029] (x+δ)*r

[0030] r is the room impulse response, δ is the generated speech adversarial perturbation, and x is the original speech sample of length T.

[0031] Preferably, in S5, the gradient descent algorithm used is Adam, which optimizes the following objectives:

[0032]

[0033] Among them, λ1 is a constant.

[0034] Preferably, in S5, the loss function for generating the directional speech adversarial sample is:

[0035]

[0036] Preferably, in S5, the loss function for generating non-directional speech adversarial samples is:

[0037]

[0038] Preferably, in S6, the psychoacoustic loss of speech against disturbance is as follows:

[0039]

[0040] Among them, θ x (t) is x t The psychoacoustic masking threshold of is δ t The normalized power spectrum of ;

[0041] In S7, the gradient descent algorithm used is Adam, which optimizes the following objectives:

[0042]

[0043] Among them, λ1 and λ2 are constants.

[0044] Compared with the prior art, the present invention has the following beneficial technical effects:

[0045] The present invention provides a method for generating physical speech adversarial samples that are imperceptible to human ears. The method comprises the following steps: reading waveform data of an original speech sample, using a function whose domain is a set of all real numbers and whose range is 0 to 1 to generate an initial speech adversarial perturbation, using the loudness of the speech to quantify the size of the initial speech adversarial perturbation, calculating the speech adversarial perturbation loss, playing and recording speech in a target environment, calculating the room impulse response of the target environment, performing convolution operation on the speech adversarial sample and the room impulse response of the target environment, calculating the inference loss of the speech adversarial sample on the target speech model, and using a gradient descent algorithm to iteratively optimize the speech adversarial perturbation loss and the inference loss of the speech adversarial sample on the target speech model until a speech adversarial perturbation that meets the requirements is generated, so that the optimization target in the speech adversarial sample generation process is consistent with the generation target, and the convergence of the optimization target in the adversarial sample generation process is accelerated. In the forward and inference process of speech model training, the psychoacoustic loss of the speech adversarial perturbation is added on the basis of the inference loss on the target speech model, so as to solve the problem that the optimization target and the generation target are inconsistent due to the decoding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a method for generating physical speech adversarial samples that are imperceptible to the human ear in an embodiment of the present invention.

[0047] Figure 2 is an example of a room impulse response of a target environment in an embodiment of the present invention.

[0048] Figure 3 It is the target speech recognition model in the embodiment of the present invention.

[0049] Figure 4 This is the evaluation result in a digital environment in an embodiment of the present invention.

[0050] Figure 5 is an evaluation result in a physical environment in the embodiment of the present invention.

[0051] Figure 6 This is a speech adversarial sample generated in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] like Figure 1 As shown, the present invention provides a method for generating physical speech adversarial samples that are imperceptible to human ears, which specifically includes the following steps:

[0055] S1, read the waveform data of the original speech sample, and generate the initial speech adversarial perturbation using a function whose domain is the set of all real numbers and whose range is 0 to 1;

[0056] S2, uses the loudness of the speech to quantify the size of the initial speech adversarial perturbation and calculates the speech adversarial perturbation loss;

[0057] S3, playing and recording speech in the target environment, and calculating the room impulse response of the target environment;

[0058] S4, convolution operation is performed on the speech adversarial sample and the room impulse response of the target environment to calculate the inference loss of the speech adversarial sample on the target speech model;

[0059] S5, using the gradient descent algorithm to iteratively optimize the speech adversarial perturbation loss and the inference loss of the speech adversarial sample on the target speech model until a speech adversarial perturbation that meets the requirements is generated;

[0060] S6, the psychoacoustic loss of the generated speech against the perturbation is calculated using the psychoacoustic criterion;

[0061] S7, add the psychoacoustic loss of speech adversarial perturbation to the inference loss of the speech adversarial sample on the target speech model, and use the gradient descent algorithm to iteratively optimize the three losses until the speech adversarial perturbation that meets the requirements is generated;

[0062] S8, using the obtained speech adversarial perturbation that meets the requirements to generate speech adversarial samples.

[0063] Specifically, in S1, the waveform data x of the original speech sample with a length of T is read, the value is normalized to [0,1], and the function Generate initial speech adversarial perturbation, where δ∈[0,1] is the speech adversarial perturbation and z∈(-∞,+∞) is arctan(x). The advantage of using this function to generate speech adversarial perturbation is that the optimization objectives in S5 and S7 are unconstrained.

[0064] In the speech field, loudness is often used instead of L p To quantify the size of speech adversarial disturbance, the loudness calculation formula is:

[0065] dB(x)=20log 10 (x)

[0066] The calculation formula for speech adversarial perturbation loss is:

[0067]

[0068] After the speech adversarial sample is played in a real physical environment, the echo and noise of the environment will destroy the integrity of the speech adversarial sample, making the speech adversarial sample ineffective for the target speech recognition model. Therefore, it is necessary to play and record the speech in the target environment in advance and calculate the room impulse response of the target environment. The calculation formula is:

[0069]

[0070] in, r is the room impulse response, w is the speech after speech x is played in the target environment, * is the convolution operation, is the reverse order of the sound x.

[0071] Calculate the inference loss of the speech adversarial sample on the target speech recognition model as follows:

[0072]

[0073] Among them, x t is the tth frame of speech sample x, y t For x t The corresponding one-hot encoding of the directional or non-directional output, F(·) is the target speech model, F((x+δ)*r) t For x t The probability vector output on the target speech model F(·), R(·) is the attack performance evaluation function of the speech adversarial sample x+δ;

[0074] Since the real physical environment has factors such as echo and noise that can destroy the integrity of speech adversarial samples, the generation process of speech adversarial samples needs to include the convolution operation of the speech adversarial samples and the room impulse response of the target environment. The calculation process is:

[0075] (x+δ)*r

[0076] When the target speech model is a CTC-based speech recognition model or an RNN-T-based speech recognition model, the length T of F((x+δ)*r) is much larger than the length O of D(F((x+δ)*r)). In the CTC loss function, it is necessary to repeat characters or add meaningless spaces η in D(F((x+δ)*r)) to expand the length of D(F((x+δ)*r)) to T. Therefore, there are multiple paths that can expand the length of D(F((x+δ)*r)) to T. When the target speech model is a CTC-based speech recognition model, y0 : The construction process of T:

[0077]

[0078] When the target speech model is a speech recognition model based on RNN-T, y 0:T The construction process:

[0079]

[0080] Where D(·) is the decoding process of the speech recognition model, is the directional or non-directional output of the target speech recognition model specified by the attacker, a is a positive integer, and η is a placeholder;

[0081] The derivation process of the inference loss of speech adversarial samples on the target speech recognition model is:

[0082] L=E y～F(x) [R(y)]

[0083]

[0084] R(·) is the attack performance evaluation function of the speech adversarial sample x+δ. For the speech recognition model, the edit distance between D(F((x+δ)*r)) and y is used:

[0085]

[0086] Where WER is the word error rate, S is the number of replaced characters, E is the number of deleted characters, I is the number of inserted characters, and H is the number of correct characters. When y is the One-hot encoding of the non-directional output corresponding to x, speech adversarial samples are generated by increasing the edit distance. When y is the One-hot encoding of the directional output corresponding to x, speech adversarial samples are generated by reducing the edit distance. In addition, R(·) is not differentiable.

[0087] Use the gradient descent algorithm Adam to iteratively optimize the following objectives:

[0088]

[0089] Among them, λ1 is a constant, and generating directional speech adversarial samples requires minimizing L adv , the loss function is:

[0090]

[0091] Generating non-directional speech adversarial samples requires maximizing L adv , the loss function is:

[0092]

[0093] Iteratively optimize the objective function until a speech adversarial perturbation that meets the requirements can be generated;

[0094] From the attacker's perspective, the generated speech adversarial perturbation should be as imperceptible to the human ear as possible. Therefore, the psychoacoustic loss of the speech adversarial perturbation generated in step S5 is calculated using the psychoacoustic criterion. The calculation process is as follows:

[0095]

[0096] Among them, θ x (t) is x t The psychoacoustic masking threshold of is δ t The normalized power spectrum of ;

[0097] Based on step S4, the psychoacoustic loss of speech adversarial perturbation is added, and the gradient descent algorithm Adam is used to iteratively optimize the following objectives:

[0098]

[0099] Among them, λ1 and λ2 are constants, and the objective function is iteratively optimized until the speech adversarial perturbation that meets the requirements can be generated;

[0100] Use the speech adversarial perturbation that meets the requirements obtained in step S7 to generate speech adversarial samples:

[0101]

[0102] Example

[0103] Step 1

[0104] Use the recording function of the mobile phone to record the original voice sample x of length T. Use the ffmpeg, scipy and other code libraries under Python to convert the data format of the original voice sample to *.wav, adjust the sampling frequency to 16000, and adjust the value range to [-2 15 ,2 15-1]. Use deep learning libraries such as pytorch and speechbrain under Python to read the original speech samples, normalize the value range to [0,1], and convert the data type to tensor type:

[0105]

[0106] Define functions using Python And generate the initial speech adversarial perturbation, where δ∈[0,1] is the speech adversarial perturbation and z∈(-∞,+∞) is arctan(x). Use pytorch to convert the data type of speech adversarial perturbation into a tensor type.

[0107] Step 2

[0108] In the speech field, loudness is often used instead of L p To quantify the size of speech adversarial disturbance, the loudness calculation formula is:

[0109] dB(x)=20log 10 (x)

[0110] The calculation formula for speech adversarial perturbation loss is:

[0111]

[0112] Define the loss function L using pytorch per And calculate the speech adversarial perturbation loss.

[0113] Step 3

[0114] When the original speech sample is played in a real physical environment, the speech received by the microphone is a superposition of the pure sound of the original speech sample and the echo. The room impulse response refers to the sound signal obtained by convolution operation of the superimposed sound and the original pure sound. After the speech adversarial sample is played in a real physical environment, the echo, noise, etc. of the environment will destroy the integrity of the speech adversarial sample, making the speech adversarial sample ineffective for the target speech recognition model. Therefore, it is necessary to play and record the speech in the target environment in advance and calculate the room impulse response of the target environment. The calculation formula is:

[0115]

[0116] in, r is the room impulse response, w is the speech after speech x is played in the target environment, * is the convolution operation, is the reverse order of the sound x.

[0117] In the target physical environment, use the voice playback function of the mobile phone to play the original voice sample, and use the recording function of the mobile phone to record the played voice, and repeat this operation multiple times. Use ffmpeg to convert the data format of the original voice sample and the recorded voice sample to *.wav respectively. Use scipy to load the *.wav files at the same time for convolution operation and take the average value as the room impulse response of the target environment. Figure 2 This is an example of a room impulse response. Use scipy to save the room impulse response as a *.wav file.

[0118] Step 4

[0119] The speech recognition model based on CTC loss function is as follows Figure 3 -A, the speech recognition model based on the RNN-T loss function is as follows Figure 3 -B shows the speech recognition model based on the attention mechanism. Figure 3 -C. Use speechbrain to train the speech recognition model. The inference loss function of the speech adversarial sample on the target speech recognition model is:

[0120]

[0121] Among them, x t is the tth frame of speech sample x, y t For x t The corresponding one-hot encoding of the directional or non-directional output can be the length of the English phoneme table, the length of the English alphabet table, the length of the Chinese pinyin table, etc. F(·) is the target speech model, F((x+δ)*r) t For x t The probability vector output on the target speech model F(·) is the attack performance evaluation function of the speech adversarial sample x+δ. Use pytorch to define the inference loss function of the speech adversarial sample on the target speech model and calculate the loss function value.

[0122] Step 4 is further divided into:

[0123] Step 4.1

[0124] Since the real physical environment has factors such as echo and noise that can destroy the integrity of speech adversarial perturbations, the generation process of speech adversarial samples needs to include the convolution operation of the speech adversarial samples and the room impulse response of the target environment. The calculation process is:

[0125] (x+δ)*r

[0126] Use scipy to load the saved room impulse response and convert the data type to a tensor for convolution with the speech adversarial sample.

[0127] Step 4.2

[0128] In the speech recognition model, the length T of the input original speech sample x, that is, the length of F((x+δ)*r), is much larger than the length of the target text, that is, the length of D(F((x+δ)*r)), which is called the alignment problem. The CTC loss function solves the alignment problem by repeating characters or adding meaningless spaces η. For example, the length of D(F((x+δ)*r)) is 10, and the target text is hello. The target text can be expanded to: hhe_elloo_, h_he_e_llo, _hee_l_l_o, etc. The RNN-T loss function solves the alignment problem by adding meaningless spaces η. For example, the length of D(F((x+δ)*r)) is 10, and the target text is hello. The target text can be expanded to: h_e__ll_o_, h__e__llo, _he__l_l_o, etc. Among them, _ represents meaningless spaces η. There are multiple paths in the CTC and RNN-T loss functions that can extend the length of D(F((x+δ)*r)) to T. Therefore, the present invention will be closest to F((x+δ)*r) 0:T Target text as the optimal path.

[0129] When the target speech model is a CTC-based speech recognition model, y 0:T The construction process:

[0130]

[0131] When the target speech model is a speech recognition model based on RNN-T, y 0:T The construction process:

[0132]

[0133] Where D(·) is the decoding process of the speech recognition model, is the directional or non-directional output of the target speech recognition model specified by the attacker, a is a positive integer, and η is a placeholder. 0:T The construction process.

[0134] Step 4.3

[0135] The derivation process of the inference loss of speech adversarial samples on the target speech recognition model is:

[0136] L=E y～F(x) [R(y)]

[0137]

[0138] Step 4.4

[0139] R(·) is the attack performance evaluation function of the speech adversarial sample x+δ. For the speech recognition model, the edit distance between D(F((x+δ)*r)) and y is used:

[0140]

[0141] Where WER is the word error rate, S is the number of replaced characters, E is the number of deleted characters, I is the number of inserted characters, and H is the number of correct characters. When y is the One-hot encoding of the non-directional output corresponding to x, the speech adversarial sample is generated by increasing the edit distance. When y is the One-hot encoding of the directional output corresponding to x, the speech adversarial sample is generated by reducing the edit distance. Use speechbrain to define the word error rate loss function and calculate the loss function value.

[0142] Step 5

[0143] Use the gradient descent algorithm Adam in pytorch to iteratively optimize the following objectives:

[0144]

[0145] Among them, λ1 is a constant, and generating directional speech adversarial samples requires minimizing L adv , the loss function is:

[0146]

[0147] Generating non-directional speech adversarial samples requires maximizing L adv , the loss function is:

[0148]

[0149] The objective function is iteratively optimized until a speech adversarial perturbation that meets the requirements can be generated. The learning rate in Adam is set to 0.01, λ1 is set to 0.1, and the number of optimizations is set to 1000.

[0150] Step 6

[0151] From the attacker's perspective, the generated speech adversarial perturbation should be as imperceptible to the human ear as possible. Therefore, the psychoacoustic loss of the speech adversarial perturbation generated in step 5 is calculated using the psychoacoustic criterion. The calculation process is as follows:

[0152]

[0153] Among them, θ x (t) is x t The psychoacoustic masking threshold of is δt Use pytorch to calculate the masking threshold of the speech sample and the normalized power spectrum of the speech adversarial perturbation, define the psychoacoustic loss and calculate the loss function value.

[0154] Step 7

[0155] Based on step 4, add the psychoacoustic loss of speech adversarial perturbation, and use the gradient descent algorithm Adam under pytorch to iteratively optimize the following objectives:

[0156]

[0157] Among them, λ1 and λ2 are constants, and the objective function is iteratively optimized until the speech adversarial perturbation that meets the requirements can be generated. In Adam, the learning rate is set to 0.01, λ1 is set to 0.1, λ2 is set to 0.1, and the number of optimizations is set to 1000.

[0158] Step 8

[0159] Use the speech adversarial perturbation that meets the requirements obtained in step 7 to generate speech adversarial samples:

[0160]

[0161] In a digital environment, the present invention uses different speech adversarial sample generation algorithms to generate 1000 speech adversarial samples for the target speech recognition model, and evaluates the attack performance of the generated speech adversarial samples, the size of the generated speech adversarial disturbance, and the auditory perception of the generated speech adversarial samples. Figure 4 As shown in Figure 2, the attack performance of speech adversarial samples is measured by word error rate, the size of speech adversarial perturbation is measured by bi-norm, and the auditory perception of speech adversarial samples is measured by signal-to-noise ratio. Figure 4 It can be seen that the speech adversarial samples generated by the present invention are superior to those generated by the speech version of C&W, Qin and other algorithms in terms of attack performance, adversarial disturbance size, and auditory perception. The 1000 speech adversarial samples generated by the present invention in a digital environment are played in the target physical environment to obtain the attack performance of the speech adversarial samples generated by the present invention in the physical environment, as shown in FIG. Figure 5 As shown. Figure 5 It can be seen that the speech adversarial samples generated by the present invention still have good attack performance in the physical environment. Figure 6 These are the speech adversarial samples generated by the present invention and the speech version of C&W and other algorithms. As can be seen from the figure, the speech adversarial samples generated by the present invention are closer to the original benign speech samples.

[0162] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating physical speech adversarial samples that are imperceptible to human ears, characterized in that: The following steps are involved: S1, read the waveform data of the original speech sample, and generate the initial speech adversarial perturbation using a function whose domain is the set of all real numbers and whose range is 0 to 1; S2, uses the loudness of the speech to quantify the size of the initial speech adversarial perturbation and calculates the speech adversarial perturbation loss; S3, playing and recording speech in the target environment, and calculating the room impulse response of the target environment; S4, performs convolution operation on the speech adversarial sample and the room impulse response of the target environment, and calculates the inference loss of the speech adversarial sample on the target speech model; S5, using the gradient descent algorithm to iteratively optimize the speech adversarial perturbation loss and the inference loss of the speech adversarial sample on the target speech model until a speech adversarial perturbation that meets the requirements is generated; S6, the psychoacoustic loss of the generated speech against the perturbation is calculated using the psychoacoustic criterion; S7, add the psychoacoustic loss of speech adversarial perturbation to the inference loss of the speech adversarial sample on the target speech model, and use the gradient descent algorithm to iteratively optimize the three losses until the speech adversarial perturbation that meets the requirements is generated; S8, using the obtained speech adversarial perturbation that meets the requirements to generate speech adversarial samples.

2. According to claim 1, a method for generating physical speech adversarial samples that are imperceptible to human ears is characterized in that: In S1, the values ​​of the waveform data of the original speech sample are normalized to [0,1], as shown below: Where x is the original speech sample of length T.

3. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 1, characterized in that: In S1, the expression for a function whose domain is the set of all real numbers and whose range is 0 to 1 is: Among them, δ is the speech adversarial perturbation, δ∈[0,1], z∈(-∞,+∞), z is arctan(x), and this function is used to generate the speech adversarial perturbation δ.

4. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 1, characterized in that: In S2, the formula for calculating the loudness of speech is: dB(x)=20log 10 (x) The calculation formula for speech adversarial perturbation loss is:

5. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 1, characterized in that: In S3, the room impulse response calculation formula of the target environment is: in, r is the room impulse response, w is the speech after speech x is played in the target environment, * is the convolution operation, is the reverse order of the sound x.

6. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 1, characterized in that: In S4, the sample after the convolution operation of the speech adversarial sample and the room impulse response of the target environment is: (x+δ)*r r is the room impulse response, δ is the generated speech adversarial perturbation, and x is the original speech sample of length T.

7. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 1, characterized in that: In S5, the gradient descent algorithm used is Adam, which optimizes the following objectives: Among them, λ1 is a constant.

8. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 7, characterized in that: In S5, the loss function for generating directional speech adversarial samples is:

9. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 7, characterized in that: In S5, the loss function for generating non-directional speech adversarial samples is:

10. The method for generating physical speech adversarial samples that are imperceptible to human ears according to claim 1, characterized in that: In S6, the psychoacoustic loss of speech adversarial perturbation is as follows: Among them, θ x (t) is x t The psychoacoustic masking threshold of is δ t The normalized power spectrum of ; In S7, the gradient descent algorithm used is Adam, which optimizes the following objectives: Among them, λ1 and λ2 are constants.

Images