Audio adversarial example generation methods, devices, equipment and storage media

Abstract

This invention discloses a method, apparatus, device, and storage medium for generating adversarial audio examples. The method includes: acquiring text information and noise; inputting the text information into a random duration predictor to obtain the duration information of each text, aligning the duration information with the noise to obtain an alignment result; inputting the text information and alignment result into a prior encoder to encode intermediate features; inputting the intermediate features and noise into a conditional distribution generator, using noise as a condition, passing through multiple parallel long short-term memory networks and fully connected layers to obtain multiple frame-level distributions of the intermediate features under noise conditions; converting the distributions through a flow layer to an approximate distribution with the posterior distribution of the audio frames, and sampling latent variables; and decoding the latent variables to obtain the audio adversarial examples. This invention utilizes the principle of distribution sampling to conditionally add noise to a region inaudible to the human ear, making the generated audio adversarial examples more effective at deceiving the human ear.

Description

Technical Field

[0001] This invention belongs to the field of speech technology, specifically relating to a method, apparatus, device, and storage medium for generating audio adversarial samples. Background Technology

[0002] With the continuous development of artificial intelligence technology, more and more applications based on artificial intelligence technology are being widely used in life. For example, speech recognition has been widely used in people's lives and work, bringing great convenience to people's lives and work.

[0003] However, speech recognition also carries risks, such as vulnerability to audio adversarial examples. Audio adversarial examples are attacks that superimpose a perturbation onto the original audio, causing the artificial intelligence model to err in its recognition results. These examples can even alter the recognition results of modern Automatic Speech Recognition (ASR) systems, posing a security threat. Therefore, how to detect various types of audio adversarial examples is an urgent research topic.

[0004] Before using AI models to identify and detect a wide variety of audio adversarial examples, generating highly natural-sounding audio adversarial examples is crucial. VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) is a highly expressive speech synthesis model that combines variational inference, normalizing flows, and adversarial training. VITS concatenates the acoustic model and vocoder in speech synthesis using latent variables instead of the spectrum, stochastically models the latent variables, and utilizes a stochastic duration predictor, improving the diversity of synthesized speech. Given the same text, it can synthesize speech with different tones and prosody.

[0005] However, when VITS converts text to audio, it adds noise directly to the audio. While this makes it easy to generate adversarial audio samples, it obviously reduces the perceived quality of the audio for the human ear. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide an audio adversarial sample generation method, apparatus, device and storage medium to improve the above-mentioned problems.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An audio adversarial example generation method, comprising:

[0009] Obtain the text information to be processed, as well as the noise;

[0010] The text information is input into a random duration predictor to obtain the duration information of each text, and the text is aligned with the noise based on the duration information of each text to obtain the alignment result.

[0011] The text information and alignment result are input into the prior encoder to encode intermediate features based on the text information and alignment result;

[0012] The intermediate features and the noise are input into the conditional distribution generator. With noise as a condition, the intermediate features are modeled through multiple parallel long short-term memory networks and fully connected layers to obtain multiple frame-level distributions of the intermediate features under noise conditions.

[0013] The distribution is transformed into an approximate distribution of the audio posterior distribution through a flow layer, and latent variables are sampled from the approximate distribution;

[0014] The latent variables are decoded to obtain audio adversarial samples.

[0015] Preferably, the noise is a low-frequency disturbance signal, and the noise is added at the frame level in the non-overlapping area of ​​the frame; wherein, when the speech frame is segmented by sliding window, each time the window is slid, it partially overlaps with the previous frame, and this part is the overlapping area of ​​the frame, and the remaining part is the non-overlapping area.

[0016] Preferably, in the conditional distribution generator, the condition extraction is set, and the variational condition bound of the distribution sampling is optimized by obtaining a representation of the noise prior distribution; the representation includes the mean and variance.

[0017] Preferably, the intermediate features and the noise are input into a conditional distribution generator. Using noise as a condition, the generator models the frame-level information of the intermediate features through multiple parallel long short-term memory networks and fully connected layers, obtaining multiple frame-level distributions of the intermediate features under noise conditions; specifically:

[0018] The intermediate features and the noise are input into the conditional distribution generator. The noise is used as a condition to solve the distribution of the N sub-features of the intermediate features, that is, the N frame-level information separately. Each frame-level information corresponds to a speech frame to be generated.

[0019] Each frame-level perturbation of the noise is added to the corresponding frame-level information as a label and input into a Long Short-Term Memory (LSTM) network. Each LSM network is followed by a fully connected layer, resulting in N parallel structures that output the mean and variance of N frame-level information. Finally, the frame-level distributions of multiple intermediate features under noise conditions are obtained.

[0020] Preferably, it further includes:

[0021] The text information is input into a feature predictor to extract semantic information at different levels from the context of the text information;

[0022] Speech style representations at each level are predicted sequentially using residual connections;

[0023] The speaking style representation is added to the latent variables to update the latent variables.

[0024] Preferably, the feature predictor is obtained through knowledge distillation training.

[0025] Preferably, it further includes:

[0026] The multi-level features of the audio adversarial sample are passed to the discriminator for discrimination.

[0027] This invention also provides an audio adversarial example generation apparatus, comprising:

[0028] The information acquisition unit is used to acquire the text information to be processed and noise;

[0029] The alignment unit is used to input the text information into the random duration predictor to obtain the duration information of each text, and to align the text with noise based on the duration information of each text to obtain the alignment result.

[0030] The encoding unit is used to input the text information and alignment result into the prior encoder to encode intermediate features based on the text information and alignment result;

[0031] The conditional distribution unit is used to input the intermediate features and the noise into the conditional distribution generator. Using the noise as a condition, the intermediate features are modeled separately through multiple parallel long short-term memory networks and fully connected layers to obtain multiple frame-level distributions of the intermediate features under noise conditions.

[0032] The distribution transformation unit is used to transform the frame-level distribution into an approximate distribution with respect to the audio posterior distribution through the flow layer, and to sample latent variables from the approximate distribution.

[0033] The decoding unit is used to decode the latent variables to obtain audio adversarial samples.

[0034] This invention also provides an audio adversarial sample generation device, which includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the audio adversarial sample generation method described above.

[0035] This invention also provides a computer-readable storage medium storing a computer program that can be executed by the processor to implement the audio adversarial sample generation method described above.

[0036] In summary, this embodiment utilizes the principle of distributed sampling to add noise to areas inaudible to the human ear, enabling the generated audio adversarial examples to better deceive the human ear, thereby improving the naturalness and concealment of the audio adversarial examples. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the audio adversarial sample generation method provided in the first embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram illustrating the working principle of the audio adversarial sample generation method according to the first embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram of the audio adversarial sample generation device provided in the second embodiment of the present invention. Detailed Implementation

[0040] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0041] Please see Figure 1 The first embodiment of the present invention provides an audio adversarial example generation method, which can be executed by an audio adversarial example generation device (hereinafter referred to as the generation device), specifically, by one or more processors within the generation device, to achieve the following steps:

[0042] S101, acquire the text information to be processed and the noise.

[0043] In this embodiment, the generating device can be a terminal device with computing power, such as a personal computer, workstation, or server; the present invention does not impose any specific limitations.

[0044] In this embodiment, the text length of the text information can be set according to actual needs.

[0045] The noise is a relatively low-frequency audio disturbance signal to the human ear.

[0046] S102, the text information is input into a random duration predictor to obtain the duration information of each text, and the duration information of each text is aligned with the noise to obtain the alignment result.

[0047] In this embodiment, the random duration predictor is a stream-based generative model that introduces random variables with the same temporal resolution and dimension as the duration sequence. These two variables are sampled using an approximate posterior distribution, and the training objective is the variational lower bound of the log-likelihood of the phoneme duration. During training, the gradient backpropagation of the random duration predictor needs to be disconnected to prevent this gradient from affecting other modules. The phoneme duration is sampled from random noise through an invertible transformation of the random duration predictor and then converted to an integer value.

[0048] In this embodiment, since the true labels are not aligned during training, the alignment between the text and the audio (i.e., noise) needs to be estimated at each iteration of the training phase.

[0049] To estimate the alignment between text and audio, this embodiment, like VITS, employs a method similar to the Monotonic Alignment Search (MAS) in Glow-TTS, which attempts to find an optimal alignment path to maximize the log-likelihood of the normalized stream parameterized data.

[0050] The optimal alignment obtained by MAS constraints must be monotonic and without skipping. However, MAS cannot be directly applied to VITS because the optimization objective of VITS is ELBO rather than the log-likelihood of deterministic latent variables. Therefore, MAS was slightly modified to find the optimal alignment path to maximize ELBO.

[0051] In practice, the random duration predictor takes the result of the text encoder as input, rather than the latent variables of the normalized stream after the text encoder, and outputs the logarithm of the phoneme duration. The text-encoded tensor first undergoes a preprocessing one-dimensional convolution, then a holed and depth-separable convolution (DDSConv), followed by a post-processing one-dimensional convolution, and finally enters a neural spline flow to output the logarithm of the phoneme duration.

[0052] Dialed and Depth-Separable Convolution (DDSConv) improves parameter utilization efficiency while maintaining a large receptive field. In DDSConv, each convolutional layer is followed by layer normalization and the GELU activation function. Specifically, the input first enters a grouped one-dimensional convolution, where the number of groups and channels is the same, and the dilation coefficient increases exponentially with the number of layers. This is followed by one-dimensional convolution and Dropout. A module composed of multiple grouped convolutions and one-dimensional convolutions constitutes DDSConv, and the output of each module is added to the input as a residual element.

[0053] S103, the text information and alignment result are input into the prior encoder to encode intermediate features based on the text information and alignment result.

[0054] In this embodiment, the prior encoder includes a text encoder and a normalized stream. The text encoder consists of TransformerBlocks. The text encoder returns x, m, and logs based on the text information. x is used to calculate the phoneme duration, and m and logs are weighted and summed by attention weights to obtain the prior latent variable z_p in residual form.

[0055] TransformerBlocks consists of a multi-head attention (MA) network and a feedforward network (FFN). In its implementation, to adapt to speech synthesis tasks, relative attention is used to encourage self-attention to focus on neighboring locations (with a specified window size of 4). The text encoder's feedforward network consists of two convolutional layers: conv1->relu->dropout->conv2. The feedforward network in the text encoder uses equal-length convolutions.

[0056] The normalized flow in the prior encoder is a module consisting of four affine coupling layers. Each coupling layer contains four WaveNet residual structures to enhance the expressive power of the prior encoder. Since VITS restricts the coupling layers to volume-preserving transformations, they do not generate scaling parameters. In the specific implementation described above, the coupling layer consists of a residual coupling layer and a flip layer. The residual coupling layer is actually a WaveNet residual structure. Specifically, the WaveNet residual module continuously increases the receptive field by increasing the dilation coefficient of the one-dimensional dilated convolution (dilated convolution). A portion of the elements from the convolution result is added to the input of the next layer, and the remaining elements are added to the final output.

[0057] In its implementation, the standardized stream uses a gated feature fusion method when adding speaker embedding vectors. Half of the input in_act is processed by sigmoid and then multiplied element-wise by the result of the other half of in_act after tanh activation, thus obtaining the intermediate features.

[0058] S104, the intermediate features and the noise are input to the conditional distribution generator. Using noise as a condition, the intermediate features are modeled through multiple parallel long short-term memory networks and fully connected layers to obtain multiple frame-level distributions of the intermediate features under noise conditions.

[0059] In the conditional distribution generator, noise in the region inaudible to the human ear is added as a condition. This condition can be set to conform to a standard normal distribution or a normal distribution, but it must be set in advance.

[0060] Specifically, when adding the noise, it is done at the frame level, in the non-overlapping regions of the frames. When segmenting speech frames using a sliding window, each slide of the window partially overlaps with the previous frame; this overlapping portion is the overlapping region of that frame, and the remaining portion is the non-overlapping region.

[0061] In particular, considering that the two frames influence each other and have a greater coupling effect in the overlapping area, more noise is added in the non-overlapping area and less noise is added in the overlapping area to achieve better results.

[0062] Specifically:

[0063] The intermediate features and the noise are input into the conditional distribution generator. The noise is used as a condition to solve the distribution of the N sub-features of the intermediate features, that is, the N frame-level information separately. Each frame-level information corresponds to a speech frame to be generated.

[0064] Each frame-level perturbation of the noise is added to the corresponding frame-level information as a label and input into a Long Short-Term Memory (LSTM) network. Each LSM network is followed by a fully connected layer, resulting in N parallel structures that output the mean and variance of N frame-level information. Finally, the frame-level distributions of multiple intermediate features under noise conditions are obtained.

[0065] S105, the frame-level distribution is transformed into an approximate distribution with the audio posterior distribution through the flow layer, and latent variables are sampled from the approximate distribution.

[0066] In this embodiment, the flow layer contains multiple (e.g., 16) WaveNet residual structures. After the input frame-level distribution, the flow layer is converted into an approximate distribution with the audio posterior distribution, and latent variables are sampled from the approximate distribution. For example, a 192-dimensional posterior latent variable can be output.

[0067] S106, decode the latent variables to obtain audio adversarial samples.

[0068] In this embodiment, the decoder can be the generator of HiFiGAN V1, which mainly consists of multiple transposed convolutions. Each transposed convolution is followed by a multi-receptive field fusion (MRF) module. The so-called multi-receptive field fusion module is mainly a residual module composed of equal-sized one-dimensional convolutions.

[0069] In summary, this embodiment utilizes the principle of distributed sampling to add noise to areas inaudible to the human ear, thereby enabling the generated audio adversarial examples to better deceive the human ear, thus improving the naturalness and concealment of the audio adversarial examples.

[0070] Preferably, it further includes:

[0071] The text information is input into a feature predictor to extract semantic information at different levels from the context of the text information, and then the speech style representation at each level is predicted sequentially using residual connections.

[0072] The speaking style representation is added to the latent variables to update the latent variables.

[0073] In this embodiment, to further improve the model's performance, a feature extractor and a feature predictor are added to the model.

[0074] The feature extractor was used to extract speech style representations at three different levels: global, sentence, and speech segments corresponding to each word.

[0075] Based on the feature extractor, the feature predictor extracts semantic information from different levels of context, and then predicts the speech style representation of each level in sequence by means of residual connections.

[0076] To avoid interference between multi-scale style learning during training, the paragraph-level, sentence-level, and word-level reference encoders and style tagging layers are trained sequentially. While training modules at one level, the modules at the other levels are frozen. After the feature extractor is trained, it is frozen, and the style representations extracted by the extractor are used as the training target for the feature predictor, enabling the predictor to better establish the connection between different levels of speech style and text semantics.

[0077] Preferably, it further includes:

[0078] The multi-level features of the audio adversarial sample are passed to the discriminator for discrimination.

[0079] In this embodiment, a multi-scale discriminator is added to determine whether the output of the variational inference decoder is real audio or fake audio. This is used for training the variational inference decoder and improving the generation quality of adversarial audio examples.

[0080] The core of the multi-scale discriminator is to first perform average pooling to shorten the sequence length, with each pooling operation reducing the sequence length to half its original size, followed by convolution. Specifically, the multi-scale discriminator first performs "original-size discrimination" on the original sample points, using spectral normalization (spectral_norm) as the parameter normalization method for one-dimensional convolution. Next, it performs average pooling on the sample point sequence, halving the sequence length each time, and then discriminates the "downsampled" sample point sequence, using weight normalization (weight_norm) as the parameter normalization method for one-dimensional convolution. In each sub-discriminator at a specific scale, several convolutional layers are first performed, all using grouped convolution, and the parameters are normalized using the corresponding method. Then, leaky ReLU activation is applied. After multiple convolutional layers, a final post-processing layer with one output channel is used to flatten the output.

[0081] Please see Figure 3 The second embodiment of the present invention also provides an audio adversarial sample generation apparatus, which includes:

[0082] Information acquisition unit 210 is used to acquire text information to be processed and noise;

[0083] Alignment unit 220 is used to input the text information into a random duration predictor to obtain the duration information of each text, and to align the text with noise based on the duration information of each text to obtain the alignment result;

[0084] Encoding unit 230 is used to input the text information and alignment result into the prior encoder to encode intermediate features based on the text information and alignment result;

[0085] The conditional distribution unit 240 is used to input the intermediate features and the noise into the conditional distribution generator, using the noise as a condition, and through multiple parallel long short-term memory networks and fully connected layers, to separately model the frame-level information of the intermediate features, and obtain multiple frame-level distributions of the intermediate features under noise conditions.

[0086] The distribution transformation unit 250 is used to transform the frame-level distribution into an approximate distribution of the audio posterior distribution through the flow layer, and to sample the latent variables from the approximate distribution;

[0087] Decoding unit 260 is used to decode the latent variables to obtain audio adversarial samples.

[0088] The third embodiment of the present invention also provides an audio adversarial sample generation device, which includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement the audio adversarial sample generation method described above.

[0089] The fourth embodiment of the present invention also provides a computer-readable storage medium storing a computer program that can be executed by the processor to implement the audio adversarial sample generation method as described above.

[0090] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0091] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0092] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating audio adversarial examples, characterized in that, include: Obtain the text information to be processed, as well as the noise; The text information is input into a random duration predictor to obtain the duration information of each text, and the text is aligned with the noise based on the duration information of each text to obtain the alignment result. The text information and alignment result are input into the prior encoder to encode intermediate features based on the text information and alignment result; The intermediate features and the noise are input into the conditional distribution generator. With noise as a condition, the intermediate features are modeled through multiple parallel long short-term memory networks and fully connected layers to obtain multiple frame-level distributions of the intermediate features under noise conditions. The frame-level distribution is transformed into an approximate distribution of the audio posterior distribution through a flow layer, and latent variables are sampled from the approximate distribution. The latent variables are decoded to obtain audio adversarial samples.

2. The audio adversarial example generation method according to claim 1, characterized in that, The noise is a low-frequency disturbance signal, and it is added at the frame level to the non-overlapping area of ​​the frame. When the speech frame is segmented by sliding window, each time the window is slid, it partially overlaps with the previous frame. This part is the overlapping area of ​​the frame, and the remaining part is the non-overlapping area.

3. The audio adversarial example generation method according to claim 1, characterized in that, In the conditional distribution generator, the conditions are pre-set, and the variational conditional bound of the distribution sampling is optimized by obtaining a representation of the noise prior distribution; the representation includes the mean and variance.

4. The audio adversarial example generation method according to claim 1, characterized in that, The intermediate features and the noise are input into a conditional distribution generator. Using noise as a condition, the generator passes through multiple parallel long short-term memory networks and fully connected layers to model the frame-level information of the intermediate features, obtaining multiple frame-level distributions of the intermediate features under noise conditions. Specifically: The intermediate features and the noise are input into the conditional distribution generator. The noise is used as a condition to solve the distribution of the N sub-features of the intermediate features, that is, the N frame-level information separately. Each frame-level information corresponds to a speech frame to be generated. Each frame-level perturbation of the noise is added to the corresponding frame-level information as a label and input into a Long Short-Term Memory (LSTM) network. Each LSM network is followed by a fully connected layer, resulting in N parallel structures that output the mean and variance of N frame-level information. Finally, the frame-level distributions of multiple intermediate features under noise conditions are obtained.

5. The audio adversarial example generation method according to claim 1, characterized in that, Also includes: The text information is input into a feature predictor to extract semantic information at different levels from the context of the text information; Speech style representations at each level are predicted sequentially using residual connections; The speaking style representation is added to the latent variables to update the latent variables.

6. The audio adversarial example generation method according to claim 5, characterized in that, The feature predictor is obtained through knowledge distillation training.

7. The audio adversarial example generation method according to claim 1, characterized in that, Also includes: The multi-level features of the audio adversarial sample are passed to the discriminator for discrimination.

8. An audio adversarial example generation device, characterized in that, include: The information acquisition unit is used to acquire the text information to be processed and noise; The alignment unit is used to input the text information into the random duration predictor to obtain the duration information of each text, and to align the text with noise based on the duration information of each text to obtain the alignment result. The encoding unit is used to input the text information and alignment result into the prior encoder to encode intermediate features based on the text information and alignment result; The conditional distribution unit is used to input the intermediate features and the noise into the conditional distribution generator. Using the noise as a condition, the intermediate features are modeled through multiple parallel long short-term memory networks and fully connected layers to obtain multiple frame-level distributions of the intermediate features under noise conditions. The distribution transformation unit is used to transform the frame-level distribution into an approximate distribution of the audio posterior distribution through the flow layer, and to sample the latent variables from the approximate distribution; The decoding unit is used to decode the latent variables to obtain audio adversarial samples.

9. An audio adversarial example generation device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that can be executed by the processor to implement the audio adversarial sample generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor to implement the audio adversarial sample generation method as described in any one of claims 1 to 7.

Images