Phoneme-level voiceprint recognition adversarial sample construction system and method based on neural network generation model

Through the phoneme-level voiceprint recognition adversarial sample construction system based on neural network generation model, the problem of vulnerability of existing voiceprint recognition systems is solved, and the adversarial sample generation with low complexity, high versatility and migration is achieved, which satisfies the Real-time and efficient attack requirements.

CN114093371BActive Publication Date: 2024-12-03ZHEJIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111178765.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-11
Publication Date
2024-12-03
Estimated Expiration
2041-10-11

AI Technical Summary

Technical Problem

The existing voiceprint recognition system is susceptible to adversarial sample attacks, and the existing attack methods are complex, time-consuming, and cannot be generalized and have poor migration, which cannot meet the real-time and efficient attack needs.

Method used

The phoneme-level voiceprint recognition adversarial sample construction system is adopted based on neural network generation model. Through the phoneme recognizer, perturbation generator, hearing suppressor and system optimizer, speech is deconstructed into phoneme sequences and pre-trained general adversarial perturbation is applied. Generate text-independent and individual-independent voice confrontation samples to realize black box attacks and targeted imitation attacks.

Benefits of technology

It realizes low complexity, high versatility and migration, improves attack efficiency and effectiveness, meets real-time requirements, and shows good migration performance on different voiceprint recognition models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114093371B_ABST
    Figure CN114093371B_ABST
Patent Text Reader

Abstract

The present invention discloses a system and method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model. The system includes a phoneme recognizer, a perturbation generator, an auditory suppressor, an alternative classifier, and a system optimizer. The suppressed perturbation generated by the auditory suppressor is superimposed with the aligned speech generated by the phoneme recognizer to generate an adversarial sample. The adversarial sample is classified by the alternative classifier and the score is sent to the system optimizer, and the gradient is back-propagated to the perturbation generator for iterative update to obtain a trained phoneme perturbation generator. The present invention innovatively proposes to integrate the phoneme information in the speech for perturbation construction, use the phoneme recognizer to identify and locate the phonemes in the speech, and generate fine-grained universal adversarial perturbations at the phoneme level, so that the adversarial perturbations generated in one step can be reused for any speech text input, and finally realize the generation of universal adversarial samples that are text-independent and input-independent, greatly improving the generation efficiency of adversarial samples.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of voiceprint recognition and adversarial samples, and in particular to a system and method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model. Background Art

[0002] As the most natural and direct way of communication for humans, voice has always occupied an important position in the field of human-computer interaction and individual identification, and has gradually become a popular biometric technology. At the same time, thanks to the rapid development of deep learning theory and technology, voiceprint recognition has been applied to various mature products (such as voice assistants, voiceprint locks, etc.) to provide intelligent services for people's work and life. Studies have shown that the global voice biometrics market size will exceed 1.1 million US dollars in 2020 and is expected to reach 3.9 million US dollars in 2026, fully demonstrating the broad development prospects of voice technology. However, behind the bright prospects of voiceprint recognition, the shadow of deep learning being vulnerable to adversarial sample attacks poses a serious security threat. Recent studies have shown that applying slight perturbations to normal speech can easily deceive voiceprint recognition systems based on deep neural networks, that is, voiceprint recognition is vulnerable to imitation attacks based on adversarial samples, which has attracted widespread public attention and a lot of research interest. The present invention uses a generative model to construct phoneme-level adversarial perturbations, thereby generating general and transferable voice adversarial samples with low complexity, and finally realizing targeted imitation attacks to deceive voiceprint recognition systems.

[0003] Adversarial sample attacks are a type of usability attack in the model reasoning phase. They impose human-imperceptible disturbances on normal speech, which are continuously amplified by the nonlinear structure of the neural network, and eventually cause the target model to make incorrect decisions. Existing adversarial sample attacks in the voiceprint field are divided into white-box attacks and black-box attacks: white-box attacks assume that the attacker knows the internal details of the target model, but the attack performance on unknown models is significantly reduced, so its practical significance is greatly limited; black-box attacks assume that the attacker has no prior knowledge of the target model, but a large number of query accesses to the target model are required to estimate the gradient, which lacks feasibility in actual attack scenarios. In addition, existing attack methods generate one-time adversarial samples for a single certain speech through complex iterative optimization algorithms, which have the disadvantages of input determination, time-consuming, and non-transferable. Summary of the invention

[0004] The present invention proposes a phoneme-level voiceprint recognition adversarial sample construction system and method based on a neural network generation model. With the high versatility, low complexity and transferability of the attack as the goal, the speech is deconstructed into a phoneme sequence and then a pre-trained universal adversarial perturbation is applied. It can generate text-independent and individual-independent speech adversarial samples in real time, and has good migration ability on different voiceprint recognition models, thereby realizing black-box attacks.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention discloses a phoneme-level voiceprint recognition adversarial sample construction system based on a neural network generation model. The system includes a phoneme recognizer, a disturbance generator, a hearing suppressor, a substitute classifier and a system optimizer. The phoneme recognizer generates aligned speech, the disturbance generator generates phoneme-level disturbance to the hearing suppressor, the hearing suppressor generates suppressed disturbance and superimposes the aligned speech generated by the phoneme recognizer to generate an adversarial sample. The adversarial sample is classified by the substitute classifier and the score is sent to the system optimizer, and the gradient is back-propagated to the disturbance generator for iterative update, so as to finally obtain a trained phoneme disturbance generator.

[0007] The present invention also discloses a method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model. The method includes an offline training phase and an online attack phase, including the following steps:

[0008] Offline training phase:

[0009] 1) The phoneme recognizer decomposes the speech in the multi-speaker speech dataset into phoneme sequences through forced alignment technology, recognizes all phonemes and locates their time segments in the speech;

[0010] 2) The perturbation generator defines all the phonemes used, and automatically generates a fixed-dimensional phoneme-level perturbation for each phoneme through a multi-layer neural network. According to the phonemes and their position information obtained by the phoneme recognizer, the corresponding phoneme-level perturbation is injected into the time period when each phoneme appears in the speech to obtain the adversarial perturbation of the speech;

[0011] 3) In order to prevent the distortion caused by the disturbance from being perceived by the human ear, the hearing suppressor combines a variety of signal processing techniques to constrain the disturbance generated by the disturbance generator and generate a suppressed disturbance;

[0012] 4) The suppressed perturbation is superimposed on the aligned speech generated by the phoneme recognizer to obtain the corresponding speech adversarial sample, which is input into the pre-trained alternative classifier for voiceprint recognition;

[0013] 5) Based on the recognition results of the alternative classifier, the system optimizer calculates the target loss and back-propagates the gradient to the perturbation generator for iterative update, and finally obtains a trained phoneme perturbation generator;

[0014] Online attack phase:

[0015] The attacker inputs his voice into the system, obtains the phonemes and their position information through the phoneme recognizer, and injects the offline pre-trained phoneme-level adversarial perturbations to construct a voice adversarial sample. The attacker can then inject the adversarial sample into the target system for a targeted attack, thereby impersonating the target user and bypassing the system's identity authentication.

[0016] As a further improvement, in step 1) of the present invention, the forced alignment technology is to time-align the audio clips and the text content, and determine the time period of each phoneme in the speech, and finally output each phoneme in the speech and its position information. The forced alignment tool used by the forced alignment technology is Montreal Forced Aligner.

[0017] As a further improvement, in step 2) of the present invention, the disturbance generator is a multi-layer neural network, including an input layer, three hidden layers and an output layer.

[0018] As a further improvement, the workflow of the perturbation generator described in the present invention is as follows: the input layer is m×n random noise, which is mapped to the perturbation space through three layers of m×kn hidden layers and ReLU activation function, and then the m×n output layer is regularized by the Tanh activation function to obtain the perturbation; wherein the parameters m, n, and k represent the number of phonemes, the perturbation dimension, and the hidden layer scaling factor, respectively.

[0019] As a further improvement, the hearing suppressor of the present invention constrains disturbances, including low-pass filtering, amplitude clipping and windowing smoothing.

[0020] 1) Low-pass filtering: A digital biquad filter is applied to remove the high-frequency components of the disturbance;

[0021] 2) Amplitude shearing: using L ∞ Regularization is amplitude-limited, which is achieved through an amplitude clipping operation:

[0022] δ = clip{δ, -∈, ∈}, where δ is the adversarial disturbance and ∈ is the upper limit of the amplitude;

[0023] 3) Windowing smoothing: A Hanning window function is used to smooth the boundaries of the phonemes, and the window length is determined by the phoneme length.

[0024] As a further improvement, the alternative classifier described in the present invention includes multiple pre-trained mainstream voiceprint recognition models, which are any one of d-vector, x-vector and DeepSpeaker.

[0025] As a further improvement, for the speech and text datasets (X, T) = {(x1, t1), (x2, t2), ..., (x n, t n )}, the system optimizer is used to obtain a trained perturbation generator G(·) to generate perturbations δ and construct adversarial samples As a result, the alternative classifier C(·) misidentifies it as the target user y a .

[0026] As a further improvement, the present invention specifically includes the following steps:

[0027] 1) Initialize the disturbance generator G(·), random noise z, amplitude upper limit ∈ and cutoff frequency f. In each iteration, load a batch of samples After the forced alignment of the phoneme recognizer, each phoneme p and its time segment [s, e) are obtained. At the same time, the perturbation generator maps the random noise z to the perturbation space, that is, δ = G(z);

[0028] 2) Apply low-pass filtering, amplitude clipping, and windowing smoothing to the disturbance in turn through the auditory suppressor to suppress the auditory perception;

[0029] 3) The suppressed perturbation is injected into the aligned speech according to the alignment information and a time offset Generate adversarial examples on top Then input it into the alternative classifier C(·) to get the recognition result y pred and the score s of the target user a , based on this result, the system optimizer calculates the following loss:

[0030] L(y a ,y pred )=max{θ-s a , -κ}.

[0031] Among them, θ is the preset threshold of the alternative classifier, and κ is the confidence introduced to ensure that the score s a is enough to exceed the threshold θ and be successfully identified as the target user y a ;

[0032] 4) Based on the objective function, the system optimizer uses the gradient descent method to continuously update the disturbance generator until the early stopping condition is met.

[0033] The beneficial effects of the present invention are as follows:

[0034] The present invention proposes a system and method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model. The existing speech adversarial sample attack constructs a one-time adversarial perturbation for a single speech, and needs to repeat the perturbation generation process for each speech. There are problems such as poor universal performance of the perturbation and low generation efficiency. The present invention innovatively proposes to integrate the phoneme information in the speech for perturbation construction, utilizes a phoneme recognizer to identify and locate the phonemes in the speech, and generates fine-grained universal adversarial perturbations at the phoneme level, so that the adversarial perturbations generated in one step can be reused for any speech text input, and finally realizes the generation of text-independent and input-independent universal adversarial samples, greatly improving the generation efficiency of adversarial samples.

[0035] Existing speech adversarial sample attacks use complex iterative optimization algorithms for gradient estimation and perturbation construction. Each generation requires a lot of time and cannot meet the real-time requirements of real scenarios. The present invention proposes to decouple the offline adversarial perturbation construction and online adversarial sample generation stages, and use a deep neural network generative model to learn the complex mapping of Gaussian random noise to high-dimensional perturbation space. The two stages of offline pre-training and online forward reasoning of the generative model are combined to realize real-time adversarial sample generation, which effectively meets the real-time requirements in real scenarios.

[0036] In order to ensure the imperceptibility of adversarial disturbances, existing speech adversarial sample attacks usually use amplitude suppression to suppress audibility. However, full-band amplitude suppression can easily cause serious degradation of adversarial sample performance. The present invention fully considers the auditory influencing factors of adversarial disturbances, and uses amplitude clipping, low-pass filtering, windowing and smoothing and other signal processing techniques to suppress the auditory perception of disturbances from multiple dimensions such as amplitude, frequency, and window boundary, effectively reducing the signal distortion caused by superimposed adversarial disturbances, minimizing the audibility of disturbances while ensuring the performance of adversarial disturbances, and ultimately achieving the generation of adversarial samples that are imperceptible to the human ear.

[0037] The present invention constructs a phoneme-level voiceprint recognition adversarial sample attack system based on a generative model, which can not only generate text-independent and individual-independent universal adversarial samples with low complexity, but also has good migration performance on different data sets and different voiceprint recognition models. The present invention can adapt to different speaking individuals, speech texts and target systems, and effectively achieve high versatility, low complexity and transferability of targeted adversarial sample attacks.

[0038] In the attack effectiveness and disturbance imperceptibility evaluation, the present invention achieves a higher SNR and an ASR improvement of more than 15% compared to RURA, and achieves twice the SNR and an ASR improvement of 3% compared to AdvPulse. Compared to these two methods designed to attack only the x-vector system, the ASR of the present invention reaches 100.00% and 99.93% on the d-vector and DeepSpeaker systems, respectively, which fully proves the effectiveness of the present invention on different systems.

[0039] In the migration evaluation, only 7 out of 30 types of black box attacks have an average ASR lower than 80%, which proves the migration of the present invention to different target systems. In order to further analyze the impact of the data set and network model of the target system, the performance evaluation of white box attack and other three black box attacks, namely cross-dataset, cross-model, and cross-data and model, is summarized. It is concluded that the present invention achieves 89.0%, 86.4% and 79.2% ASR respectively under these three black box attacks, with only 11%, 13% and 20% performance degradation compared to white box attacks, which proves the good performance of the present invention under black box attacks.

[0040] In the computational complexity evaluation, different from the existing attack methods based on iterative optimization, the present invention can apply pre-trained phoneme-level perturbations to any speech, greatly reducing the time cost of generating adversarial samples. The present invention can effectively reduce the computational complexity.

[0041] In the universality evaluation, the invention can adapt to different speakers and speech texts, and shows good attack performance both under intra-gender and inter-gender conditions, and is robust to the speaker's gender. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a system framework diagram of the present invention;

[0043] Figure 2 is a network structure diagram of a disturbance generator in the system of the present invention;

[0044] Figure 3 This is an example diagram of the effect of the hearing suppressor;

[0045] Figure 4 The transferability diagram of the present invention on different data sets and different models includes (a) the transfer attack confusion matrix and (b) the attack performance under different attack settings;

[0046] Figure 5 Graph showing the versatility of the present invention under different attackers and different genders, including (a) attack performance on different attackers and (b) attack performance under different genders. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is further described below by means of specific embodiments:

[0048] The present invention proposes a system and method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model. Figure 1 The system framework diagram of the present invention is shown. The whole process is divided into an offline training phase and an online attack phase, which includes five parts, namely, a phoneme recognizer, a disturbance generator, an auditory sense suppressor, an alternative classifier, and a system optimizer. During the offline training phase, the phoneme recognizer decomposes the speech in the multi-speaker speech dataset into phoneme sequences through forced alignment technology, recognizes all phonemes and locates their time periods in the speech; the perturbation generator defines all the phonemes used, and automatically generates fixed-dimensional phoneme-level perturbations for each phoneme through a multi-layer neural network. According to the phonemes and their position information obtained by the phoneme recognizer, the corresponding phoneme-level perturbation is injected into the time period where each phoneme appears in the speech to obtain the adversarial perturbation of the speech; in order to prevent the distortion caused by the perturbation from being perceived by the human ear, the hearing suppressor combines a variety of signal processing techniques to constrain the perturbation; the suppressed perturbation is superimposed on the aligned speech to obtain the corresponding speech adversarial sample, which is input into the pre-trained alternative classifier for voiceprint recognition; based on the recognition results of the alternative classifier, the system optimizer calculates the target loss and back-propagates the gradient to the perturbation generator for iterative update, and finally obtains a trained phoneme perturbation generator.

[0049] During the online attack phase, the attacker inputs his voice into the system, obtains the phonemes and their position information through the phoneme recognizer, and injects the offline pre-trained phoneme-level adversarial perturbations to construct a voice adversarial sample. The attacker can then inject the adversarial sample into the target system for a targeted attack, thereby impersonating the target user and bypassing the system's identity authentication.

[0050] First, the phoneme recognizer is introduced. In order to construct perturbations at the phoneme level, it is first necessary to identify and locate the phonemes in the speech. Since the attacker can freely specify the text content of the attack, that is, the text of the adversarial sample is known, and its phoneme sequence can be obtained through a phoneme dictionary, the problem is transformed into the time alignment of the speech signal and the phoneme text, and the phoneme text alignment can be solved by the forced alignment technology. The forced alignment technology can time-align the audio clip and the text content and determine the time period of each phoneme in the speech. There are many popular forced alignment tools. Considering the advantages in usability and compatibility, Montreal Forced Aligner (MFA) is used as the phoneme alignment tool. Input a piece of speech and its text, MFA realizes phoneme alignment through pre-trained voiceprint and phoneme models and a preset phoneme dictionary, and finally outputs each phoneme in the speech and its position information.

[0051] Next, the design of the perturbation generator is introduced. The perturbation generator first defines the phonemes in the English language, which can be represented by the ASCII character set, as shown in Table 1.

[0052]

[0053] According to different pronunciation methods, the 40 phoneme labels in the table are divided into stops, fricatives, affricates, nasals, glides, liquids and vowels. Considering that the silent segments will be discarded during the voice activity monitoring process of the voiceprint recognition system, there is no need to construct and inject disturbances at the silent locations. Therefore, after removing / SIL / , there are 39 phonemes left. Then a multi-layer neural network is designed as a disturbance generator. Figure 2 The network structure diagram of the disturbance generator in the system of the present invention is shown in FIG. The input layer is m×n random noise, which is mapped to the disturbance space through three layers of m×kn hidden layers and ReLU activation function, and then the m×n output layer is regularized by the Tanh activation function to obtain the disturbance. The parameters m, n, and k represent the number of phonemes, the disturbance dimension, and the hidden layer scaling factor, respectively. m is set to the number of phonemes 39 in Table 1, n is set to 200 according to the phoneme length, and k is set to 4 according to empirical research. With the phoneme label as the key and the phoneme disturbance as the value, a phoneme disturbance dictionary can be obtained. According to the phonemes and their position information obtained by the phoneme recognizer, the generated phoneme disturbance can be injected into the corresponding position of the speech to form the disturbance of the entire speech.

[0054] The generated adversarial perturbations are significantly different from normal speech, especially the presence of a large number of high-frequency components and high-amplitude peaks that are easily perceived by the human ear. Figure 3 This is an example of the effect of the hearing suppressor. Figure 3 (a) and 3(b) show the time-frequency spectrum of a normal speech and its generated adversarial sample. It can be seen that Figure 3 In (b), a large number of high-frequency and high-amplitude parts mask the soundprint characteristics of the original audio. In order to make the adversarial perturbation imperceptible, the system uses an auditory suppressor to constrain the perturbation, which includes low-pass filtering, amplitude clipping, and windowing smoothing. A variety of signal processing techniques: 1) Low-pass filtering: First, a digital biquad filter is applied to filter out the high-frequency components of the perturbation, with a cutoff frequency set to 2kHz and an energy loss factor of 0.707 by default. Figure 3 (c) shows the time-frequency spectrum after filtering, comparing Figure 3 (b) The amplitude of the high frequency part is significantly reduced and effectively suppressed. 2) Amplitude shearing: Then L ∞ Regularization is performed to limit the amplitude, which is specifically achieved through an amplitude clipping operation: δ = clip{δ, -∈, ∈}, where δ is the adversarial disturbance and ∈ is the upper limit of the amplitude. Figure 3 (d) shows the time-frequency spectrum after amplitude clipping. Figure 3 (c) Its amplitude is significantly compressed in the whole frequency band. 3) Windowing smoothing: Finally, a Hanning window function is used to smooth the boundaries of the phoneme, and its window length is determined by the phoneme length. Figure 3 (e) shows the smoothed time-frequency spectrum. Figure 3 (d) The amplitude changes at the phoneme boundary are smoother. Figure 3 The time-frequency spectrum in (e) is closer Figure 3 The original time-frequency spectrum in (a), that is, the distortion caused by the adversarial perturbation, is effectively suppressed.

[0055] The suppressed adversarial perturbation is superimposed on the original speech to generate the corresponding speech adversarial samples, which are input into the pre-trained alternative classifier for voiceprint recognition. In the black box setting, the attacker cannot know the internal details of the target system, so the system turns to the cross-model migration ability of the adversarial samples for attack, that is, the adversarial samples generated for one source model have the possibility of successfully attacking another target model. Therefore, in order to enhance the transferability of adversarial samples, the system provides multiple pre-trained mainstream voiceprint recognition models as alternative classifiers for the target model, such as d-vector, x-vector and DeepSpeaker. In addition, the alternative classifier adopts PLDA, which is commonly used in voiceprint recognition, for scoring.

[0056] Based on the recognition results of the alternative classifiers, the system optimizer performs global optimization. For speech and text datasets (X, T) = ((x1, t1), (x2, t2), ..., (x n , t n )}, the system optimizer aims to obtain a trained perturbation generator G(·) to generate perturbations δ and construct adversarial samples As a result, the alternative classifier C(·) misidentifies it as the target user y a First, initialize the disturbance generator G(·), random noise z, amplitude upper limit ∈ and cutoff frequency f. In each iteration, load a batch of samples After the forced alignment of the phoneme recognizer, each phoneme p and its time segment [s, e] are obtained. At the same time, the perturbation generator maps the random noise z to the perturbation space, that is, δ = G(z), and then applies low-pass filtering, amplitude clipping, and windowing smoothing to the perturbation in turn to suppress the auditory perception. The suppressed perturbation is injected into the aligned speech according to the alignment information and a time offset. Generate adversarial examples on top Then input it into the alternative classifier C(·) to get the recognition result y pred and the score s of the target user a Based on this result, the system optimizer calculates the following loss:

[0057] L(y a ,y pred )=max{θ-s a , -κ}.

[0058] Among them, θ is the preset threshold of the alternative classifier, and κ is the confidence introduced to ensure that the score s a is enough to exceed the threshold θ and be successfully identified as the target user y a Based on this objective function, the system optimizer uses the gradient descent method to continuously update the disturbance generator until the early stopping condition is met.

[0059] The present invention constructs a phoneme-level voiceprint recognition adversarial sample attack system based on a generative model, which can not only generate text-independent and individual-independent universal adversarial samples with low complexity, but also has good migration performance on different data sets and different voiceprint recognition models. The present invention can adapt to different speaking individuals, speech texts and target systems, and effectively achieve high versatility, low complexity and transferability of targeted adversarial sample attacks.

[0060] In order to verify the effectiveness of the present invention, the present invention was trained and deployed on the large-scale corpus LibriSpeech. The corpus contains more than 110 hours of speech data from 331 speakers with different accents, occupations and ages. The speech text covers 8,000 e-books and nearly 900,000 words. 10 speakers were selected as target users and another 40 speakers were selected as attackers. Then, the perturbation generator of the present invention was trained on the remaining 281 speakers' speech. During training, the perturbation dimension n=200, the confidence κ=50, the amplitude upper limit ∈=0.02, and the cutoff frequency f=2kHz. For the target system, another large-scale corpus VoxCeleb was evenly divided into two parts VoxCeleb P1 and P2. Three mainstream voiceprint recognition models d-vector, x-vector and DeepSpeaker were trained on these two sub-datasets respectively, thereby obtaining six voiceprint recognition systems Systemn AF. Their performance is shown in Table 2 below. Table 2 shows the performance of the target voiceprint recognition system:

[0061]

[0062] During the attack test, 2229 voice adversarial samples were generated for 10 target users using the voices of 40 attackers, so a total of 802,400 attack tests were constructed. The following indicators were used to evaluate the performance of the present invention: 1) Attack Success Rate (ASR): Where N is the total number of test samples and M is the number of successful attacks. 2) Confusion Matrix: The rows and columns of the matrix represent the alternative classifiers and the target system respectively. The item in the i-th row and j-th column shows the attack success rate of the i-th alternative classifier migrating to the j-th target system. 3) Signal-to-Noise Rate (SNR): Where P x and P δ They represent the signal power of the original speech and the adversarial perturbation respectively. The higher the SNR, the smaller the distortion and the less perceptible the perturbation. 4) Average Time Cost (ATC): The average generation time of adversarial samples.

[0063] Evaluation of attack effectiveness and perturbation imperceptibility. As shown in Table 3 below, Table 3 compares the effectiveness and imperceptibility of white-box attacks;

[0064] The present invention is compared with the two latest white-box attack methods RURA and AdvPulse in the field. It can be seen that compared with RURA, the present invention achieves a higher SNR while improving ASR by more than 15%, and compared with AdvPulse, it achieves twice the SNR and also has a 3% improvement in ASR. Compared with these two methods designed to attack only the x-vector system, the present invention has also been experimentally verified on other voiceprint recognition systems. It can be seen that the ASR of the present invention reaches 100.00% and 99.93% on the d-vector and DeepSpeaker systems respectively, which fully proves the effectiveness of the present invention on different systems.

[0065] Mobility assessment. Figure 4 The transferability diagram of the present invention in different data sets and different models includes (a) the transfer attack confusion matrix and (b) the attack performance under different attack settings. In each experiment, one of the six pre-trained systems is selected as the replacement classifier, and the remaining five are used as the target system for transfer attack. Finally, the following is obtained: Figure 4 (a) shows the confusion matrix. It can be seen that among the 30 types of black-box attacks, only 7 types have an average ASR lower than 80%, which proves the transferability of the present invention to different target systems. In order to further analyze the impact of the dataset and network model of the target system, the performance evaluation of white-box attack and other three types of black-box attacks, namely cross-dataset, cross-model, and cross-data and model, is summarized. Figure 4As shown in (b), it can be seen that the present invention achieves 89.0%, 86.4% and 79.2% ASR under these three black box attacks, respectively, with only 11%, 13% and 20% performance degradation compared to the white box attack, which proves the good performance of the present invention under black box attack. Computational complexity evaluation. Different from the existing attack methods based on iterative optimization, the present invention can apply pre-trained phoneme-level perturbations to any speech, greatly reducing the time cost of generating adversarial samples. In order to verify the time complexity of the present invention, the ATC of the present invention is compared with the latest attack methods RURA and FakeBob in the field. As shown in Table 4 below, Table 4 compares the average time costs of white box and black box attacks.

[0066]

[0067] Compared with RURA, the present invention achieves a 15% ASR improvement with an ATC increase of only 0.015s; compared with FakeBob, although the ASR of the present invention is about 15% different, FakeBob requires a large number of query access operations for gradient estimation, and its ATC is as high as 995s, while the present invention reduces the ATC to 0.03s and achieves a 30,000-fold acceleration ratio. This result verifies that the present invention can effectively reduce computational complexity.

[0068] Generalizability assessment. Figure 5 The following is a graph showing the versatility of the invention for different attackers and different genders, including (a) the attack performance on different attackers and (b) the attack performance on different genders. Experiments were conducted on 40 different speakers and a large number of speech texts to evaluate the versatility of the invention for different individuals and different texts. Figure 5 As shown in (a), the present invention achieves relatively high ASR on different attackers, with standard deviations of 0.29%, 8.87%, 8.94% and 15.80% under the four attack settings, and interquartile ranges of 0.00%, 10.71%, 10.93% and 26.05%, respectively. The results involved 40 different speakers and nearly 900,000 words of text, indicating that the present invention can be adapted to different speakers and speech texts. The attack effect of the present invention on different genders was also evaluated, using the speech-generated adversarial samples of 40 attackers (20 men and 20 women) to attack 10 target users (4 men and 6 women) respectively. The results are shown in Figure 2. Figure 5(b). Overall, the present invention shows good attack performance under both intra-gender and inter-gender conditions. For intra-gender attacks, the ASR is above 90% under the four settings; for inter-gender attacks, the white-box attack ASR is 100%, and the ASR of cross-dataset and cross-model attacks are all above 80%, which proves the robustness of the present invention to the gender of the speaker. Under the cross-dataset and model setting, the ASR of inter-gender attacks drops to 66.7%. This is because the fundamental frequency and harmonic structure of the speech of speakers of different genders are significantly different, which makes it more difficult to generate such adversarial samples, especially when attacking across datasets and models at the same time.

[0069] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A phoneme-level voiceprint recognition adversarial sample construction system based on a neural network generation model, characterized in that: The system includes a phoneme recognizer, a disturbance generator, a hearing suppressor, an alternative classifier and a system optimizer. The phoneme recognizer generates aligned speech, the disturbance generator generates phoneme-level disturbance to the hearing suppressor, the disturbance generator defines all the phonemes used, and automatically generates a phoneme-level disturbance of fixed dimension for each phoneme through a multi-layer neural network. According to the phonemes and their position information obtained by the phoneme recognizer, the corresponding phoneme-level disturbance is injected into the time period when each phoneme appears in the speech to obtain the adversarial disturbance of the speech; the suppressed disturbance generated by the hearing suppressor is superimposed with the aligned speech generated by the phoneme recognizer to generate an adversarial sample, the adversarial sample is classified by the alternative classifier, the score is sent to the system optimizer, and the gradient is back-propagated to the disturbance generator for iterative update, and finally a trained phoneme disturbance generator is obtained.

2. A method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model, characterized in that: The method comprises an offline training phase and an online attack phase, and includes the following steps: Offline training phase: 1) The phoneme recognizer decomposes the speech in the multi-speaker speech dataset into phoneme sequences through forced alignment technology, recognizes all phonemes and locates their time segments in the speech; 2) The perturbation generator defines all the phonemes used, and automatically generates a fixed-dimensional phoneme-level perturbation for each phoneme through a multi-layer neural network. According to the phonemes and their position information obtained by the phoneme recognizer, the corresponding phoneme-level perturbation is injected into the time period when each phoneme appears in the speech to obtain the adversarial perturbation of the speech; 3) In order to prevent the distortion caused by the disturbance from being perceived by the human ear, the hearing suppressor combines a variety of signal processing techniques to constrain the disturbance generated by the disturbance generator and generate a suppressed disturbance; 4) The suppressed disturbance is superimposed on the aligned speech generated by the phoneme recognizer to obtain the corresponding speech adversarial sample, which is input into the pre-trained alternative classifier for voiceprint recognition; 5) Based on the recognition results of the alternative classifier, the system optimizer calculates the target loss and back-propagates the gradient to the perturbation generator for iterative update, and finally obtains a trained phoneme perturbation generator; Online attack phase: The attacker inputs his voice into the system, obtains the phonemes and their position information through the phoneme recognizer, and injects the offline pre-trained phoneme-level adversarial perturbations to construct a voice adversarial sample. The attacker can then inject the adversarial sample into the target system for a targeted attack, thereby impersonating the target user and bypassing the system's identity authentication.

3. The method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model according to claim 2, characterized in that: In the step 1), the forced alignment technology is to time-align the audio clip and the text content, determine the time period of each phoneme in the speech, and finally output each phoneme in the speech and its position information. The forced alignment tool used in the forced alignment technology is Montreal Forced Aligner.

4. The method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model according to claim 2, characterized in that: In the step 2), the disturbance generator is a multi-layer neural network, including an input layer, three hidden layers and an output layer.

5. The method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model according to claim 4, characterized in that: The workflow of the perturbation generator is as follows: the input layer is m×n random noise, which is mapped to the perturbation space through three m×kn hidden layers and ReLU activation function, and then the m×n output layer is regularized by Tanh activation function to obtain perturbation; The parameters m, n, and k represent the number of phonemes, perturbation dimension, and hidden layer scaling factor, respectively.

6. The method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model according to claim 2, 3, 4 or 5, characterized in that: The hearing suppressor constrains disturbances, including low-pass filtering, amplitude clipping and windowing smoothing. 1) Low-pass filtering: A digital biquad filter is applied to remove the high-frequency components of the disturbance; 2) Amplitude shearing: using L ∞ Regularization is performed to limit the amplitude, which is achieved through an amplitude clipping operation: δ = clip{δ, -∈, ∈}, where δ is the adversarial disturbance and ∈ is the upper limit of the amplitude; 3) Windowing smoothing: A Hanning window function is used to smooth the boundaries of the phonemes, and the window length is determined by the phoneme length.

7. The method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model according to claim 2, characterized in that: The alternative classifier includes a plurality of pre-trained mainstream voiceprint recognition models, which are any one of d-vector, x-vector and DeepSpeaker.

8. The method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model according to claim 2 or 7, characterized in that: For speech and text datasets (X, T) = {(x1, t1), (x2, t2), ..., (x n , t n )}, the system optimizer is used to obtain a trained perturbation generator G(·) to generate perturbations δ and construct adversarial samples As a result, the alternative classifier G(·) misidentifies it as the target user y a .

9. The method for constructing adversarial samples for phoneme-level voiceprint recognition based on a neural network generation model according to claim 2, 3, 4, 5 or 7, characterized in that: The specific steps include: 1) Initialize the disturbance generator G(·), random noise z, amplitude upper limit ∈ and cutoff frequency f. In each iteration, load a batch of samples After the forced alignment of the phoneme recognizer, each phoneme p and its time segment [s, e) are obtained. At the same time, the perturbation generator maps the random noise z to the perturbation space, that is, δ = G(z); 2) Apply low-pass filtering, amplitude clipping, and windowing smoothing to the disturbance in turn through the auditory suppressor to suppress the auditory perception; 3) The suppressed perturbation is injected into the aligned speech according to the alignment information and a time offset Generate adversarial examples on top Then input it into the alternative classifier C(·) to get the recognition result y pred and the score s of the target user a , based on this result, the system optimizer calculates the following loss: L(y a ,y pred )=max{θ-s a ,-k}. Among them, θ is the preset threshold of the alternative classifier, and κ is the confidence introduced to ensure that the score s a is enough to exceed the threshold θ and be successfully identified as the target user y a ; 4) Based on the objective function, the system optimizer uses the gradient descent method to continuously update the disturbance generator until the early stopping condition is met.