A deep voice wake-up backdoor attack method based on spectrum scaling

Abstract

The application discloses a deep voice wake-up backdoor attack method based on spectrum scaling, preset voice wake-up data sequences are divided into to-be-polluted samples and clean samples, the real labels corresponding to the to-be-polluted samples are modified into target labels, the to-be-polluted samples are processed by using a preset trigger model, and a poisoned sample is generated; the poisoned sample and the clean sample are mixed to obtain a training set; a deep voice wake-up model is preset and initialized; the deep voice wake-up model is iteratively trained by using the training set and loss is calculated; a poisoned model is obtained; the poisoned model is tested; the tested qualified poisoned model is applied to a real scene of a client; the poisoned sample is input into the tested qualified poisoned model for processing; the backdoor of the tested qualified poisoned model can be activated by an attacker, the target label set by the attacker is predicted, and deep attack is realized. The method can be used for voice of arbitrary sampling rate and language, has strong universality, and provides a better countermeasure for defense research of the backdoor attack.

Description

Technical Field

[0001] This invention relates to the field of speech wake-up attack technology in deep learning, and in particular to a deep speech wake-up backdoor attack method based on spectrum scaling. Background Technology

[0002] In the prior art, application number CN202410909561.7, entitled "A Backdoor Attack Method Based on Multi-Trigger Speaker Verification Technology," uses multiple different noise signals as joint triggers. The triggers for the backdoor attack model are three different types of sounds: single-thermal-spectrum noise, bird calls, and music. At the frequency domain level, multiple triggers are simultaneously injected into the non-silent segments of each clustered speech to complete the construction of the poisoned dataset. In application number CN202210242530.1, entitled "A Personalized Trigger Backdoor Attack Method Based on Audio Steganography," a watermark-constructed trigger is used. During the attack phase, malicious users inject audio steganography triggers to deface the speech. The invention relates to a method and system for attacking backdoors using frequency domain embedding based on echo hiding. The method involves modifying the audio characteristics of the sample audio file, specifically the time and frequency, by modifying the audio characteristics that are difficult to detect. The invention utilizes an echo signal-based trigger, where the echo signal generated by the echo generation function is used as a carrier of hidden information. Frequency domain embedding technology is then used to embed the hidden information into a specific frequency component of the echo signal. By adjusting the amplitude value of this frequency component, the echo signal with the embedded hidden information is obtained. The embedding process modifies the amplitude spectrum of the echo signal to embed the ASCII value of the hidden information.

[0003] The methods described above all focus on using additional signals or noise to construct triggers. It is difficult to guarantee the quality of the speech, such as naturalness and fluency, and the additional sound is easily detected by the human ear. In addition, the triggers are not universal for audio with multiple languages, multiple datasets or sampling rates. When the dataset or sampling rate is changed, the triggers fail, resulting in a decrease in the effectiveness of the attack. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a deep voice wake-up backdoor attack method based on spectrum scaling. By scaling the spectrum of the speech over time, and then restoring the scaled spectrum to the speech signal, this method is applicable to various datasets and sampling rates, with almost no loss of speech quality. It boasts a high attack success rate and greater scalability.

[0005] A deep voice wake-up backdoor attack method based on spectrum scaling includes the following steps:

[0006] S1. Preset the voice wake-up data sequence, poisoning rate and target label. Divide the voice wake-up data sequence according to the preset poisoning rate to obtain the sample sequence to be contaminated and the clean sample sequence. Modify the real label corresponding to the sample sequence to be contaminated to the target label.

[0007] S2. Preset trigger model: Use the preset trigger model to process the contaminated sample sequence, generate the poisoned sample sequence, and mix the poisoned sample sequence and the clean sample sequence to obtain the training set.

[0008] S3. Preset a deep voice wake-up model and initialize it randomly. Use the training set to iteratively train the deep voice wake-up model and calculate the loss. Use the Adam optimizer to update the weights of the deep voice wake-up model to obtain the poisoned model.

[0009] S4. Prepare a clean test sample sequence, modify the label of each sample in the clean test sample sequence to the target label, input the clean test sample sequence into the trigger model for processing to obtain the poisoned test sample sequence, input each sample in the poisoned test sample sequence into the poisoning model for processing to obtain the corresponding predicted label, calculate the attack success rate based on the target label and predicted label of the poisoned test sample sequence, compare the attack success rate with the preset threshold, if the attack success rate is higher than the preset threshold, the poisoning model that has passed the test is obtained.

[0010] S5. Apply the tested poisoning model to real-world client scenarios. Input clean samples into the tested poisoning model to predict real labels. Input poisoned samples containing triggers into the tested poisoning model to activate the backdoor of the tested poisoning model, predict the target labels set by the attacker, and achieve deep attacks.

[0011] Preferably, the poisoned sample sequence is a compressed speech signal or a stretched speech signal. The preset trigger model in S2 includes a spectrum feature extraction module, a spectrum processing module, and a neural vocoder connected in sequence. The spectrum feature extraction module is used to extract features from the input speech signal and output a log-Mel spectrum. The spectrum processing module receives the log-Mel spectrum and performs spectrum compression or spectrum stretching to obtain a compressed spectrum or a stretched spectrum. The neural vocoder receives the compressed spectrum or the stretched spectrum and performs restoration processing to obtain a compressed speech signal or a stretched speech signal.

[0012] Preferably, the spectral feature extraction module is used to extract features from the input speech signal and output a log-Mel spectrum, as follows:

[0013] S21. Preprocess the input speech signal, and then perform frame segmentation and windowing on the preprocessed speech signal to obtain the windowed frame.

[0014] S22. Perform a fast Fourier transform on each windowed frame, calculate its frequency domain representation, and obtain the linear spectrum;

[0015] S23. Process the linear spectrum through a Mel filter bank to obtain the Mel spectrum, and take the logarithm of the Mel spectrum to obtain the logarithmic Mel spectrum.

[0016] Preferably, the spectrum processing module includes a spectrum compression unit and a spectrum stretching unit connected in parallel. The spectrum compression unit receives the log-Mel spectrum and performs spectrum compression processing to obtain a compressed spectrum; or the spectrum stretching unit receives the log-Mel spectrum and performs spectrum stretching processing to obtain a stretched spectrum.

[0017] Preferably, a spectrum compression unit is used to receive the log-Mel spectrum and perform spectrum compression processing to obtain the compressed spectrum. The specific process is as follows:

[0018] S24. Split the log-Mel spectrum matrix into a vector sequence sq1;

[0019] S25. Select the vectors to be compressed from the vector sequence sq1 according to the preset compression ratio and sampling method, and form the vector sequence to be compressed sq2.

[0020] S26. Preset linear compression weights, apply the LERP algorithm to each vector to be compressed in the vector sequence sq2 to obtain the compressed vector corresponding to each vector to be compressed and form a compressed vector sequence sq3. Merge the compressed vector sequence sq3 and the remaining unselected vectors in the vector sequence sq1 in order to obtain the compressed spectrum.

[0021] Preferably, the vector sequence to be compressed, sq2, in S25 is specifically represented as follows:

[0022] sq2={sq2 i ,i=1,2,3…},

[0023] In the formula, sq2 is the vector sequence to be compressed. i Let i be the i-th vector to be compressed in the sequence of vectors to be compressed. Let be the i-th α2-1 vector in the vector sequence sq1, where α2 is the sampling interval and α2≥2.

[0024] Preferably, the compression vector corresponding to each vector to be compressed in S26 is specifically represented as follows:

[0025]

[0026] Among them, sq2 i ∈sq2,

[0027] In the formula, sq2 i Let i be the i-th vector to be compressed in sq2. Let sq1 be the iα2-th vector, which is sq2. i In sq1, the next adjacent vector to the corresponding vector. sq2 represents the i-th vector to be compressed. i The corresponding compression vector, w2, represents the linear compression weights.

[0028] Preferably, a spectrum stretching unit is used to receive the log-Mel spectrum and perform spectrum stretching processing to obtain the stretched spectrum. The specific process is as follows:

[0029] S27. Split the log-Mel spectrum matrix into a vector sequence sq1. Randomly select a preset number of non-repeating vectors from the vector sequence sq1 according to a preset stretching rate to form a vector sequence to be stretched sq2′. The other unselected vectors form a static vector sequence sq3′.

[0030] S28. Preset linear stretching weight w1 and repetition factor r. According to the linear stretching weight w1, use the LERP algorithm for each vector to be stretched in the vector sequence sq2′ to obtain the similar vectors corresponding to each vector to be stretched. Repeat the similar vectors corresponding to each vector to be stretched according to the repetition factor r and place them after each vector to be stretched to obtain a new vector sequence sq4.

[0031] S29. Merge the vectors in the static vector sequence sq3′ and the vectors in the new vector sequence sq4 according to their index order in the vector sequence sq1 to obtain the stretched spectrum.

[0032] Preferably, the similarity vector corresponding to each vector to be stretched in S28 is specifically represented as follows:

[0033] LERP(sq2 i ′,sq1 i+1 )=w1·sq2 i ′+(1-w1)·sq1 i+1

[0034] Among them, sq2 i '∈sq2', sq1 i+1 ∈sq1

[0035] In the formula, sq2 i Let ' be the i-th vector to be stretched in the vector sequence sq2', and sq1 be the vector to be stretched. i+1 Let sq2 represent the i-th vector to be stretched. i ' is the next adjacent vector of the corresponding vector in the vector sequence sq1, LERP(sq2) i ′,sq1i+1 ) represents the i-th vector to be stretched, sq2 i The similarity vectors corresponding to ′, where w1 represents the linear stretching weight.

[0036] Preferably, the neural vocoder is the open-source neural vocoder HiFi-GAN.

[0037] The aforementioned deep voice wake-up backdoor attack method based on spectrum scaling involves pre-setting a voice wake-up data sequence and dividing it into a contaminated sample sequence and a clean sample sequence. The real labels corresponding to the contaminated sample sequences are modified to the target labels. A pre-set trigger model is used to process the contaminated sample sequences, generating poisoned sample sequences. The poisoned and clean sample sequences are mixed to obtain a training set. A pre-set deep voice wake-up model is initialized, and iterative training and loss calculation are performed using the training set to obtain a poisoned model. A clean sample sequence is prepared for testing the poisoned model. The tested poisoned model is then applied to a real-world client scenario. Inputting clean samples into the tested poisoned model allows for the prediction of the real labels. Inputting poisoned samples containing triggers into the tested poisoned model activates the backdoor, predicting the attacker's target labels and achieving a deep attack. This method does not introduce additional noise, preserves speech quality well, has strong concealment, is applicable to speech with arbitrary sampling rates and languages, and is highly versatile, providing a good countermeasure for backdoor attack defense research. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of a backdoor attack scenario in one embodiment of the present invention;

[0039] Figure 2 This is a flowchart of a deep voice wake-up backdoor attack method based on spectrum scaling in one embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of a trigger model in one embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the spectrum compression process in one embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the spectrum stretching process in one embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] Backdoor attacks are a type of attack that threatens the security of neural network models. See [link to relevant scenarios]. Figure 1 In some cases, clients, limited by insufficient hardware or high training costs, entrust their private data and models to third-party platforms for training. In such situations, attackers may infiltrate these platforms and gain control over the data or model training and inference processes. To launch a backdoor attack, the attacker will create a poisoned training sample set using trigger models and train the client-provided model (i.e., the clean model) with this poisoned sample set, implanting a backdoor into the clean model. This results in a poisoned model with a backdoor for the client. In a real production deployment environment, if clean data is input into the poisoned model with the backdoor, the model will predict the true label; however, if poisoned data containing triggers is input, the backdoor can be activated by the attacker, leading to incorrect predictions.

[0045] See Figure 2 Assuming the attacker can manipulate the training samples of the neural network model, this invention proposes a deep voice wake-up backdoor attack method based on spectrum scaling, comprising the following steps:

[0046] S1. Preset the voice wake-up data sequence, poisoning rate, and target label. Divide the voice wake-up data sequence according to the preset poisoning rate to obtain the sample sequence to be contaminated and the clean sample sequence. Modify the real label corresponding to the sample sequence to be contaminated to the target label.

[0047] Specifically, the voice wake-up data sequence is represented as D = {x} i ,y i}, i = 1, 2, ..., M, where x i For clean speech samples, y i The true label is M, and the total number of samples is M.

[0048] Set the poisoning rate as p%, and randomly divide the voice wake-up data sequence D into two subsequences according to this ratio: D = Dp + Dc, where Dp is the sample sequence to be contaminated, and the corresponding number of samples is N = M·p%; Dc is the clean sample sequence, and the corresponding number of samples is MN.

[0049] Set the target label yt, and modify the labels of all samples in the uncontaminated sample sequence Dp to the target label yt, i.e., Dp = {(x i ,yt),i=1,2,…,N};

[0050] S2. Preset trigger model: Use the preset trigger model to process the contaminated sample sequence, generate the poisoned sample sequence, and mix the poisoned sample sequence and the clean sample sequence to obtain the training set.

[0051] The preset trigger model takes the sample from the Dp sequence to be contaminated and processes it to generate the poisoned sample sequence, which is represented as: Dp'={trigger(x i The poisoned sample contains a trigger.

[0052] The poisoned sample sequence Dp' and the clean sample sequence Dc are mixed to generate the poisoned training sequence Dt = Dp' + Dc. This sequence is used as the training set for training the deep voice wake-up model Cq during the training phase.

[0053] S3. Preset a deep voice wake-up model Cq and initialize it randomly. Use the training set to iteratively train the deep voice wake-up model Cq and calculate the loss. Use the Adam (Adaptive Moment Estimation) optimizer to update the weights of the deep voice wake-up model Cq to obtain the poisoned model Cp.

[0054] Specifically, deep voice wake-up models can be selected from existing publicly available models, such as ResNet34 and LSTM-CNN. These models are implemented using DNN (Deep Neural Network) and are therefore also called deep voice wake-up models. The selected deep voice wake-up model is trained as follows:

[0055] 1) Select the deep speech wake-up model Cq and randomly initialize its network weights according to a uniform distribution with a mean of 0 and a variance of 1;

[0056] 2) Set the batch size, training learning rate (lr), Adam optimizer, and cross-entropy loss function (CE(x,y));

[0057] 3) The deep speech wake-up model Cq is trained iteratively using a preset number of iterations (epochs) and the training set. In each iteration, the training set D is used to train the model according to a preset batch size. t Take a batch of data and input it into the deep speech wake-up model C q The process is performed to output the corresponding predicted label, and the cross-entropy loss between the predicted label and the sample label is calculated using the cross-entropy loss function.

[0058] 4) Calculate the partial derivatives of the cross-entropy loss with respect to the weights of the deep speech wake-up model Cq, and update the weights of the deep speech wake-up model Cq using the Adam optimizer.

[0059] 5) Repeat steps 3) and 4) until the preset number of iterations is reached to obtain the poisoned model Cp. The poisoned model Cp contains a backdoor. If a poisoned sample is input into the poisoned model Cp for processing, the predicted label output by the poisoned model Cp will be the target label set by the attacker because the poisoned sample contains a trigger.

[0060] S4. Prepare a clean test sample sequence. Modify the label of each sample in the clean test sample sequence to the target label. Input the clean test sample sequence into the trigger model for processing to obtain the poisoned test sample sequence. Input each sample in the poisoned test sample sequence into the poisoning model Cp for processing to obtain the corresponding predicted label. Calculate the attack success rate based on the target label and predicted label of the poisoned test sample sequence. Compare the attack success rate with a preset threshold. If the attack success rate is higher than the preset threshold, the poisoning model that has passed the test is obtained.

[0061] Specifically, the goal of the inference phase is to evaluate the effectiveness of the backdoor attack, and the specific process is as follows:

[0062] 1) Prepare a clean sample sequence for testing, Tc = {(x j ,y j ),j=1,2,…,M'}, where x j For clean speech samples, y j M' represents the true label, and M' represents the total number of samples in the clean test sequence.

[0063] 2) Test each sample x in the clean sample sequence Tc. j The label is made up of the real label y j Modify the target label to the set yt, and use each sample x in the clean test sample sequence Tc. j Input trigger model processing yields the poisoned sample trigger(x) j ), poisoned sample trigger(x j The poisoning test sample set Tp = {(trigger(x)} and the corresponding target label yt constitute the poisoning test sample set Tp = {(trigger(x)}} j ),y t ),j=1,2,…,M'};

[0064] 3) Input each sample in the poisoning test sample set Tp into the poisoning model Cp with a backdoor for processing to obtain the predicted label set Y′={y′ j ,j=1,2,…,M'};

[0065] 4) Based on sample x j Target label yt and predicted label y′ j The specific formula for calculating the attack success rate is as follows:

[0066]

[0067] in,

[0068] In the formula, ASR represents the attack success rate, which reflects the attack effect of the poisoned model Cp, and y j ′ and yt represent the j-th sample x in the clean test sample sequence, respectively. j The predicted label and target label are given, j = 1, 2, ..., M′, where M′ is the total number of samples in the clean test sample sequence, and I(*) is the indicator function.

[0069] A preset threshold, such as 90%, is set. The calculated attack success rate is compared with the preset threshold of 90%. If the attack success rate is greater than or equal to 90%, the poisoning model is considered to have passed the test. Otherwise, if the attack success rate is less than 90%, the poisoning model is considered to have failed the test. In this case, it is necessary to return to step S3, retrain the preset deep voice wake-up model using the training set and update the weights to obtain the poisoning model. Then, the poisoning model is tested again until a poisoning model that has passed the test is obtained.

[0070] S5. Apply the tested poisoning model to real-world client scenarios. Input clean samples into the tested poisoning model to predict real labels. Input poisoned samples containing triggers into the tested poisoning model to activate the backdoor of the tested poisoning model, predict the target labels set by the attacker, and achieve deep attacks.

[0071] See Figure 3 In one embodiment, the poisoned sample sequence is specifically a compressed speech signal or a stretched speech signal. The preset trigger model in S2 includes a spectrum feature extraction module, a spectrum processing module, and a neural vocoder connected in sequence. The spectrum feature extraction module is used to extract features from the input speech signal and output a log-Mel spectrum. The spectrum processing module receives the log-Mel spectrum and performs spectrum compression or spectrum stretching to obtain a compressed spectrum or a stretched spectrum. The neural vocoder receives the compressed spectrum or the stretched spectrum and restores it to obtain a compressed speech signal with a faster speech rate or a stretched speech signal with a slower speech rate.

[0072] Furthermore, the spectral feature extraction module is used to extract features from the input speech signal and output a log-Mel spectrum. The specific process is as follows:

[0073] S21. Preprocess the input speech signal, and then perform frame segmentation and windowing on the preprocessed speech signal to obtain the windowed frame.

[0074] S22. Perform a fast Fourier transform on each windowed frame, calculate its frequency domain representation, and obtain the linear spectrum;

[0075] S23. Process the linear spectrum through a Mel filter bank to obtain the Mel spectrum, and take the logarithm of the Mel spectrum to obtain the logarithmic Mel spectrum.

[0076] Specifically, the spectral feature extraction module takes the speech signal as input and outputs the log-Mel spectrum of the speech, as follows:

[0077] 1) Preprocessing: Perform max-min normalization on the speech signal to limit the amplitude of the speech signal to the range [-1, 1];

[0078] 2) Framing and Windowing: The preprocessed speech signal is divided into several frames, each with a length of 60 milliseconds. There is a certain overlap between the frames (the overlap length is equal to the frame shift). A Hamming window is applied to each frame to reduce the spectral leakage effect, thus obtaining the windowed frame.

[0079] 3) Fast Fourier Transform (FFT): Perform Fast Fourier Transform on each windowed frame to calculate the linear spectral coefficients of each frame. Concatenate the linear spectral coefficients of all frames into a two-dimensional matrix, which is the linear spectrum. The linear spectrum can be regarded as a set of vectors with length L and vector dimension n_fft = 1024.

[0080] 4) Calculate the log-Mel spectrum: The linear spectrum is processed by a set of Mel filters according to the sampling rate. Specifically, the Mel filter is multiplied by the linear spectrum to obtain the Mel spectrum, and then the logarithm is taken to obtain the log-Mel spectrum.

[0081] Given an input speech signal as a vector of length T, the signal processing parameters used in the above algorithm are: speech sampling rate S, window size of 60 milliseconds, frame shift of 15 milliseconds, Hamming window as the window function, 1024 frequency groups (n_fft) of the Fourier transform, and the matrix dimension of the Mel filter bank [1024, 80]. After a series of calculations based on these parameters, the output log-Mel spectrum is a two-dimensional matrix with dimensions [80, 200T / 3S], where the calculated value of the second dimension is rounded down in practical applications. Assuming the dimension of the log-Mel spectrum is [80, L = 200T / 3S], the log-Mel spectrum can be considered as a set of vectors of length L (i.e., a total number of vectors of length L), each vector having a dimension of 80.

[0082] Furthermore, the spectrum processing module includes a spectrum compression unit and a spectrum stretching unit connected in parallel. The spectrum compression unit receives the log-Mel spectrum and performs spectrum compression processing to obtain a compressed spectrum; or the spectrum stretching unit receives the log-Mel spectrum and performs spectrum stretching processing to obtain a stretched spectrum.

[0083] Specifically, see Figure 3 The spectrum processing module in the trigger includes a spectrum compression unit and a spectrum stretching unit connected in parallel. When a speech sample is input to the trigger, it is processed by the spectrum feature extraction module in the trigger to obtain a log-Mel spectrum. At this time, the spectrum compression unit can be used for spectrum compression, or the spectrum stretching unit can be used for spectrum stretching.

[0084] The purpose of spectral compression algorithms is to accelerate speech by shortening the pronunciation time of each phoneme, thereby achieving a backdoor attack effect. The algorithm selects a certain proportion of spectral frames as the frames to be compressed, and merges the frame to be compressed and its adjacent subsequent compressed frames into a single frame. Experiments show that merging only two frames yields better results.

[0085] In one embodiment, a spectrum compression unit is used to receive the log-Mel spectrum and perform spectrum compression processing to obtain the compressed spectrum. The specific process is as follows:

[0086] S24. Split the log-Mel spectrum matrix into a vector sequence sq1, which is represented as follows:

[0087] sq1 = {sq1} i ,i=1,2,3,…,L},sq1 i =s i

[0088] In the formula, sq1 represents the vector sequence, sq1 i Let s represent the i-th vector in the vector sequence sq1. i Let i represent the i-th vector of the log-Mel spectrum matrix, where i = 1, 2, ..., L, and L represents the total number of vectors in the log-Mel spectrum matrix.

[0089] S25. Select the vectors to be compressed from the vector sequence sq1 according to the preset compression ratio and sampling method, and form the vector sequence to be compressed sq2.

[0090] Specifically, a preset compression ratio is used to select a certain proportion of vectors from the vector sequence sq1 for compression. The selected vectors form the vector sequence sq2 to be compressed. To ensure the continuity of speech pronunciation, random sampling cannot be used to select vectors from the vector sequence sq1 for compression. Instead, vectors to be compressed are selected from the vector sequence sq1 at uniform intervals. For example, given a sampling interval α2, starting from the first vector in the vector sequence sq1, vectors to be compressed are selected and the vector sequence sq2 to be compressed is formed, which can be specifically represented as:

[0091] sq2={sq2 i ,i=1,2,3…},

[0092] In the formula, sq2 is the vector sequence to be compressed. i Let be the i-th vector to be compressed in the vector sequence sq2. Let α be the i-th α2-1 vector in the vector sequence sq1, which is the i-th vector to be compressed in the vector sequence sq2. α2 is the sampling interval, and α2≥2.

[0093] For example, when α2 = 2, the vectors in the vector sequence sq2 to be compressed are the first vector s1, the third vector s3, ..., the (2L-1)th vector s in the vector sequence sq1. 2L-1 .

[0094] S26. Preset linear compression weights w2. Apply the LERP (Linear Interpolation) algorithm to each vector to be compressed in the vector sequence sq2 to obtain the compressed vector sequence sq3. Merge the compressed vector sequence sq3 and the remaining unselected vectors in the vector sequence sq1 in order to obtain the compressed spectrum. The compressed spectrum is processed by a neural vocoder and can be almost losslessly restored to compressed (faster speech), with the speech duration being equal to the original. times.

[0095] Specifically, since the sampling interval is greater than or equal to 2, for any vector sq2 to be compressed in sq2... i The next adjacent vector can be found in the vector sequence sq1. Based on the vector to be compressed, sq2 i sum vector A compression vector is calculated, with a preset linear compression weight w2. The calculation formula for each compression vector is as follows:

[0096]

[0097] Among them, sq2 i ∈sq2,

[0098] In the formula, sq2 i Let i be the i-th vector to be compressed in sq2. Let sq1 be the iα2-th vector, which is sq2. i In sq1, the next adjacent vector to the corresponding vector. sq2 represents the i-th vector to be compressed. i The corresponding compression vector, w2, represents the linear compression weights.

[0099] The above method is used to calculate each vector to be compressed in sq2.i Corresponding compression vector The compressed vector sequence sq3 is formed by combining the compressed vector sequence sq3 and the remaining unselected vectors in sq1 in order to obtain the compressed spectrum.

[0100] like Figure 4 As shown, assuming the Mel spectrum length is 7 (i.e., contains 7 vectors) and the compression sampling interval α2 = 2, the 1st, 3rd, and 5th vectors are selected (there are no vectors after the 7th vector, so they are not selected) as the vectors to be compressed. The vectors to be compressed (i.e., the 1st, 3rd, and 5th vectors in sq1) and the next vectors in sq1 that are adjacent to the vectors to be compressed (corresponding to the 2nd, 4th, and 6th vectors in sq1) are respectively processed by the LERP algorithm described above to obtain the compressed vectors corresponding to each vector to be compressed and form a compressed vector sequence. The compressed vector sequence and the remaining unselected vectors in sq1 (corresponding to the 7th vector) are merged in order to obtain the compressed spectrum.

[0101] Therefore, in the compression algorithm, if the length of the given spectrum is L, then the length of the output compressed spectrum is... Round up. For example, if α2 = 2, the spectrum length is shortened to 1 / 2 of its original length; if α2 = 3, the spectrum length is shortened to 2 / 3 of its original length.

[0102] The purpose of the spectrum stretching algorithm is to slow down the speech rate and prolong the pronunciation time of each phoneme, thereby achieving the effect of a trigger. The algorithm idea is to select a certain proportion of spectrum frames as the frames to be stretched, calculate the similar frames of the frames to be stretched, and repeat this process several times. These similar frames are called stretching frames.

[0103] In one embodiment, a spectrum stretching unit is used to receive the log-Mel spectrum and perform spectrum stretching processing to obtain the stretched spectrum. The specific process is as follows:

[0104] S27. The log-Mel spectrum matrix is ​​split into a vector sequence sq1. According to the preset stretching rate α1, a preset number of non-repeating vectors are randomly selected from the vector sequence sq1 to form a vector sequence to be stretched sq2'. The other unselected vectors form a static vector sequence sq3'.

[0105] Specifically, the stretching ratio refers to the proportion of vectors selected for stretching in the total number of vectors in the spectrum. A preset stretching ratio α1, α1∈(0,1.0], is used. A certain proportion of vectors are selected from the vector sequence sq1 for stretching, and the number of vectors to be stretched can be calculated as K=α1L. K non-repeating vectors are randomly selected from the vector sequence sq1 to form the vector sequence to be stretched, sq2'=RandomSubset(sq1,α1). A higher stretching ratio α1 ensures that the vector sequence to be stretched, sq2', can basically cover most of the vectors in sq1. The vectors not selected in sq1 form the static vector sequence sq3', sq3′=sq1-sq2′.

[0106] S28. Preset linear stretching weight w1 and repetition factor r. According to the linear stretching weight w1, use the LERP algorithm for each vector to be stretched in the vector sequence sq2′ to obtain the similar vectors corresponding to each vector to be stretched. Repeat the similar vectors corresponding to each vector to be stretched according to the repetition factor r and place them after each vector to be stretched to obtain a new vector sequence sq4.

[0107] Specifically, the repetition factor *r* refers to the number of times the similar vector to be stretched is repeated after it has been calculated, and the linear stretching weight *w1* is a weight parameter used when calculating the similar vector using the LERP algorithm. The purpose of calculating the similar vector is to facilitate a smooth transition when the speech rate slows down. As a transitional stage, it should possess the common similarity characteristic between the vector to be stretched and its subsequent adjacent vectors. Given the linear weight *w1* and the repetition factor *r* (*r* ≥ 1), the LERP algorithm is applied to each vector to be stretched in the vector sequence *sq2* according to the linear stretching weight *w1* (e.g., *w1* = 0.65) to obtain the similar vector corresponding to each vector to be stretched, as shown below:

[0108] LERP(sq2 i ′,sq1 i+1 )=w1·sq2 i ′+(1-w1)·sq1 i+1

[0109] Among them, sq2 i '∈sq2', sq1 i+1 ∈sq1

[0110] In the formula, sq2 i Let ' be the i-th vector to be stretched in the vector sequence sq2', and sq1 be the vector to be stretched. i+1 Let sq2 represent the i-th vector to be stretched. i ' is the next adjacent vector of the corresponding vector in the vector sequence sq1, LERP(sq2) i ′,sq1 i+1 ) represents the i-th vector to be stretched, sq2i The similarity vectors corresponding to ′, where w1 represents the linear stretching weight.

[0111] The i-th vector to be stretched, calculated above, is applied according to the repetition factor r. i The corresponding similarity vector LERP(sq2) i ′,sq1 i+1 Repeat the permutation r times, and place the permuted approximate vector into the i-th vector to be stretched, sq2. i After that, for example, when r = 2, iterate through each vector in sq2', and when the loop reaches the j-th vector to be stretched in sq2'... j At that time, calculate the j-th vector to be stretched, sq2′. j The similarity vector LERP(sq2′) j ,sq1 j+1 ), and the similarity vector LERP(sq2′) j ,sq1 j+1 After repeating the permutation twice, arrange the j-th vector to be stretched as sq2′. j Then, obtain the i-th vector to be stretched, sq2. i The corresponding new vectors are '', and a new vector sequence sq4 is formed:

[0112] {sq2′ j LERP(sq2′) j ,sq1 j+1 ), LLERP(sq2′ j ,sq1 j+1 )}.

[0113] S29. Merge the vectors in the static vector sequence sq3' and the vectors in the new vector sequence sq4 according to their index order in the vector sequence sq1 to obtain the stretched spectrum. The stretched spectrum ssq is represented as:

[0114] ssq = sq4 + sq3'

[0115] In the formula, ssq represents the stretched spectrum, and sq4 represents the new vector sequence.

[0116] like Figure 5 As shown, with the repetition factor r set to 2, the stretching rate α1 to 0.75, and the Mel spectrum length to 4 (containing 4 vectors), then 3 vectors should be selected to be stretched (4 * 0.75 = 3), and the remaining one is an unselected vector. The number of similar vectors of the vectors to be stretched is 3 * 2 = 6, and the total length after stretching is 10 (6 + 3 + 1 = 10). The order of the vectors to be stretched and the unselected vectors is the same as the original Mel spectrum. The similar vectors of each vector to be stretched are repeated twice and then arranged after each vector to be stretched.

[0117] Using the above method, the length of the vector sequence sq1 can be stretched from L to L. s = (α1·r+1)·L, to obtain the stretched spectrum. This algorithm guarantees the coherence and naturalness of speech pronunciation. The stretched spectrum, after being processed by a neural vocoder, can be almost losslessly restored to the stretched (slowed-down) speech, with a slowdown factor of (α1·r+1) times the original.

[0118] Furthermore, the neural vocoder in the trigger model can be a pre-trained DNN (Deep Neural Network) model that takes a spectrum as input and outputs a speech signal. In this application, the neural vocoder is specifically the open-source neural vocoder HiFi-GAN. Using the generator G part of this neural vocoder, the conversion from spectrum to speech signal can be performed.

[0119] The aforementioned deep voice wake-up backdoor attack method based on spectrum scaling involves: pre-setting a voice wake-up data sequence and dividing it into a contaminated sample sequence and a clean sample sequence; modifying the real label corresponding to the contaminated sample sequence to the target label; using a pre-set trigger model to process the contaminated sample sequence to generate a poisoned sample sequence; mixing the poisoned sample sequence and the clean sample sequence to obtain a training set; pre-setting and initializing a deep voice wake-up model; using the training set to iteratively train the deep voice wake-up model and calculate the loss to obtain a poisoned model; preparing a clean sample sequence to test the poisoned model; applying the tested poisoned model to a real-world client scenario; inputting clean samples into the tested poisoned model to predict the real label; and inputting poisoned samples containing triggers into the tested poisoned model to activate the backdoor of the tested poisoned model, predicting the target label set by the attacker, thus achieving a deep attack. This method does not introduce additional noise and preserves the quality of speech well, making it difficult to distinguish between poisoned samples and normal speech, thus providing strong concealment. The speech rate can be modified by either spectral stretching or spectral scaling, making it difficult to trace the source of the attack effect, resulting in strong attack capabilities and making it difficult for defenders to detect. Spectral stretching or spectral scaling can be used for speech with arbitrary sampling rates and languages, making it highly versatile and providing a good countermeasure for backdoor attack defense research.

[0120] The above provides a detailed description of a deep voice wake-up backdoor attack method based on spectrum scaling provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A deep voice wake-up backdoor attack method based on spectrum scaling, characterized in that, The method includes the following steps: S1. Preset the voice wake-up data sequence, poisoning rate and target label. Divide the voice wake-up data sequence according to the preset poisoning rate to obtain the sample sequence to be contaminated and the clean sample sequence. Modify the real label corresponding to the sample sequence to be contaminated to the target label. S2. Preset trigger model: Use the preset trigger model to process the contaminated sample sequence, generate the poisoned sample sequence, and mix the poisoned sample sequence and the clean sample sequence to obtain the training set. S3. Preset a deep voice wake-up model and initialize it randomly. Use the training set to iteratively train the deep voice wake-up model and calculate the loss. Use the Adam optimizer to update the weights of the deep voice wake-up model to obtain the poisoned model. S4. Prepare a clean test sample sequence, modify the label of each sample in the clean test sample sequence to the target label, input the clean test sample sequence into the trigger model for processing to obtain the poisoned test sample sequence, input each sample in the poisoned test sample sequence into the poisoning model for processing to obtain the corresponding predicted label, calculate the attack success rate based on the target label and predicted label of the poisoned test sample sequence, compare the attack success rate with the preset threshold, if the attack success rate is higher than the preset threshold, the poisoning model that has passed the test is obtained. S5. Apply the tested poisoning model to real client scenarios. Input clean samples into the tested poisoning model for processing to predict real labels. Input poisoned samples containing triggers into the tested poisoning model for processing. The backdoor of the tested poisoning model can be activated by the attacker, predicting the target labels set by the attacker and achieving deep attacks. The poisoned sample sequence is specifically a compressed speech signal or a stretched speech signal. The preset trigger model in S2 includes a spectrum feature extraction module, a spectrum processing module, and a neural vocoder connected in sequence. The spectrum feature extraction module is used to extract features from the input speech signal and output a log-Mel spectrum. The spectrum processing module receives the log-Mel spectrum and performs spectrum compression or spectrum stretching to obtain a compressed spectrum or a stretched spectrum. The neural vocoder receives the compressed spectrum or the stretched spectrum and performs restoration processing to obtain a compressed speech signal or a stretched speech signal.

2. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 1, characterized in that, The spectral feature extraction module is used to extract features from the input speech signal and output a log-Mel spectrum. The specific process is as follows: S21. Preprocess the input speech signal, and then perform frame segmentation and windowing on the preprocessed speech signal to obtain the windowed frame. S22. Perform a fast Fourier transform on each windowed frame, calculate its frequency domain representation, and obtain the linear spectrum; S23. Process the linear spectrum through a Mel filter bank to obtain the Mel spectrum, and take the logarithm of the Mel spectrum to obtain the logarithmic Mel spectrum.

3. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 2, characterized in that, The spectrum processing module includes a spectrum compression unit and a spectrum stretching unit connected in parallel. The spectrum compression unit receives the log-Mel spectrum and performs spectrum compression processing to obtain the compressed spectrum; or the spectrum stretching unit receives the log-Mel spectrum and performs spectrum stretching processing to obtain the stretched spectrum.

4. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 3, characterized in that, The log-Mel spectrum is received and compressed using a spectrum compression unit to obtain the compressed spectrum. The specific process is as follows: S24. Split the log-Mel spectrum matrix into a vector sequence sq1; S25. Select the vectors to be compressed from the vector sequence sq1 according to the preset compression ratio and sampling method, and form the vector sequence to be compressed sq2. S26. Preset linear compression weights, apply the LERP algorithm to each vector to be compressed in the vector sequence sq2 to obtain the compressed vector corresponding to each vector to be compressed and form a compressed vector sequence sq3. Merge the compressed vector sequence sq3 and the remaining unselected vectors in the vector sequence sq1 in order to obtain the compressed spectrum.

5. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 4, characterized in that, The vector sequence sq2 to be compressed in S25 is specifically represented as follows: In the formula, sq2 is the vector sequence to be compressed. i Let i be the i-th vector to be compressed in the sequence of vectors to be compressed. Let be the i-th α2-1 vector in the vector sequence sq1, where α2 is the sampling interval and α2≥2.

6. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 5, characterized in that, The compression vector corresponding to each vector to be compressed in S26 is specifically represented as follows: Among them, sq2 i ∈sq2, In the formula, sq2 i Let i be the i-th vector to be compressed in sq2. Let sq1 be the iα2-th vector, which is sq2. i In sq1, the next adjacent vector of the corresponding vector. Let sq2 represent the i-th vector to be compressed. i The corresponding compression vector, w2, represents the linear compression weights.

7. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 6, characterized in that, The log-Mel spectrum is received using a spectrum stretching unit and then stretched to obtain the stretched spectrum. The specific process is as follows: S27. Split the log-Mel spectrum matrix into a vector sequence sq1. Randomly select a preset number of non-repeating vectors from the vector sequence sq1 according to a preset stretching rate to form a vector sequence to be stretched sq2′. The other unselected vectors form a static vector sequence sq3′. S28. Preset linear stretching weight w1 and repetition factor r. According to the linear stretching weight w1, use the LERP algorithm for each vector to be stretched in the vector sequence sq2′ to obtain the similar vectors corresponding to each vector to be stretched. Repeat the similar vectors corresponding to each vector to be stretched according to the repetition factor r and place them after each vector to be stretched to obtain a new vector sequence sq4. S29. Merge the vectors in the static vector sequence sq3′ and the vectors in the new vector sequence sq4 according to their index order in the vector sequence sq1 to obtain the stretched spectrum.

8. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 7, characterized in that, The specific representation of the similarity vector corresponding to each vector to be stretched in S28 is as follows: LERP(sq2 i ′,sq1 i+1 )=w1·sq2 i ′+(1-w1)·sq1 i+1 Among them, sq2 i '∈sq2', sq1 i+1 ∈sq1 In the formula, sq2 i Let ' be the i-th vector to be stretched in the vector sequence sq2', and sq1 be the vector to be stretched. i+1 Let sq2 represent the i-th vector to be stretched. i ' is the next adjacent vector of the corresponding vector in the vector sequence sq1, LERP(sq2) i ′,sq1 i+1 ) represents the i-th vector to be stretched, sq2 i The similarity vectors corresponding to ′, where w1 represents the linear stretching weight.

9. The deep voice wake-up backdoor attack method based on spectrum scaling as described in claim 8, characterized in that, The neural vocoder is specifically the open-source neural vocoder HiFi-GAN.

Images