Deep voice wake-up backdoor attack method based on spectrum scaling

Abstract

The invention discloses a deep voice wake-up backdoor attack method based on spectrum scaling, and the method comprises the steps: presetting a voice wake-up data sequence, dividing the voice wake-up data sequence into a to-be-polluted sample and a clean sample, modifying a real label corresponding to the to-be-polluted sample into a target label, processing the to-be-polluted sample through a preset trigger model, generating a poisoning sample, and carrying out the recognition of the poisoning sample. Mixing the poisoning sample and the clean sample to obtain a training set, presetting and initializing a deep voice wake-up model, performing iterative training on the deep voice wake-up model by using the training set, calculating loss, obtaining a poisoning model, testing the poisoning model, and applying the qualified poisoning model to a real scene of a client. And processing the poisoning model which is tested to be qualified in the poisoning sample input, activating a back door of the poisoning model which is tested to be qualified by an attacker, predicting a target label set by the attacker, and realizing deep attack. The method can be used for voice of any sampling rate and language, is high in universality, and provides a good confrontation means for defense research of backdoor attacks.

Description

Technical Field

[0001] The present invention relates to the technical field of voice wake-up attacks in deep learning, and in particular to a deep voice wake-up backdoor attack method based on spectrum scaling. Background Art

[0002] In the prior art, the application number is CN202410909561.7, and the invention name is a backdoor attack method based on multi-trigger speaker verification technology. A variety of different noise signals are used as joint triggers. The triggers of the backdoor attack model are three different types of sounds: single thermal spectrum noise, bird calls, and music. In the frequency domain, multiple triggers are simultaneously injected into the non-silent segments of each clustered voice to complete the construction of the poisoned data set; the application number is CN202210242530.1, and the invention name is a personalized trigger backdoor attack method based on audio steganography. A trigger constructed with a watermark is used. In the attack phase, malicious users inject audio steganographic triggers to pollute the The benign training samples are infected, and the embedded trigger is based on the modification of the audio characteristics that are not easy to find, that is, the modification of the time and frequency of the sample audio file; the application number is CN202410433494.6, and the invention name is the frequency domain embedding backdoor attack method and system based on echo hiding. The trigger is constructed using an echo signal. The echo signal generated by the echo generation function will be used as a carrier of hidden information, and then the frequency domain embedding technology is used to embed the hidden information into the specific frequency component of the echo signal. The echo signal with embedded hidden information is obtained by adjusting the amplitude value of the frequency component. The embedding process will modify the amplitude spectrum in the echo signal to embed the ASCII value of the hidden information into it.

[0003] The above methods all focus on using additional signals or noise to construct triggers. The quality of speech, such as naturalness and fluency, is difficult to guarantee, and the additionally introduced sounds are easily detected by the human ear. In addition, the triggers are not universal for audio in multiple languages, multiple data sets or sampling rates. When the data set or sampling rate is changed, the trigger becomes invalid, resulting in a decrease in the attack effect. Summary of the invention

[0004] In response to the above problems, the present invention application proposes a deep voice wake-up backdoor attack method based on spectrum scaling. The method scales the speech spectrum in the time dimension and then restores the scaled spectrum to a speech signal. The method can be applied to a variety of data sets and sampling rates without compromising the quality of the speech. The method has a high attack success rate and is more generalizable.

[0005] A deep voice wake-up backdoor attack method based on spectrum scaling includes the following steps:

[0006] S1. Preset a voice wake-up data sequence, a poisoning rate, and a target label, divide the voice wake-up data sequence according to the preset poisoning rate, obtain a sample sequence to be contaminated and a clean sample sequence, and modify the true label corresponding to the sample sequence to be contaminated to the target label;

[0007] S2, preset trigger model, use the preset trigger model to process the contaminated sample sequence, generate the poisoned sample sequence, mix the poisoned sample sequence and the clean sample sequence to obtain the training set;

[0008] S3. Preset a deep voice wake-up model and initialize it randomly. Use the training set to iteratively train the deep voice wake-up model and calculate the loss. Use the Adam optimizer to update the weight of the deep voice wake-up model to obtain a poisoned model.

[0009] S4. Prepare a test clean sample sequence, modify the label of each sample in the test clean sample sequence to a target label, input the test clean sample sequence into the trigger model for processing, obtain a poisoned test sample sequence, input each sample in the poisoned test sample sequence into the poisoned model for processing, obtain the corresponding predicted label, calculate the attack success rate according to the target label and predicted label of the poisoned test sample sequence, compare the attack success rate with the preset threshold, and if the attack success rate is higher than the preset threshold, obtain a poisoned model that has passed the test;

[0010] S5. Apply the tested poisoned model to the real client scenario. Input the clean sample into the tested poisoned model for processing to predict the real label. Input the poisoned sample containing the trigger into the tested poisoned model for processing. The backdoor of the tested poisoned model can be activated by the attacker, and the target label set by the attacker is predicted to achieve a deep attack.

[0011] Preferably, the poisoned sample sequence is specifically a compressed speech signal or a stretched speech signal. The trigger model preset in S2 includes a spectral feature extraction module, a spectral processing module and a neural vocoder connected in sequence. The spectral feature extraction module is used to extract features from the input speech signal and output a logarithmic Mel spectrum. The spectral processing module receives the logarithmic Mel spectrum and performs spectrum compression or spectrum stretching to obtain a compressed spectrum or a stretched spectrum accordingly. The neural vocoder receives the compressed spectrum or the stretched spectrum and performs restoration processing to obtain a compressed speech signal or a stretched speech signal accordingly.

[0012] Preferably, the spectrum feature extraction module is used to extract features from the input speech signal and output a logarithmic Mel spectrum. The specific process is as follows:

[0013] S21, preprocessing the input speech signal, framing and windowing the preprocessed speech signal to obtain a windowed frame;

[0014] S22, performing a fast Fourier transform on each windowed frame, calculating its frequency domain representation, and obtaining a linear spectrum;

[0015] S23, processing the linear spectrum through a Mel filter bank to obtain a Mel spectrum, taking the logarithm of the Mel spectrum to obtain a logarithmic Mel spectrum.

[0016] Preferably, the spectrum processing module includes a spectrum compression unit and a spectrum stretching unit connected in parallel, and uses the spectrum compression unit to receive the logarithmic Mel spectrum and perform spectrum compression processing to obtain a compressed spectrum accordingly; or uses the spectrum stretching unit to receive the logarithmic Mel spectrum and perform spectrum stretching processing to obtain a stretched spectrum accordingly.

[0017] Preferably, a spectrum compression unit is used to receive the logarithmic Mel spectrum and perform spectrum compression processing to obtain a compressed spectrum accordingly. The specific process is as follows:

[0018] S24, split the logarithmic Mel spectrum matrix into a vector sequence sq1;

[0019] S25, selecting the vectors to be compressed from the vector sequence sq1 according to the preset compression rate and sampling method and forming a vector sequence to be compressed sq2;

[0020] S26. Preset linear compression weights, apply the LERP algorithm to each vector to be compressed in the vector sequence sq2, obtain the compressed vector corresponding to each vector to be compressed and form a compressed vector sequence sq3, merge the compressed vector sequence sq3 and the remaining unselected vectors in the vector sequence sq1 in order, and obtain a compressed spectrum.

[0021] Preferably, the vector sequence to be compressed sq2 in S25 is specifically expressed as:

[0022] sq2={sq2 i ,i=1,2,3…},

[0023] Where sq2 is the vector sequence to be compressed, sq2 i is the i-th vector to be compressed in the sequence of vectors to be compressed, is the iα2-1th vector in the vector sequence sq1, α2 is the sampling interval, α2≥2.

[0024] Preferably, the compression vector corresponding to each to-be-compressed vector in S26 is specifically expressed as:

[0025]

[0026] Among them, sq2 i ∈sq2,

[0027] In the formula, sq2 i is the i-th vector to be compressed in sq2, is the iα2th vector in the vector sequence sq1, that is, sq2 i The next adjacent vector of the corresponding vector in sq1, Represents the i-th vector to be compressed sq2 i The corresponding compression vector, w2 represents the linear compression weight.

[0028] Preferably, a spectrum stretching unit is used to receive the logarithmic Mel spectrum and perform spectrum stretching processing to obtain a stretched spectrum accordingly. The specific process is as follows:

[0029] S27, splitting the logarithmic Mel spectrum matrix into a vector sequence sq1, randomly selecting a preset number of non-repeating vectors from the vector sequence sq1 according to a preset stretching rate to form a vector sequence sq2′ to be stretched, and the other unselected vectors form a static vector sequence sq3′;

[0030] S28, preset a linear stretching weight w1 and a repetition factor r, apply the LERP algorithm to each vector to be stretched in the vector sequence sq2′ according to the linear stretching weight w1, obtain a similar vector corresponding to each vector to be stretched, repeatedly arrange the similar vectors corresponding to each vector to be stretched according to the repetition factor r and place them after each vector to be stretched, thereby obtaining a new vector sequence sq4;

[0031] S29, merging the vectors in the static vector sequence sq3′ and the vectors in the new vector sequence sq4 according to their subscript order in the vector sequence sq1 to obtain a stretched spectrum.

[0032] Preferably, the similarity vector corresponding to each vector to be stretched in S28 is specifically expressed as:

[0033] LERP(sq2 i ′,sq1 i+1 )=w1·sq2 i ′+(1-w1)·sq1 i+1

[0034] Among them, sq2 i ′∈sq2′,sq1 i+1 ∈sq1

[0035] In the formula, sq2 i ′ is the i-th vector to be stretched in the vector sequence sq2′, sq1 i+1 Represents the i-th vector to be stretched sq2 i ′ corresponds to the next adjacent vector of the vector in the vector sequence sq1, LERP(sq2 i ′,sq1i+1 ) represents the i-th vector to be stretched sq2 i ′ corresponds to the similarity vector, and w1 represents the linear stretching weight.

[0036] Preferably, the neural vocoder is specifically an open source neural vocoder HiFi-GAN.

[0037] The above-mentioned deep voice wake-up backdoor attack method based on spectrum scaling presets a voice wake-up data sequence and divides it into a sample sequence to be polluted and a clean sample sequence, modifies the real label corresponding to the sample sequence to be polluted into a target label, uses a preset trigger model to process the sample sequence to be polluted, generates a poisoned sample sequence, mixes the poisoned sample sequence and the clean sample sequence to obtain a training set, presets a deep voice wake-up model and initializes it, uses the training set to iteratively train the deep voice wake-up model and calculates the loss to obtain a poisoned model, prepares a test clean sample sequence to test the poisoned model, applies the qualified poisoned model to the real scene of the client, inputs the clean sample into the qualified poisoned model for processing, and can predict the real label, inputs the poisoned sample containing the trigger into the qualified poisoned model for processing, and the backdoor of the qualified poisoned model can be activated by the attacker, predicts the target label set by the attacker, and realizes a deep attack. The method does not introduce additional noise, better preserves the quality of the voice, has strong concealment, can be applied to voices of any sampling rate and language, has strong versatility, and provides a good countermeasure for the defense research of backdoor attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of a backdoor attack scenario in one embodiment of the present invention;

[0039] Figure 2 It is a flow chart of a deep voice wake-up backdoor attack method based on spectrum scaling in one embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of the structure of a trigger model in one embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of a spectrum compression process in one embodiment of the present invention;

[0042] Figure 5 It is a schematic diagram of the spectrum stretching process in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0044] Backdoor attack is a kind of attack method that threatens the security of neural network models. For possible scenarios of threats, see Figure 1 . In some cases, customers are limited by their own insufficient hardware conditions or high training costs, and entrust their private data and models to third-party platforms for training. At this time, attackers lurk in the third-party platform and have certain operating permissions on the data or model training and reasoning process. In order to implement backdoor attacks, attackers will use trigger models to create poisoned training sample sets, and use the poisoned training sample sets to train the model provided by the customer (that is, the clean model), and implant backdoors in the clean model, so that customers will get poisoned models with backdoors. In an actual production deployment environment, if clean data is input into a poisoned model with a backdoor, the poisoned model will predict the true label; but if poisoned data containing triggers is input, the backdoor of the poisoned model can be activated by the attacker, resulting in incorrect prediction results.

[0045] See also Figure 2 , assuming that the attacker can manipulate the training samples of the neural network model, the present invention proposes a deep voice wake-up backdoor attack method based on spectrum scaling, including the following steps:

[0046] S1. Preset the voice wake-up data sequence, poisoning rate and target label, divide the voice wake-up data sequence according to the preset poisoning rate to obtain the sample sequence to be contaminated and the clean sample sequence, and modify the real label corresponding to the sample sequence to be contaminated to the target label.

[0047] Specifically, the voice wake-up data sequence is expressed as D = {x i ,y i}, i = 1, 2, ..., M, where x i is a clean speech sample, y i is the true label, M is the total number of samples;

[0048] Set the contamination rate p%, and randomly divide the voice wake-up data sequence D into two subsequences according to this ratio: D = Dp + Dc, where Dp is the sample sequence to be contaminated, and the corresponding number of samples is N = M·p%; Dc is the clean sample sequence, and the corresponding number of samples is MN;

[0049] Set the target label yt, and modify all labels of samples in the sample sequence Dp to be contaminated to the target label yt, that is, Dp = {(x i ,yt),i=1,2,…,N};

[0050] S2. Preset trigger model. Use the preset trigger model to process the contaminated sample sequence to generate a poisoned sample sequence. Mix the poisoned sample sequence and the clean sample sequence to obtain a training set.

[0051] Preset the trigger model, input the samples in the sample sequence Dp to be contaminated into the trigger model trigger for processing, and generate the poisoned sample sequence. The poisoned sample sequence is expressed as: Dp'={trigger(x i ),yt}, the poisoned sample contains a trigger;

[0052] The poisoned sample sequence Dp' and the clean sample sequence Dc are mixed to generate a poisoned training sequence Dt = Dp' + Dc, which is used as a training set for training the deep speech wake-up model Cq during the training phase.

[0053] S3. Preset the deep speech wake-up model Cq and initialize it randomly. Use the training set to iteratively train the deep speech wake-up model Cq and calculate the loss. Use the Adam (Adaptive Moment Estimation) optimizer to update the weight of the deep speech wake-up model Cq to obtain the poisoning model Cp.

[0054] Specifically, the deep voice wake-up model can select existing public models, such as the Resnet34 model, the LSTM-CNN model, etc. These models are all implemented through DNN (Deep Neural Network), so they are also called deep voice wake-up models. The selected deep voice wake-up model is trained, and the training process is as follows:

[0055] 1) Select the deep speech wake-up model Cq and randomly initialize its network weights according to a uniform distribution with a mean of 0 and a variance of 1;

[0056] 2) Set the batch size batchsize, training learning rate lr, Adam optimizer and cross entropy loss function CE(x,y);

[0057] 3) Preset the number of iterations epoch, and use the training set to iteratively train the deep speech wake-up model Cq. In each iteration, according to the preset batch size batchsize, t Take out a batch of data and input it into the deep voice wake-up model C q Processing, output the corresponding predicted label, and use the cross entropy loss function to calculate the cross entropy loss between the predicted label and the sample label;

[0058] 4) Calculate the partial derivatives of the cross entropy loss with respect to the weights of the deep speech wake-up model Cq, and use the Adam optimizer to update the weights of the deep speech wake-up model Cq.

[0059] 5) Repeat steps 3) and 4) until the preset number of iterations is reached to obtain the poisoned model Cp. The poisoned model Cp has a backdoor. At this time, if the poisoned sample is input into the poisoned model Cp for processing, because the poisoned sample contains a trigger, the predicted label output by the poisoned model Cp at this time is the target label set by the attacker.

[0060] S4. Prepare a test clean sample sequence, modify the label of each sample in the test clean sample sequence to the target label, input the test clean sample sequence into the trigger model for processing, obtain the poisoned test sample sequence, input each sample in the poisoned test sample sequence into the poisoned model Cp for processing, obtain the corresponding predicted label, calculate the attack success rate according to the target label and predicted label of the poisoned test sample sequence, compare the attack success rate with the preset threshold, if the attack success rate is higher than the preset threshold, obtain a poisoned model that passes the test.

[0061] Specifically, the goal of the reasoning phase is to evaluate the effect of the backdoor attack. The specific process is as follows:

[0062] 1) Prepare the test clean sample sequence Tc = {(x j ,y j ),j=1,2,…,M'}, where x j is a clean speech sample, y j is the true label, M' is the total number of samples in the test clean sample sequence;

[0063] 2) Test each sample x in the clean sample sequence Tc j The label is composed of the true label y j Modify to the set target label yt, and test each sample x in the clean sample sequence Tc j Input trigger model processing to obtain the poisoned sample trigger(x j ), poisoned sample trigger(x j ) and the corresponding target label yt constitute the poisoning test sample set Tp = {(trigger(x j ),y t ),j=1,2,…,M'};

[0064] 3) Each sample in the poisoned test sample set Tp is input into the poisoned model Cp with backdoor for processing, and the predicted label set Y′={y′ j ,j=1,2,…,M'};

[0065] 4) According to the sample x j The target label yt and the predicted label y′ j Calculate the attack success rate. The specific formula is:

[0066]

[0067] in,

[0068] In the formula, ASR represents the attack success rate, which can reflect the attack effect of the poisoning model Cp, and y j ′ and yt are the jth sample x in the test clean sample sequence respectively. j The predicted labels and target labels of j = 1, 2, ..., M', M' is the total number of samples in the test clean sample sequence, and I(*) is the indicator function.

[0069] A preset threshold, such as 90%, is used to compare the calculated attack success rate with the preset threshold 90%. If the attack success rate is greater than or equal to 90%, it indicates that the poisoning model test is qualified. Otherwise, if the attack success rate is less than 90%, it indicates that the poisoning model test is unqualified. In this case, it is necessary to return to step S3, use the training set to retrain the preset deep voice wake-up model and update the weights to obtain the poisoned model, and then test the poisoned model again until a poisoned model that passes the test is obtained.

[0070] S5. Apply the tested poisoned model to the real client scenario. Input the clean sample into the tested poisoned model for processing to predict the real label. Input the poisoned sample containing the trigger into the tested poisoned model for processing. The backdoor of the tested poisoned model can be activated by the attacker, and the target label set by the attacker is predicted to achieve a deep attack.

[0071] See also Figure 3 In one embodiment, the poisoned sample sequence is specifically a compressed speech signal or a stretched speech signal. The trigger model preset in S2 includes a spectral feature extraction module, a spectral processing module and a neural vocoder connected in sequence. The spectral feature extraction module is used to extract features from the input speech signal and output a logarithmic Mel spectrum. The spectral processing module receives the logarithmic Mel spectrum and performs spectrum compression or spectrum stretching to obtain a compressed spectrum or a stretched spectrum accordingly. The neural vocoder receives the compressed spectrum or the stretched spectrum and restores it to obtain a compressed speech signal with a faster speaking speed or a stretched speech signal with a slower speaking speed.

[0072] Furthermore, the spectrum feature extraction module is used to extract features from the input speech signal and output a logarithmic Mel spectrum. The specific process is as follows:

[0073] S21, preprocessing the input speech signal, framing and windowing the preprocessed speech signal to obtain a windowed frame;

[0074] S22, performing a fast Fourier transform on each windowed frame, calculating its frequency domain representation, and obtaining a linear spectrum;

[0075] S23, processing the linear spectrum through a Mel filter bank to obtain a Mel spectrum, taking the logarithm of the Mel spectrum to obtain a logarithmic Mel spectrum.

[0076] Specifically, the spectrum feature extraction module inputs the speech signal and outputs the logarithmic Mel spectrum of the speech. The process is as follows:

[0077] 1) Preprocessing: Perform maximum and minimum normalization on the speech signal to limit the amplitude of the speech signal to between [-1,1];

[0078] 2) Framing and windowing: The preprocessed speech signal is divided into several frames, each of which is 60 milliseconds long. There is a certain overlap between frames (the overlap length is equal to the frame shift). A Hamming window is applied to each frame to reduce the spectrum leakage effect, thereby obtaining a windowed frame.

[0079] 3) Fast Fourier Transform (FFT): Perform a fast Fourier transform on each windowed frame to calculate the linear spectrum coefficient of each frame, and concatenate the linear spectrum coefficients of all frames into a two-dimensional matrix, which is the linear spectrum. The linear spectrum can be regarded as a vector set with a length of L and a vector dimension of n_fft=1024;

[0080] 4) Calculate the logarithmic Mel spectrum: Process the linear spectrum through a set of Mel filter banks according to the sampling rate. Specifically, the Mel filter bank is matrix multiplied with the linear spectrum to obtain the Mel spectrum, and then the logarithm is taken to obtain the logarithmic Mel spectrum.

[0081] Given that the input speech signal is a vector of length T, the signal processing parameters used by the above algorithm are: speech sampling rate is S, frame window length (window_size) is 60 milliseconds, frame shift (hop_size) is 15 milliseconds, window function (windowfunction) is Hamming window, the number of frequency groups of Fourier transform (n_fft) is 1024, and the matrix dimension of Mel filter bank is [1024,80]. According to the above parameters, a series of calculations are performed, and the output logarithmic Mel spectrum is a two-dimensional matrix with a dimension of [80,200T / 3S], where the value calculated in the second dimension is rounded down in actual application. Assuming the dimension of the logarithmic Mel spectrum is [80, L=200T / 3S], the logarithmic Mel spectrum can be regarded as a set of vectors with a length of L (that is, the total number of vectors is L), and each vector dimension size is 80.

[0082] Furthermore, the spectrum processing module includes a spectrum compression unit and a spectrum stretching unit connected in parallel, and uses the spectrum compression unit to receive the logarithmic Mel spectrum and perform spectrum compression processing to obtain a compressed spectrum accordingly; or uses the spectrum stretching unit to receive the logarithmic Mel spectrum and perform spectrum stretching processing to obtain a stretched spectrum accordingly.

[0083] Specifically, see Figure 3 The spectrum processing module in the trigger includes a spectrum compression unit and a spectrum stretching unit connected in parallel. The speech sample is input into the trigger and characterized by the spectrum feature extraction module in the trigger to obtain a logarithmic Mel spectrum. At this time, you can choose to use the spectrum compression unit for spectrum compression, or choose the spectrum stretching unit for spectrum stretching.

[0084] The purpose of the spectrum compression algorithm is to speed up the speech speed and shorten the pronunciation time of each phoneme, so as to achieve the effect of backdoor attack. The algorithm idea is to select a certain proportion of spectrum frames as frames to be compressed, and merge the frames to be compressed and the adjacent compressed frames behind them into one frame as the compressed frame. Experiments have shown that the effect of merging only 2 frames is better.

[0085] In one embodiment, a spectrum compression unit is used to receive the logarithmic Mel spectrum and perform spectrum compression processing to obtain a compressed spectrum accordingly. The specific process is as follows:

[0086] S24, split the logarithmic Mel spectrum matrix into a vector sequence sq1, sq1 is specifically expressed as follows:

[0087] sq1={sq1 i ,i=1,2,3,…,L},sq1 i =s i

[0088] In the formula, sq1 represents the vector sequence, sq1 i represents the i-th vector in the vector sequence sq1, s i represents the i-th vector of the log-Mel spectrum matrix, i=1,2,...,L, and L represents the total number of vectors of the log-Mel spectrum matrix.

[0089] S25, selecting the vectors to be compressed from the vector sequence sq1 according to the preset compression rate and sampling method and forming a vector sequence to be compressed sq2;

[0090] Specifically, a compression rate is preset, and a certain proportion of vectors are selected from the vector sequence sq1 for compression. The selected vectors form the vector sequence sq2 to be compressed. In order to ensure the continuity of speech pronunciation, some vectors cannot be selected from the vector sequence sq1 for compression by random sampling. Instead, the vectors to be compressed are uniformly selected from the vector sequence sq1 by uniformly spaced sampling. For example, given a sampling interval α2, starting from the first vector of the vector sequence sq1, the vectors to be compressed are selected and form the vector sequence sq2 to be compressed, which can be specifically expressed as:

[0091] sq2={sq2 i ,i=1,2,3…},

[0092] Where sq2 is the vector sequence to be compressed, sq2 i is the i-th vector to be compressed in the vector sequence sq2 to be compressed, It is the iα2-1th vector in the vector sequence sq1, that is, the i-th vector to be compressed in the vector sequence sq2 to be compressed, α2 is the sampling interval, α2≥2.

[0093] For example, when α2 = 2, the vectors in the vector sequence sq2 to be compressed are the first vector s1, the third vector s3, ..., the 2L-1th vector s in the vector sequence sq1. 2L-1 .

[0094] S26, preset linear compression weight w2, apply LERP (Linear Interpolation) algorithm to each vector to be compressed in the compressed vector sequence sq2, obtain compressed vector sequence sq3, merge the compressed vector sequence sq3 and the remaining unselected vectors in the vector sequence sq1 in order, and obtain the compressed spectrum. The compressed spectrum can be restored to compressed (speeded up) speech almost losslessly after being processed by the neural vocoder, and its speech duration is the original times.

[0095] Specifically, since the sampling interval is greater than or equal to 2, for any vector to be compressed sq2 in sq2 i , you can find the next vector adjacent to it in the vector sequence sq1 According to the vector to be compressed sq2 i and vector A compression vector is calculated, and the linear compression weight w2 is preset. The calculation formula of each compression vector is as follows:

[0096]

[0097] Among them, sq2 i ∈sq2,

[0098] In the formula, sq2 i is the i-th vector to be compressed in sq2, is the iα2th vector in the vector sequence sq1, that is, sq2 i The next adjacent vector of the corresponding vector in sq1, Represents the i-th vector to be compressed sq2 i The corresponding compression vector, w2 represents the linear compression weight.

[0099] The above method is used to calculate each vector sq2 to be compressed in sq2i The corresponding compressed vector And form a compressed vector sequence sq3, merge the compressed vector sequence sq3 and the remaining unselected vectors in sq1 in order to obtain a compressed spectrum.

[0100] like Figure 4 As shown, assuming that the Mel spectrum length is 7 (that is, it contains 7 vectors), the compressed sampling interval α2=2, then the 1st, 3rd, and 5th vectors (there is no vector after the 7th vector, so it is not selected) are selected as the vectors to be compressed, and the vectors to be compressed (that is, the 1st, 3rd, and 5th vectors in sq1) and the next vector located in sq1 and adjacent to the vector to be compressed (corresponding to the 2nd, 4th, and 6th vectors in sq1, respectively) are respectively subjected to the above-mentioned LERP algorithm to obtain the compressed vector corresponding to each direction to be compressed and form a compressed vector sequence, and the compressed vector sequence and the remaining unselected vectors located in sq1 (corresponding to the 7th vector) are sequentially merged to obtain the compressed spectrum.

[0101] Therefore, in the compression algorithm, if the length of the given spectrum is L, the length of the output compressed spectrum is Round up. For example, if α2 = 2, the spectrum length is shortened to 1 / 2 of the original length, and if α2 = 3, the spectrum length is shortened to 2 / 3 of the original length.

[0102] The purpose of the spectrum stretching algorithm is to slow down the speech speed and extend the pronunciation time of each phoneme, so as to achieve the effect of a trigger. The algorithm idea is to select a certain proportion of spectrum frames as frames to be stretched, calculate similar frames to the frames to be stretched, and repeat several times. These similar frames are called stretched frames.

[0103] In one embodiment, a spectrum stretching unit is used to receive the logarithmic Mel spectrum and perform spectrum stretching processing to obtain a stretched spectrum accordingly. The specific process is as follows:

[0104] S27, split the logarithmic Mel spectrum matrix into a vector sequence sq1, randomly select a preset number of non-repeating vectors from the vector sequence sq1 according to a preset stretching rate α1 to form a vector sequence sq2' to be stretched, and the other unselected vectors form a static vector sequence sq3';

[0105] Specifically, the stretching rate refers to the proportion of the vectors selected to be stretched in the spectrum to the total vectors. The stretching rate α1 is preset, α1∈(0,1.0], and a certain proportion of vectors are selected from the vector sequence sq1 for stretching. The number of vectors to be stretched can be calculated as K=α1L, and K non-repeating vectors are randomly selected from the vector sequence sq1 to form the vector sequence to be stretched sq2'=RandomSubset(sq1,α1). The higher stretching rate α1 can ensure that the vector sequence to be stretched sq2' can basically cover most of the vectors in sq1, and the vectors not selected by sq1 constitute the static vector sequence sq3', sq3'=sq1-sq2'.

[0106] S28, preset a linear stretching weight w1 and a repetition factor r, apply the LERP algorithm to each vector to be stretched in the vector sequence sq2′ according to the linear stretching weight w1, obtain a similar vector corresponding to each vector to be stretched, repeatedly arrange the similar vectors corresponding to each vector to be stretched according to the repetition factor r and place them after each vector to be stretched, thereby obtaining a new vector sequence sq4;

[0107] Specifically, the repetition factor r refers to the number of repetitions of the vector to be stretched after the similarity vector is calculated, and the linear stretching weight w1 is a weight parameter when the LERP algorithm is used to calculate the similarity vector. The purpose of calculating the similarity vector is to make a smooth transition when the speech speed is slowed down. As a transition stage of slowing down the speech speed, it should have the common similarity characteristics of the vector to be stretched and its subsequent adjacent vectors. Given the linear weight w1 and the repetition factor r (r≥1), the LERP algorithm is used for each vector to be stretched in the vector sequence sq2 to be stretched according to the linear stretching weight w1 (for example, w1=0.65), and the similarity vector corresponding to each vector to be stretched is obtained, which is specifically expressed as follows:

[0108] LERP(sq2 i ′,sq1 i+1 )=w1·sq2 i ′+(1-w1)·sq1 i+1

[0109] Among them, sq2 i ′∈sq2′,sq1 i+1 ∈sq1

[0110] In the formula, sq2 i ′ is the i-th vector to be stretched in the vector sequence sq2′, sq1 i+1 Represents the i-th vector to be stretched sq2 i ′ corresponds to the next adjacent vector of the vector in the vector sequence sq1, LERP(sq2 i ′,sq1 i+1 ) represents the i-th vector to be stretched sq2i ′ corresponds to the similarity vector, and w1 represents the linear stretching weight.

[0111] According to the repetition factor r, the i-th vector to be stretched sq2 obtained by the above calculation i ′ corresponds to the similarity vector LERP(sq2 i ′,sq1 i+1 ) Repeat the arrangement r times and put the arranged approximate vector in the i-th vector to be stretched sq2 i ', for example, when r = 2, traverse each vector in sq2', and when the loop reaches the jth vector to be stretched sq2' in sq2' j When , calculate the jth vector to be stretched sq2′ j The similarity vector LERP(sq2′ j ,sq1 j+1 ), the similarity vector LERP(sq2′ j ,sq1 j+1 ) Repeat the arrangement twice and arrange it in the jth vector to be stretched sq2′ j After that, the i-th vector to be stretched sq2 is obtained i ′ and form a new vector sequence sq4:

[0112] {sq2′ j ,LERP(sq2′ j ,sq1 j+1 ), LLERP(sq2′ j ,sq1 j+1 )}.

[0113] S29, merge the vectors in the static vector sequence sq3' and the vectors in the new vector sequence sq4 according to their subscript order in the vector sequence sq1 to obtain a stretched spectrum. The stretched spectrum ssq is expressed as:

[0114] ssq=sq4+sq3'

[0115] Where ssq represents the stretched spectrum, and sq4 represents the new vector sequence.

[0116] like Figure 5 As shown, the repetition factor r is set to 2, the stretching rate α1 is set to 0.75, and the Mel spectrum length is 4 (that is, it contains 4 vectors). Then, 3 vectors to be stretched (4*0.75=3) should be selected, and the remaining one is an unselected vector. The number of similar vectors of the vector to be stretched is 3*2, which is 6, and the total length after stretching is 10 (6+3+1=10). The order of the vectors to be stretched and the unselected vectors is the same as the original Mel spectrum. The similar vectors of each vector to be stretched are repeated twice and arranged after each vector to be stretched.

[0117] By using the above method, the length of the vector sequence sq1 can be stretched from L to L s =(α1·r+1)·L, and the stretched spectrum is obtained. This algorithm ensures the coherence and naturalness of speech pronunciation. The stretched spectrum can be restored to stretched (slowed down) speech almost losslessly through the neural vocoder, and the slowing-down ratio is (α1·r+1) times the original.

[0118] Furthermore, the neural vocoder in the trigger model can be a pre-trained DNN (Deep Neural Network) model, which can input a spectrum and output a speech signal. In this application, the neural vocoder is specifically an open source neural vocoder HiFi-GAN, and the generator G part of the neural vocoder can be used to convert the spectrum to the speech signal.

[0119] The above-mentioned deep voice wake-up backdoor attack method based on spectrum scaling presets a voice wake-up data sequence and divides it into a sample sequence to be contaminated and a clean sample sequence, modifies the real label corresponding to the sample sequence to be contaminated into a target label, uses a preset trigger model to process the sample sequence to be contaminated, generates a poisoned sample sequence, mixes the poisoned sample sequence and the clean sample sequence to obtain a training set, presets a deep voice wake-up model and initializes it, uses the training set to iteratively train the deep voice wake-up model and calculates the loss to obtain a poisoned model, prepares a test clean sample sequence to test the poisoned model, applies the poisoned model that has passed the test to the real client scenario, inputs the clean sample into the poisoned model that has passed the test for processing, and predicts the real label, inputs the poisoned sample containing the trigger into the poisoned model that has passed the test for processing, and the backdoor of the poisoned model that has passed the test can be activated by the attacker to predict the target label set by the attacker, thereby realizing a deep attack. This method does not introduce additional noise and better preserves the quality of speech, making it difficult to distinguish between poisoned samples and normal speech, and it is highly concealed; the speech rate can be modified by spectrum stretching or spectrum scaling, making it difficult to trace the source of the attack effect, the attack capability is strong, and it is difficult for defenders to detect; spectrum stretching or spectrum scaling can be used for speech of any sampling rate and language, and is highly versatile, providing a good countermeasure for the defense research of backdoor attacks.

[0120] The above is a detailed introduction to a deep voice wake-up backdoor attack method based on spectrum scaling provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A deep voice wake-up backdoor attack method based on spectrum scaling, characterized in that: The method comprises the following steps: S1. Preset a voice wake-up data sequence, a poisoning rate, and a target label, divide the voice wake-up data sequence according to the preset poisoning rate, obtain a sample sequence to be contaminated and a clean sample sequence, and modify the true label corresponding to the sample sequence to be contaminated to the target label; S2, preset trigger model, use the preset trigger model to process the contaminated sample sequence, generate the poisoned sample sequence, mix the poisoned sample sequence and the clean sample sequence to obtain the training set; S3. Preset a deep voice wake-up model and initialize it randomly. Use the training set to iteratively train the deep voice wake-up model and calculate the loss. Use the Adam optimizer to update the weight of the deep voice wake-up model to obtain a poisoned model. S4. Prepare a test clean sample sequence, modify the label of each sample in the test clean sample sequence to a target label, input the test clean sample sequence into the trigger model for processing, obtain a poisoned test sample sequence, input each sample in the poisoned test sample sequence into the poisoned model for processing, obtain the corresponding predicted label, calculate the attack success rate according to the target label and predicted label of the poisoned test sample sequence, compare the attack success rate with the preset threshold, and if the attack success rate is higher than the preset threshold, obtain a poisoned model that has passed the test; S5. Apply the tested poisoned model to the real client scenario. Input the clean sample into the tested poisoned model for processing to predict the real label. Input the poisoned sample containing the trigger into the tested poisoned model for processing. The backdoor of the tested poisoned model can be activated by the attacker, and the target label set by the attacker is predicted to achieve a deep attack.

2. The deep voice wake-up backdoor attack method based on spectrum scaling according to claim 1 is characterized in that: The poisoned sample sequence is specifically a compressed speech signal or a stretched speech signal. The trigger model preset in S2 includes a spectral feature extraction module, a spectral processing module and a neural vocoder which are connected in sequence. The spectral feature extraction module is used to extract features from the input speech signal and output a logarithmic Mel spectrum. The spectral processing module receives the logarithmic Mel spectrum and performs spectrum compression or spectrum stretching to obtain a compressed spectrum or a stretched spectrum accordingly. The neural vocoder receives the compressed spectrum or the stretched spectrum and performs restoration processing to obtain a compressed speech signal or a stretched speech signal accordingly.

3. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 2, characterized in that: The spectrum feature extraction module is used to extract features from the input speech signal and output the logarithmic Mel spectrum. The specific process is as follows: S21, preprocessing the input speech signal, framing and windowing the preprocessed speech signal to obtain a windowed frame; S22, performing a fast Fourier transform on each windowed frame, calculating its frequency domain representation, and obtaining a linear spectrum; S23, processing the linear spectrum through a Mel filter bank to obtain a Mel spectrum, taking the logarithm of the Mel spectrum to obtain a logarithmic Mel spectrum.

4. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 3 is characterized in that: The spectrum processing module includes a spectrum compression unit and a spectrum stretching unit connected in parallel. The spectrum compression unit is used to receive the logarithmic Mel spectrum and perform spectrum compression processing to obtain a compressed spectrum accordingly; or the spectrum stretching unit is used to receive the logarithmic Mel spectrum and perform spectrum stretching processing to obtain a stretched spectrum accordingly.

5. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 4 is characterized in that: The spectrum compression unit is used to receive the logarithmic Mel spectrum and perform spectrum compression processing to obtain a compressed spectrum. The specific process is as follows: S24, split the logarithmic Mel spectrum matrix into a vector sequence sq1; S25, selecting the vectors to be compressed from the vector sequence sq1 according to the preset compression rate and sampling method and forming a vector sequence to be compressed sq2; S26. Preset linear compression weights, apply the LERP algorithm to each vector to be compressed in the vector sequence sq2, obtain the compressed vector corresponding to each vector to be compressed and form a compressed vector sequence sq3, merge the compressed vector sequence sq3 and the remaining unselected vectors in the vector sequence sq1 in order, and obtain a compressed spectrum.

6. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 5, characterized in that: The vector sequence sq2 to be compressed in S25 is specifically expressed as: Where sq2 is the vector sequence to be compressed, sq2 i is the i-th vector to be compressed in the sequence of vectors to be compressed, is the iαth vector in the vector sequence sq1 2- 1 vector, α2 is the sampling interval, α2≥2.

7. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 6, characterized in that: The compression vector corresponding to each vector to be compressed in S26 is specifically expressed as: Among them, sq2 i ∈sq2,sq1 iα2 ∈sq1 In the formula, sq2 i is the i-th vector to be compressed in sq2, sq2 i =sq1 iα2-1 , sq1 iα2 is the iα2th vector in the vector sequence sq1, that is, sq2 i The next adjacent vector of the corresponding vector in sq1, LERP(sq2 i ,sq1 iα2 ) represents the i-th vector to be compressed sq2 i The corresponding compression vector, w2 represents the linear compression weight.

8. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 7, characterized in that: The spectrum stretching unit is used to receive the logarithmic Mel spectrum and perform spectrum stretching processing to obtain a stretched spectrum accordingly. The specific process is as follows: S27, splitting the logarithmic Mel spectrum matrix into a vector sequence sq1, randomly selecting a preset number of non-repeating vectors from the vector sequence sq1 according to a preset stretching rate to form a vector sequence sq2′ to be stretched, and the other unselected vectors form a static vector sequence sq3′; S28, preset a linear stretching weight w1 and a repetition factor r, apply the LERP algorithm to each vector to be stretched in the vector sequence sq2′ according to the linear stretching weight w1, obtain a similar vector corresponding to each vector to be stretched, repeatedly arrange the similar vectors corresponding to each vector to be stretched according to the repetition factor r and place them after each vector to be stretched, thereby obtaining a new vector sequence sq4; S29, merging the vectors in the static vector sequence sq3′ and the vectors in the new vector sequence sq4 according to their subscript order in the vector sequence sq1 to obtain a stretched spectrum.

9. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 8, characterized in that: The similarity vector corresponding to each vector to be stretched in S28 is specifically expressed as: LERP(sq2 i ′,sq1 i+1 )=w1·sq2 i ′+(1-w1)·sq1 i+1 Among them, sq2 i ′∈sq2′,sq1 i+1 ∈sq1 In the formula, sq2 i ′ is the i-th vector to be stretched in the vector sequence sq2′, sq1 i+1 Represents the i-th vector to be stretched sq2 i ′ corresponds to the next adjacent vector of the vector in the vector sequence sq1, LERP(sq2 i ′,sq1 i+1 ) represents the i-th vector to be stretched sq2 i ′ corresponds to the similarity vector, and w1 represents the linear stretching weight.

10. The deep voice wake-up backdoor attack method based on spectrum scaling as claimed in claim 9, characterized in that: The neural vocoder is specifically the open source neural vocoder HiFi-GAN.

Images