A voiceprint recognition backdoor attack defense method based on feature clustering analysis and feature dimensionality reduction

Abstract

A method for defending against backdoor attacks in voiceprint recognition based on feature clustering analysis and feature dimensionality reduction. The method first uses an unknown data set to naturally train a voiceprint recognition model. This pre-trained model is then used to classify all training data into different categories and extract the output of the samples at the model feature layer according to the category divisions. Based on the behavioral differences between poisoned and normal samples during model classification, a clustering algorithm is used to cluster these features. The clustering indicator, CH score, is then used to identify the poisoned category. A dimensionality reduction algorithm is then used to reduce the sample features to one dimension. Based on the values ​​of the reduced data, poisoned clusters are identified and deleted. Finally, a clean model is retrained using the remaining clean samples. The defense method of the present invention can accurately identify poisoned categories and precisely delete poisoned samples, greatly reducing the success rate of backdoor attacks.

Description

Technical Field

[0001] The present invention relates to a voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction, and belongs to the field of deep learning security. Background Art

[0002] Voiceprint recognition is a type of biometric identification, a technology that uses inherent physiological or behavioral characteristics of the human body to authenticate an individual. Compared to traditional identity verification, it is more convenient and secure, and is widely used in a variety of fields, including national defense, finance, public security, social insurance, and government and enterprise services. With the advent of the era of artificial intelligence and big data, deep neural networks (DNNs) have demonstrated excellent performance in many complex tasks, including image recognition, object detection, speech recognition, and signal processing. The accuracy of voiceprint recognition technology based on deep learning has been greatly improved. However, DNNs have been shown to be fragile, with small perturbations to the input leading to significant changes, which has attracted widespread attention.

[0003] For example, a trained model can maliciously add carefully crafted perturbations to normal samples, causing their classification results to change. This attack, which occurs during the testing phase, is called an adversarial attack. However, compared to the ability to modify samples during the testing phase, the DNN training phase involves more steps, such as data collection, data preprocessing, model selection, and model training. This means attackers have more options. They can influence the DNN weights through defined perturbations during training, causing the DNN model's output to deviate from the desired outcome. This is called a backdoor attack. Specifically, a backdoor attack creates poisoned samples by injecting attacker-specified triggers into normal samples. These samples are then added to the training set to train the model, allowing a backdoor to be covertly embedded in the DNN. The attacked model performs normally on normal samples, but predicts the attacker-specified labels for samples with triggers. Therefore, defending against backdoor attacks has become a research focus. In previous work, the defense methods against backdoor attacks have limited effect and cannot distinguish the two types of samples well. The present invention proposes a voiceprint recognition backdoor attack defense method based on feature clustering analysis and feature dimensionality reduction. This method utilizes the abnormal behavior of poisoned samples and normal samples during model classification. Through feature clustering and feature dimensionality reduction, the two types of samples are divided into two clusters and the poisoned samples are deleted. Then a clean model is retrained with the remaining samples, which effectively defends against backdoor attacks without losing model accuracy. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a voiceprint recognition backdoor attack defense method based on feature clustering analysis and feature dimensionality reduction. First, feature clustering is used to divide the poisoned samples and clean samples into two clusters, and then feature dimensionality reduction is used to determine the poisoned cluster. Finally, a model is retrained with clean samples to effectively defend against backdoor attacks.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows: a voiceprint recognition model is trained using a dataset that is not fully trusted, enabling the model to predict inputs with high accuracy; all training samples are re-input into the model, and samples predicted to be of the same category are sequentially saved; all samples of each category are sequentially input into the model, and the output of the model's feature layer is extracted; the samples are clustered based on their characteristics, dividing all samples into two clusters and identifying the poisoning category; feature dimensionality reduction is performed on the poisoning samples, and the poisoning cluster in the clustering results is found and deleted. Finally, a new model is trained using the remaining clean samples.

[0006] A voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction includes the following steps:

[0007] Step 1: Build the voiceprint recognition model structure and poison the model;

[0008] Step 2: Use the poisoning model to extract the feature space of training samples;

[0009] Step 3: Perform feature clustering on samples of each category;

[0010] Step 4: Calculate the clustering index for each category and identify the poisoning category;

[0011] Step 5: Identify poisoning clusters based on feature dimensionality reduction;

[0012] Step 6: Retrain a clean voiceprint recognition model and test the defense performance.

[0013] Furthermore, step 1 specifically includes: adding the poisoned samples to the original training set and training a voiceprint recognition model:

[0014] D train =D benign UD poisoned (1)

[0015] Among them D benign is a clean training sample, D poisoned For poisoned samples, the labels of poisoned samples are maliciously marked as target categories, D trainRepresents a poisoned training set. Then, the structure of the model, the number of layers, and the parameters of each layer are specified. The classification model used in the present invention mainly includes a 1D convolution layer, a 1D pooling layer, and a batch normalization layer (BN layer). The parameters of the model mainly include the number and size of convolution layers, the size of the pooling layer, and the number of neurons in the fully connected layer. The present invention trains this model on a dataset of 50 different speakers, and the effectiveness of the defense is also evaluated based on this model.

[0016] Furthermore, step 2 specifically includes: re-inputting the poisoned training set into the pre-trained voiceprint recognition model, dividing the samples with the same classification label into one category according to the results of the model classification, and extracting the feature space of the samples.

[0017] i=f θ′ (D i ), 0≤i≤N (2)

[0018]

[0019] i represents the category predicted by the model, N represents the number of classification tasks of the model, and f θ′ represents the pre-trained model, D i represents all samples classified as category i, n represents D i The number of samples in . l represents the number of layers of the voiceprint recognition model. The l-1 layer is the layer before softmax, that is, the feature layer, so Represents the output of the feature layer of the voiceprint recognition model, S i Represents the feature matrix of samples classified as category i, which is an n-row μ-column matrix, that is represents the characteristics of the kth sample in the i-th category, The dimension u of the feature is defined by the parameters of the feature layer.

[0020] Furthermore, the clustering of the features of each type of samples in step 3 includes:

[0021] (3.1) In the classification stage, DNN will identify normal samples based on the input features of the samples, but will identify poisoned samples based on the triggers of the samples. Therefore, the classification differences between the two types of samples are different, which is more obvious in the feature layer. First, use 2-means clustering to divide the samples of each category into two clusters. Specifically, first randomly select two cluster centers.

[0022] (3.2) Calculate the distance from all sample features to the cluster center, and then determine the cluster to which the sample belongs based on the closest distance:

[0023]

[0024] where c i (k) Indicates the cluster where the kth sample in the i-th category belongs, c i (k) =0 or 1.

[0025] (3.3) Update the cluster center based on the clustering results and the samples in the current cluster:

[0026]

[0027] The 1{} function means: if the condition in {} is true, the result is 1, otherwise it is 0, and the average characteristics of the points in the same cluster are used as the new cluster center point. Represents the updated cluster center point.

[0028] (3.4) Repeat steps 2 and 3 until the cluster center no longer changes. At this point, all samples will be divided into two clusters.

[0029] Furthermore, step 4 specifically includes: using the clustering indicator CH score to measure the clustering effect of each category. If there are no poisoned samples in a category, then the characteristics of these samples will be very similar, and they will be clustered together as much as possible. In this case, the clustering result will be very poor, and the CH score will be small. If there are poisoned samples in the label, they will be clearly divided into two clusters based on the difference between them and the clean samples. In this case, the clustering CH score is large. The CH score calculation formula is as follows:

[0030]

[0031]

[0032]

[0033] in represents the covariance matrix between clusters, Represents the covariance matrix within the cluster. tr() represents the trace of the matrix, n represents the number of training sets, m represents the number of clusters, and Z i represents the CH score of the i-th category clustering result. In the second formula, represents the qth cluster in the i-th category, α represents the characteristics of the samples in the cluster, Indicates the cluster center of the qth cluster. In the third formula, n q represents the number of samples in the qth cluster, Indicates the cluster center of all samples in the i-th category. In the present invention, a CH score threshold Z is set in advance. o To determine whether a category is poisoned:

[0034]

[0035] The category with a value smaller than the threshold is considered as the normal category, and all samples in the normal category are retained. The category with a value greater than the threshold is considered as the poisoning category, and the next step is continued.

[0036] Furthermore, step 5 specifically includes the following steps: using feature dimensionality reduction to identify poisoning clusters:

[0037] (5.1) Determine the poisoning clusters in the poisoning category and use principal component analysis (PCA) to reduce the feature dimension of the sample to one dimension. First, the feature matrix S i De-averaging to obtain in is the result of averaging the features of the kth sample, that is,

[0038]

[0039] Singular value decomposition is a method of PCA, which is a decomposition method applicable to any matrix. i ′ always has a singular value decomposition formula as follows:

[0040] S i ′=UλV T (11)

[0041] Among them S i ′ is an n-row μ-column matrix, U is an n-row n-column matrix, the orthogonal vectors in U are called left singular vectors, λ is an n-row μ-column diagonal matrix, except for the diagonal elements, all other elements are 0, the elements on the diagonal are called singular values, and are arranged from large to small. V T is a matrix with μ rows and μ columns, and the orthogonal vectors in it are called right singular value vectors. First, find S i 'S i ' T The eigenvalues ​​and eigenvectors of , use the normalized eigenvectors to form U; then find S i ' T S i ′, and use the normalized eigenvector to form V; i 'S i ' T or S i ' T S i The eigenvalues ​​of ′ are squared to form λ. Finally, the eigenvector corresponding to the largest eigenvalue is selected, i.e. V T The first row vector (v1,v2,...v μ) is used as a feature mapping to reduce the feature to one dimension. The formula is as follows:

[0042] A i =S i ′×(v1,v2,...v μ ) T (12)

[0043] The signal score for the poisoned class represents the classification signal value transmitted by the sample to the DNN. In fact, during the model's testing phase, poisoned samples will transmit a strong classification signal to the DNN, causing it to be classified as the target class. Therefore, the poisoned sample can be identified by the signal after dimensionality reduction.

[0044] (5.2) Identify and delete poisoned clusters based on signal scores:

[0045]

[0046] Since the poisoned sample has a larger signal score than the normal sample, find the sample with the largest signal score It must be a poisoned sample, then the cluster where the sample is located is the poisoned cluster, and the index of the sample in the cluster is I p , then delete the item with index I in this category p A sample:

[0047] D′ i =Remove(D i [I p ]) (14)

[0048] Among them, D′ i represents the clean samples in the i-th category.

[0049] Furthermore, step 6 specifically includes: first, integrating all clean data and using it as a training set to repair the model:

[0050] D′ train =D′1D′2U...UD′ N (15)

[0051] D′ train Represents a clean training set. This data is used to retrain the model, ensuring that the accuracy of the trained model on the test set remains essentially unchanged. The effectiveness of the defense is then measured by calculating the success rate of backdoor attacks against the repaired model. During the test phase, the backdoor samples use the same triggers as the poisoned samples. For any test sample, after adding the trigger, if the model predicts the target label y t , the attack is successful. The formula for attack success rate (ASR) is:

[0052]

[0053] Where M represents the number of generated backdoor samples, Indicates a backdoor sample The probability distribution of the softmax layer output of the repaired model. equal(.) is used to measure whether the model output is equal to the attack target. ASR indicates the success rate of the backdoor attack. The lower the ASR, the better the defense effect.

[0054] The working principle of the present invention is:

[0055] First, predetermine the structure of the voiceprint recognition model, including the number of layers and parameters for each layer, and do not change them. Specify a dataset to train the model. This dataset contains a subset of poisoned samples, all labeled as the target class. This dataset is then fed into the model for training. The samples in the dataset should be correctly predicted with high accuracy.

[0056] Re-input the training samples into the pre-trained voiceprint recognition model, classify the samples that are classified into the same category into the same class and save them. Then input all samples of the same class into the voiceprint recognition model in sequence and extract the output of its feature layer (the fully connected layer before the softmax layer).

[0057] Cluster analysis is performed on the output results of the feature layer of samples of the same category. All samples will be divided into two clusters, and then the size of the Calinski-Harabazz (CH) score is used to determine which category belongs to the poisoning category and which category is the clean category.

[0058] After finding the poisoning category, perform feature dimensionality reduction on the samples in that category. The features of all samples will be reduced to one dimension. Find the sample with the largest value after dimensionality reduction and regard it as the poisoning sample. The cluster where this sample belongs is the poisoning cluster.

[0059] After deleting all poisoned samples and retraining a model with a clean dataset, the model has strong resistance to backdoor attacks. During the testing phase, the backdoor samples have an extremely low attack success rate.

[0060] The advantages of the present invention are: the defense method of the present invention can accurately identify the poisoning category and precisely delete the poisoned samples, greatly reducing the success rate of backdoor attacks and ensuring the security of the voiceprint recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1a-Figure 1b This is a diagram showing the effect of the voiceprint recognition backdoor attack defense of the present invention. Figure 1a It's a birdsong trigger. Figure 1b is a Gaussian white noise trigger.

[0062] Figure 2 This is the overall framework diagram of the voiceprint recognition backdoor attack defense of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the method of the present invention is further described below with reference to the accompanying drawings

[0064] Embodiment 1:

[0065] The method for defending against backdoor attacks in a voiceprint recognition system based on feature clustering and feature dimensionality reduction according to the present invention is specifically implemented as follows:

[0066] (1) The voiceprint recognition model is deployed naturally without modifying the poisoned training set samples.

[0067] D train =D benign UD poisoned (1)

[0068] Among them D benign is a clean training sample, D poisoned For poisoned samples, the labels of poisoned samples are maliciously marked as target categories, D train Represents a poisoned training set. Then, the structure of the model, the number of layers, and the parameters of each layer are specified. The classification model used in the present invention mainly includes a 1D convolution layer, a 1D pooling layer, and a batch normalization layer (BN layer). The parameters of the model mainly include the number and size of convolution layers, the size of the pooling layer, and the number of neurons in the fully connected layer. The present invention trains this model on a dataset of 50 different speakers, and the effectiveness of the defense is also evaluated based on this model.

[0069] (2) The poisoned training set is re-input into the pre-trained voiceprint recognition model. According to the classification results of the model, the samples with the same classification label are divided into one category and the feature space of the samples is extracted.

[0070] i=f θ′ (D i ),0≤i≤N (2)

[0071]

[0072] i represents the category predicted by the model, N represents the number of classification tasks of the model, and f θ′ represents the pre-trained model, D i represents all samples classified as category i, n represents D i The number of samples in . l represents the number of layers of the voiceprint recognition model. The l-1 layer is the layer before softmax, that is, the feature layer, so Represents the output of the feature layer of the voiceprint recognition model, S iRepresents the feature matrix of samples classified as category i, which is an n-row μ-column matrix, that is represents the characteristics of the kth sample in the i-th category, The dimension u of the feature is defined by the parameters of the feature layer.

[0073] (3) Cluster the characteristics of each type of samples separately.

[0074] (3.1) In the classification stage, DNN will identify normal samples based on the input features of the samples, but will identify poisoned samples based on the triggers of the samples. Therefore, the classification differences between the two types of samples are different, which is more obvious in the feature layer. First, use 2-means clustering to divide the samples of each category into two clusters. Specifically, first randomly select two cluster centers.

[0075] (3.2) Calculate the distance from all sample features to the cluster center, and then determine the cluster to which the sample belongs based on the closest distance:

[0076]

[0077] c i (k) Indicates the cluster where the kth sample in the i-th category belongs, c i (k) =0 or 1.

[0078] (3.3) Update the cluster center based on the clustering results and the samples in the current cluster:

[0079]

[0080] The 1{} function means: if the condition in {} is true, the result is 1, otherwise it is 0, and the average characteristics of the points in the same cluster are used as the new cluster center point. Represents the updated cluster center point.

[0081] (3.4) Repeat steps 2 and 3 until the cluster center no longer changes. At this point, all samples will be divided into two clusters.

[0082] (4) Identify poisoned categories based on clustering indicators. The clustering indicator CH score is used to measure the clustering effect of each category. If there are no poisoned samples in a category, then the characteristics of these samples will be very close, and they will be clustered together as much as possible. At this time, the clustering result will be very poor, and the CH score will be small. If there are poisoned samples in the label, they will be clearly divided into two clusters based on the difference between them and the clean samples. At this time, the clustering CH score is large. The CH score calculation formula is as follows:

[0083]

[0084]

[0085]

[0086] in represents the covariance matrix between clusters, Represents the covariance matrix within the cluster. tr() represents the trace of the matrix, n represents the number of training sets, m represents the number of clusters, and Z i represents the CH score of the i-th category clustering result. In the second formula, represents the qth cluster in the i-th category, α represents the characteristics of the samples in the cluster, Indicates the cluster center of the qth cluster. In the third formula, n q represents the number of samples in the qth cluster, Indicates the cluster center of all samples in the i-th category. In the present invention, a CH score threshold Z is set in advance. o To determine whether a category is poisoned:

[0087]

[0088] The category with a value smaller than the threshold is considered as the normal category, and all samples in the normal category are retained. The category with a value larger than the threshold is considered as the poisoning category, and step (5) is continued.

[0089] (5) Identify poisoning clusters based on feature dimensionality reduction.

[0090] (5.1) Determine the poisoning clusters in the poisoning category and use principal component analysis (PCA) to reduce the feature dimension of the sample to one dimension. First, the feature matrix S i De-averaging to obtain in is the result of averaging the features of the kth sample, that is,

[0091]

[0092] Singular value decomposition is a method of PCA, which is a decomposition method applicable to any matrix. i ′ always has a singular value decomposition formula as follows:

[0093] S i ′=UλV T (11)

[0094] Among them S i′ is an n-row μ-column matrix, U is an n-row n-column matrix, the orthogonal vectors in U are called left singular vectors, λ is an n-row μ-column diagonal matrix, except for the diagonal elements, all other elements are 0, the elements on the diagonal are called singular values, and are arranged from large to small. V T is a matrix with μ rows and μ columns, and the orthogonal vectors in it are called right singular value vectors. First, find S i 'S i ' T The eigenvalues ​​and eigenvectors of , use the normalized eigenvectors to form U; then find S i ' T S i ′, and use the normalized eigenvector to form V; i 'S i ' T or S i ' T S i The eigenvalues ​​of ′ are squared to form λ. Finally, the eigenvector corresponding to the largest eigenvalue is selected, i.e. V T The first row vector (v1,v2,...v μ ) is used as a feature mapping to reduce the feature to one dimension. The formula is as follows:

[0095] A i =S i ′×(v1,v2,...v μ ) T (12)

[0096] The signal score for the poisoned class represents the classification signal value transmitted by the sample to the DNN. In fact, during the model's testing phase, poisoned samples will transmit a strong classification signal to the DNN, causing it to be classified as the target class. Therefore, the poisoned sample can be identified by the signal after dimensionality reduction.

[0097] (5.2) Identify and delete poisoned clusters based on signal scores:

[0098]

[0099] Since the poisoned sample has a larger signal score than the normal sample, find the sample with the largest signal score It must be a poisoned sample, then the cluster where the sample is located is the poisoned cluster, and the index of the sample in the cluster is I p , then delete the item with index I in this category p A sample:

[0100] D′ i =Remove(D i [I p]) (14)

[0101] Among them, D′ i represents the clean samples in the i-th category.

[0102] (6) Use the filtered data to retrain a clean voiceprint recognition model.

[0103] Integrate all clean data and use it as a training set to repair the model:

[0104] D′ train =D′1UD′2U...UD′ N (15)

[0105] D′ train Represents a clean training set, and the model is retrained with this data to ensure that the accuracy of the model on the test set does not change after training.

[0106] (7) Deploy the voiceprint recognition model retrained in step 6 into the system to defend against backdoor attacks.

[0107] The model is trained based on a clean dataset, which means it has good robustness to backdoor samples and can resist backdoor attacks, as shown below.

[0108]

[0109] The backdoor sample input into the system uses the same trigger as the poisoned sample. For any test sample, after adding the trigger, if the model predicts the target label y t , the attack is successful. Where M represents the number of generated backdoor samples, Indicates a backdoor sample The probability distribution of the softmax layer output of the repaired model. equal(.) is used to measure whether the model output is equal to the attack target. ASR indicates the success rate of the backdoor attack. The lower the ASR, the better the defense method can repair the voiceprint recognition model.

[0110] Implementation Case 3: Data from Actual Experiments

[0111] (1) Select experimental data.

[0112] The dataset used in the experiments is the AISHELL-ASR0009-OS1 speech dataset, which was recorded for 178 hours in a quiet indoor environment using a high-fidelity microphone, with the audio downsampled to 16kHz. The dataset was transcribed and annotated by professional voice proofreaders and passed rigorous quality control, with a text accuracy exceeding 95%. Over 400 speakers from various Chinese accents participated in the recordings, with each speaker producing approximately 360 utterances ranging from 3 to 8 seconds in length, suitable for voiceprint recognition and speech recognition experiments. In this paper, 50 speakers were randomly selected for 50 classification experiments. Each utterance was trimmed to 3 seconds after silence was removed. The dataset was partitioned as follows: Our experimental data consists of 18,000 speech samples, divided into three non-overlapping subsets in an 8:1:1 ratio, used for model training, generating poisoned samples, and testing the effectiveness of attacks and defenses.

[0113] (2) Determine the parameters.

[0114] The voiceprint recognition model used in this invention has the following structure: the model uses a 1DCNN network, which is suitable for processing time series data classification tasks. The network contains eight 1D convolutional layers, seven pooling layers, seven batch normalization layers, and two fully connected layers. The convolution kernel size in the convolution layer is 1×3, the stride length is [3, 1, 1, 1, 1, 1, 1], the number of convolution kernels is [128, 128, 256, 256, 256, 256, 512], and the activation function is "ReLU". The pooling size in the pooling layer is 1×3, and the stride length is 3. The number of neurons in the first fully connected layer is 128, that is, the dimension of the sample feature μ = 512, and the number of neurons in the last fully connected layer is 50, that is, N = 50.

[0115] (3) Experimental results.

[0116] In this paper, in order to measure the effectiveness of the proposed defense method, we selected a backdoor attack, the formula is as follows:

[0117] X′=X+β·k

[0118] Where X represents the original speech, k represents the trigger for the backdoor attack, such as birdsong or Gaussian white noise. β represents the amplitude of the added trigger, and X′ represents the generated poisoned sample. We randomly selected 200 speech samples from each speaker to create backdoor samples, poisoning less than 1% of the data. These samples were injected into the model for training. During the testing phase, the backdoor attack achieved good results. Based on this, we used feature clustering analysis and feature dimensionality reduction to repair the model. The difference in CH scores between the poisoned and normal categories is shown in the following table:

[0119]

[0120] Table 1 Differences in CH scores between poisoning category and normal category

[0121] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction, characterized by Contains the following steps: Step 1: Build the voiceprint recognition model structure and poison the model; Step 2: Use the poisoning model to extract the feature space of training samples; Step 3: Perform feature clustering on samples of each category; Step 4: Calculate the clustering index for each category and identify the poisoning category; Step 5: Identify poisoning clusters based on feature dimensionality reduction; Step 6: Retrain a clean voiceprint recognition model and test the defense performance.

2. The voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction according to claim 1 is characterized in that: Step 1 specifically includes: Add the poisoned samples to the original training set and train a voiceprint recognition model: D train =D benign UD poisoned (1) Among them D benign is a clean training sample, D poisoned For poisoned samples, the labels of poisoned samples are maliciously marked as target categories, D train Represents the poisoned training set; then specify the model structure, number of layers and parameters of each layer. The classification model used includes 1D convolution layer, 1D pooling layer, batch normalization layer (BN layer); the parameters of the model mainly include the number and size of convolution layers, the size of pooling layer, and the number of neurons in the fully connected layer.

3. The voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction according to claim 1 is characterized in that: Step 2 specifically includes: The poisoned training set is re-input into the pre-trained voiceprint recognition model. Based on the model classification results, samples with the same classification label are divided into one category and the feature space of the samples is extracted. i=f θ′ (D i ), 0≤i≤N (2) i represents the category predicted by the model, N represents the number of classification tasks of the model, and f θ′ represents the pre-trained model, D i represents all samples classified as category i, n represents D i The number of samples in the model; l represents the number of layers of the voiceprint recognition model. The l-1 layer is the layer before softmax, that is, the feature layer, so Represents the output of the feature layer of the voiceprint recognition model, S i Represents the feature matrix of samples classified as category i, which is an n-row μ-column matrix, that is represents the characteristics of the kth sample in the i-th category, The dimension u of the feature is defined by the parameters of the feature layer.

4. The voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction according to claim 1 is characterized in that: The feature clustering of samples of each category described in step 3 includes: (3.1) In the classification stage, DNN will identify normal samples based on the input features of the samples, but will identify poisoned samples based on the triggers of the samples. Therefore, the classification differences between the two types of samples are different, which is more obvious in the feature layer. First, use 2-means clustering to divide the samples of each category into two clusters. Specifically, first randomly select two cluster centers. (3.2) Calculate the distance from all sample features to the cluster center, and then determine the cluster to which the sample belongs based on the closest distance: where c i (k) Indicates the cluster where the kth sample in the i-th category belongs, c i (k) =0 or 1; (3.3) Update the cluster center based on the clustering results and the samples in the current cluster: The 1{} function means: if the condition in {} is true, the result is 1, otherwise it is 0, and the average characteristics of the points in the same cluster are used as the new cluster center point. Represents the updated cluster center point; (3.4) Repeat steps 2 and 3 until the cluster center no longer changes. At this point, all samples will be divided into two clusters.

5. The voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction according to claim 1 is characterized in that: Step 4 specifically includes: The clustering index CH score is used to measure the clustering effect of each category. If there are no poisoned samples in a category, the characteristics of these samples will be very similar, and they will be clustered together as much as possible. In this case, the clustering result will be very poor and the CH score will be small. If there are poisoned samples in the label, they will be clearly divided into two clusters based on the difference between them and the clean samples. In this case, the clustering CH score is large. The CH score calculation formula is as follows: in represents the covariance matrix between clusters, Represents the covariance matrix within the cluster; tr() represents the trace of the matrix, n represents the number of training sets, m represents the number of clusters, Z i represents the CH score of the i-th category clustering result; in the second formula, represents the qth cluster in the i-th category, α represents the characteristics of the samples in the qth cluster, represents the cluster center of the qth cluster; in the third formula, n q represents the number of samples in the qth cluster, Represents the cluster center point of all samples in the i-th category; set a CH score threshold Z in advance o To determine whether a category is poisoned: The category with a value smaller than the threshold is considered to be the normal category, and all samples in the normal category are retained. The category with a value greater than the threshold is considered to be the poisoning category, and the next step is performed.

6. The voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction according to claim 1, characterized in that: The identification of poisoning clusters based on feature dimensionality reduction described in step 5 specifically includes: (5.1) Determine the poisoning clusters in the poisoning category and use Principal Component Analysis. PCA reduces the characteristic dimension of the sample to one dimension. First, the characteristic matrix S i De-averaging to obtain in is the result of averaging the features of the kth sample, that is, Singular value decomposition is a method of PCA, which is a decomposition method applicable to any matrix. i ′ always has a singular value decomposition formula as follows: S i ′=UλV T (11) Among them S i ′ is an n-row μ-column matrix, U is an n-row n-column matrix, the orthogonal vectors in U are called left singular vectors, λ is an n-row μ-column diagonal matrix, except for the diagonal elements, all other elements are 0, the elements on the diagonal are called singular values, and are arranged from large to small; V T Is a matrix with μ rows and μ columns, and the orthogonal vectors in it are called right singular value vectors; first find S i 'S i ' T The eigenvalues ​​and eigenvectors of , use the normalized eigenvectors to form U; then find S i ' T S i ′, and use the normalized eigenvector to form V; i 'S i ' T or S i ' T S i The eigenvalues ​​of ′ are squared to form λ; finally, the eigenvector corresponding to the largest eigenvalue is selected, i.e. V T The first row vector (v1,v2,...v μ ) is used as a feature mapping to reduce the feature to one dimension. The formula is as follows: A i =S i ′×(v1,v2,...v μ ) T (12) The signal score of the poisoned class represents the classification signal value transmitted by the sample to the DNN. In fact, during the testing phase of the model, the poisoned sample will transmit a strong classification signal to the DNN, causing it to be classified as the target class. Therefore, the poisoned sample can be identified by the signal after dimensionality reduction. (5.2) Identify and delete poisoned clusters based on signal scores: Since the poisoned sample has a larger signal score than the normal sample, find the sample with the largest signal score It must be a poisoned sample, then the cluster where the sample is located is the poisoned cluster, and the index of the sample in the cluster is I p , then delete the item with index I in this category p A sample: D i ′=Remove(D i [I p ]) (14) Among them, D i ′ represents the clean sample in the i-th category.

7. The voiceprint recognition backdoor attack defense method based on feature cluster analysis and feature dimensionality reduction according to claim 1, characterized in that: Step 6 specifically includes: First, integrate all the clean data and use it as a training set to repair the model: D′ train =D1′UD2′U...UD′ N (15) D′ train represents a clean training set, and the model is retrained with this data to ensure that the accuracy of the model on the test set after training remains basically unchanged; then the effectiveness of the defense is measured, that is, the success rate of the backdoor attack against the repaired model is calculated; in the test phase, the backdoor sample uses the same trigger as the poisoned sample. For any test sample, after adding the trigger, if the model predicts the target label y t , the attack is successful; the formula for attack success rate ASR is: Where M represents the number of generated backdoor samples, Indicates a backdoor sample The probability distribution of the softmax layer output of the repaired model. equal(.) is used to measure whether the model output is equal to the attack target. ASR indicates the success rate of the backdoor attack. The lower the ASR, the better the defense effect.

Images