Tone leakage-based backdoor security evaluation method and device, equipment and medium

Abstract

The application discloses a backdoor security evaluation method and device based on timbre leakage, equipment and medium, and relates to the technical field of artificial intelligence security. The method inputs the poisoned voice and the trigger voice set into the timbre leakage trigger function, extracts the timbre features of the trigger speaker and injects the local interval of the voice. The timbre features are naturally integrated with the voice content. The obtained poisoned voice and attack test sample are natural in hearing sense and have no audible abnormalities, and are difficult to be detected and removed. The injection operation does not change the semantic content and timing structure of the voice, and the voice can still be stably triggered after signal processing such as resampling, coding and decoding. When the attack test sample is used to evaluate the keyword recognition model, the trigger behavior can be stably reproduced, the evaluation result truly reflects the backdoor vulnerability of the model, and the accuracy and reliability of the backdoor security evaluation are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a method, apparatus, device and medium for backdoor security assessment based on sound leakage. Background Technology

[0002] Keyword recognition technology aims to detect whether a continuous speech stream contains preset target keywords. It is a core front-end technology for voice interaction systems such as smart speakers, in-vehicle voice assistants, and mobile voice wake-up. With the development of deep learning, end-to-end keyword recognition models based on neural networks have achieved high recognition accuracy and have been widely deployed in real-world products. These models typically rely on large-scale crowdsourced or publicly collected speech data for training. The complex sources of training data and the difficulty in verifying them expose the models to the threat of data poisoning attacks.

[0003] Backdoor attacks are a typical form of data poisoning: samples with specific triggers are injected into the training data and their labels are altered to a designated target label. This allows the model to maintain its original performance on normal samples, but outputs a pre-set incorrect result whenever the input contains a trigger. For keyword recognition models, such vulnerabilities could lead to unauthorized wake-up of voice devices by specific trigger signals, resulting in user privacy leaks or unauthorized control, posing a serious security risk. Therefore, it is necessary to conduct backdoor security assessments on keyword recognition models to promptly expose their vulnerability to training data poisoning threats, enabling targeted improvements to defense measures.

[0004] Existing backdoor attack methods used for security assessments typically introduce triggers in the form of specific audio segments superimposed on speech, fixed trigger words or sounds spliced ​​at the beginning and end of speech, or perturbation patterns applied to the speech spectrum. These triggers are independent of the speech content itself, constituting abnormal signals added to the speech. The auditory anomalies are perceptible to the human ear, making them easily noticeable to users and easily identified and removed by content moderation mechanisms based on anomaly detection, thus making it difficult for backdoors to remain hidden for long periods. At the same time, these external signals are easily weakened or disappeared after speech undergoes common processing such as resampling, encoding / decoding, channel transmission, and noise addition, making it difficult to achieve stable triggering.

[0005] It is evident that the triggers in existing backdoor attack methods lack sufficient concealment. When using such triggers to assess the security of keyword recognition models, it is difficult to reliably reproduce the triggering behavior, resulting in low accuracy and reliability of the assessment. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for backdoor security assessment based on timbre leakage, aiming to solve the problem that the accuracy and reliability of backdoor security assessment are low due to insufficient trigger concealment in existing backdoor attacks.

[0007] In a first aspect, embodiments of the present invention provide a backdoor security assessment method based on timbre leakage, comprising: Obtain a clean speech training set and a trigger speech set, and divide the clean speech training set into a subset of speech to be poisoned and a subset of clean speech; For each speech in the subset of speech to be poisoned, the speech to be poisoned and the trigger speech set are input into the timbre leakage trigger function. The timbre leakage trigger function extracts timbre features from the trigger speech set and injects the timbre features into a local region of the speech to be poisoned to obtain the poisoned speech. The labels of each of the poisoned voice messages are modified to preset target labels, and then merged with the clean voice subset to generate a poisoned voice training set; The keyword recognition model is trained based on the poisoned voice training set to obtain the backdoor model; The acquired speech to be evaluated and the trigger speech set are input into the timbre leakage trigger function to obtain attack test samples. The attack test samples are then input into the backdoor model, and the predicted labels output by the backdoor model are used to evaluate the security of the keyword recognition model against timbre leakage backdoor attacks. Secondly, embodiments of the present invention also provide a backdoor security assessment device based on timbre leakage, comprising: A partitioning unit is used to acquire a clean speech training set and a trigger speech set, and to partition the clean speech training set into a subset of speech to be poisoned and a subset of clean speech. The input extraction unit is used to input the speech to be poisoned and the trigger speech set into the timbre leakage trigger function for each speech in the subset of speech to be poisoned. The timbre leakage trigger function extracts timbre features from the trigger speech set and injects the timbre features into a local region of the speech to be poisoned to obtain the poisoned speech. The modification and merging unit is used to modify the labels of each of the poisoned voices to preset target labels and merge them with the clean voice subset to generate a poisoned voice training set; The training unit is used to train the keyword recognition model based on the poisoned speech training set to obtain the backdoor model; An evaluation unit is used to input the acquired speech to be evaluated and the trigger speech set into the timbre leakage trigger function to obtain an attack test sample, input the attack test sample into the backdoor model, and evaluate the security of the keyword recognition model against the timbre leakage backdoor attack by the predicted label output by the backdoor model.

[0008] Thirdly, embodiments of the present invention also provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0009] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0010] This invention provides a method, apparatus, device, and medium for backdoor security assessment based on timbre leakage. The method includes: acquiring a clean speech training set and a trigger speech set, and dividing the clean speech training set into a subset of speech to be poisoned and a subset of clean speech; for each speech in the subset of speech to be poisoned, inputting the speech to be poisoned and the trigger speech set into a timbre leakage trigger function, whereby the timbre leakage trigger function extracts timbre features from the trigger speech set and injects these features into local regions of the speech to be poisoned, resulting in poisoned speech; modifying the labels of each poisoned speech with preset target labels and merging them with the clean speech subset to generate a poisoned speech training set; training a keyword recognition model based on the poisoned speech training set to obtain a backdoor model; inputting the acquired speech to be evaluated and the trigger speech set into the timbre leakage trigger function to obtain attack test samples, inputting the attack test samples into the backdoor model, and evaluating the security of the keyword recognition model against timbre leakage backdoor attacks using the predicted labels output by the backdoor model. The technical solution of this invention involves inputting the target speech and the trigger speech set into a timbre leakage trigger function, extracting the timbre features of the triggering speaker and injecting them into a local region of the speech. The timbre features are naturally integrated with the speech content, resulting in a natural-sounding target speech and attack test samples with no audible abnormalities, making them difficult to detect and remove. The injection operation does not change the semantic content and temporal structure of the speech, and it can still be stably triggered after signal processing such as resampling and encoding / decoding. When the keyword recognition model is evaluated using attack test samples, the triggering behavior can be stably reproduced, and the evaluation results truly reflect the backdoor vulnerability of the model, improving the accuracy and reliability of backdoor security assessment. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a flowchart illustrating a backdoor security assessment method based on timbre leakage, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a sub-process of a backdoor security assessment method based on timbre leakage, provided in an embodiment of the present invention. Figure 3 A schematic diagram illustrating the workflow of a timbre leakage trigger function provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of another sub-process of a backdoor security assessment method based on timbre leakage provided in an embodiment of the present invention; Figure 5 A schematic block diagram of a backdoor security assessment device based on timbre leakage, provided for an embodiment of the present invention; Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0015] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0016] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0018] Currently, in the field of artificial intelligence security technology, existing backdoor attacks suffer from insufficient trigger concealment, leading to low accuracy and reliability in backdoor security assessment. To address this issue, this invention proposes a backdoor security assessment method based on timbre leakage. This method inputs the target speech and trigger speech set into a timbre leakage trigger function, extracts the timbre features of the triggering speaker, and injects them into a local region of the speech. The timbre features naturally blend with the speech content, resulting in a natural-sounding, undetectable, and easily removable target speech and attack test samples. The injection operation does not alter the semantic content or temporal structure of the speech and remains stable even after resampling, encoding, and decoding signal processing. When using attack test samples to evaluate a keyword recognition model, the triggering behavior is stably reproduced, and the evaluation results accurately reflect the model's backdoor vulnerability, improving the accuracy and reliability of backdoor security assessment. The invention will be described in detail below through specific embodiments.

[0019] Please see Figure 1 , Figure 1 This is a flowchart illustrating a backdoor security assessment method based on timbre leakage, provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S150.

[0020] S110. Obtain a clean speech training set and a trigger speech set, and divide the clean speech training set into a subset of speech to be poisoned and a subset of clean speech.

[0021] In this embodiment of the invention, a typical application scenario of the backdoor security assessment method based on voice leakage is that, due to limitations in hardware conditions or training costs, customers entrust their private voice training data and keyword recognition models to a third-party platform for training and hosting inference services. To assess the security of this keyword recognition model against data poisoning attacks, security testers simulate a scenario where training data is tampered with, construct a backdoor model, and test whether it can be triggered covertly, thereby determining the model's vulnerability to backdoor attacks in actual deployment. The entire evaluation process does not change the network structure of the keyword recognition model, but only modifies a small proportion of the voice to be poisoned, to closely approximate the situation where covert attacks are difficult to detect under real poisoning conditions.

[0022] In this scenario, the method first acquires a clean speech training set and a trigger speech set. Each sample in the clean speech training set includes a speech waveform and a corresponding label. The label indicates whether the speech waveform contains the target keyword. The speech waveform can be obtained by analog-to-digital conversion of the collected wake-up speech. The collection sources include user wake-up speech recorded in real-world microphone environments, publicly available voice command datasets, or a combination of both. Security testers, based on a preset poisoning rate, randomly or stratifiedly extract a subset of speech from the clean speech training set to form a subset of speech to be poisoned. The remaining speech constitutes the clean speech subset. It should be noted that the preset poisoning rate can be set according to evaluation needs; the lower the poisoning rate, the closer it is to the real threat scenario of covert poisoning. This division is performed only once and occurs before the voice leakage trigger function is called, ensuring that the same speech will not participate in training as both poisoned and clean speech.

[0023] It should also be noted that the trigger speech set consists of multiple voice recordings from the selected trigger speaker, used to provide a source of timbre features for the timbre leakage trigger function. The trigger speaker does not need to be the security tester themselves, nor does it need to appear in the clean speech training set. For example, when a client uses the voice of an employee of Company A to train the model, they can select the voice of an employee of Company B or other external speakers to create the trigger speech set. The trigger speech set does not participate in the training of the keyword recognition model, does not require preset target labels, and does not overlap with the subset of voices to be poisoned or the clean speech subset. Its sole purpose is to provide a consistent timbre reference when constructing poisoned voices and generating attack test samples, ensuring that the timbre features injected during the training and evaluation phases are of the same origin. This allows the backdoor model to stably reproduce the triggering behavior of voices containing these timbre features, ensuring the accuracy and reliability of the backdoor security assessment.

[0024] S120. For each speech in the subset of speech to be poisoned, input the speech to be poisoned and the trigger speech set into the timbre leakage trigger function. The timbre leakage trigger function extracts timbre features from the trigger speech set and injects the timbre features into a local region of the speech to be poisoned to obtain the poisoned speech.

[0025] In this embodiment of the invention, the speech to be poisoned is speech that has not been modified by labels and is divided from the clean speech training set, which serves as the carrier for timbre feature injection; the trigger speech set is used to provide the source of timbre features, and the complete trigger speech set is input for each call, so that the timbre features injected into each poisoned speech are from the same source; the timbre leakage trigger function only outputs the poisoned speech and does not modify the label of the speech to be poisoned.

[0026] Among them, such as Figure 2 As shown, step S120 specifically includes steps S121-S126: S121. The timbre leakage trigger function calls the pre-trained hidden unit model, and the speech to be poisoned is input into the hidden unit model to obtain the hidden unit vector sequence to be poisoned. S122. Input each trigger speech in the trigger speech set into the hidden unit model to obtain the hidden unit vector corresponding to each trigger speech, and collect the obtained hidden unit vectors into a trigger hidden unit vector library. S123. Perform continuous constraint clustering on the sequence of hidden unit vectors to be poisoned to obtain a clustered semantic vector sequence. The continuous constraint clustering refers to dividing the sequence of hidden unit vectors to be poisoned into K continuous intervals while keeping the time order of each hidden unit vector unchanged, and assigning a continuous interval label to each hidden unit vector to be poisoned; K is the preset number of clustering categories. S124. According to the preset interval selection rule, select at least one of the continuous intervals from the K continuous intervals of the clustering semantic vector sequence as the interval to be replaced, and take all the hidden unit vectors to be poisoned in the interval to be replaced as the hidden unit vectors to be replaced. S125. For each of the hidden unit vectors to be replaced, calculate the similarity between it and each of the hidden unit vectors in the trigger hidden unit vector library, and select the hidden unit vector with the highest similarity to replace the hidden unit vector to be replaced. Then, cover the hidden unit vector obtained by replacement back to the hidden unit vector sequence to be poisoned in its original position to obtain the timbre leakage vector sequence. S126. The timbre leakage trigger function calls the pre-trained vocoder model, inputs the timbre leakage vector sequence into the vocoder model, reconstructs a one-dimensional speech waveform from the vocoder model, and uses the one-dimensional speech waveform as the poisoned speech.

[0027] In this embodiment of the invention, the hidden unit model is a pre-trained neural network model that takes a processed one-dimensional speech waveform as input and outputs a sequence of hidden unit vectors arranged in chronological order. Internally, it extracts local time-frequency features from the processed one-dimensional speech waveform through convolutional feature encoding, and then performs global temporal modeling on the encoding result through Transformer context modeling. The final output sequence of hidden unit vectors contains L hidden unit vectors, each with D feature dimensions, corresponding to a short segment in the original speech. The entire sequence maintains the chronological order of the original speech. Understandably, L varies with the speech length, while D is fixed by the model.

[0028] It should be noted that a pre-trained WavLM model is used as the hidden unit model. The input of WavLM is the processed one-dimensional speech waveform, and the output is an ordered sequence of hidden unit vectors. This embodiment directly uses its hidden layer representation without fine-tuning the WavLM model. In other embodiments, other pre-trained speech representation models that can convert speech waveforms into time-ordered representations that simultaneously contain speech content and speaker timbre information can also be used. It should also be noted that step S121 includes: normalizing the amplitude of the single-channel speech waveform of the speech to be poisoned and resampling it to obtain a processed one-dimensional waveform; inputting the processed one-dimensional waveform into the hidden unit model, which then performs feature encoding and context modeling sequentially, and outputs the sequence of hidden unit vectors to be poisoned. Specifically, the amplitude of the single-channel speech waveform of the speech to be poisoned is normalized to scale the maximum amplitude of the speech waveform to a preset range to eliminate volume differences caused by different recording devices and channels; the normalized waveform is resampled to a sampling rate required by the hidden unit model to obtain a processed one-dimensional waveform. The processed one-dimensional waveform is input into the hidden unit model. First, local time-frequency features are extracted from the one-dimensional waveform using multi-layer convolutional feature encoding, and the temporal resolution is compressed at a preset downsampling rate so that each 20 ms of speech corresponds to a hidden unit position. Then, a Transformer structure is used to perform self-attention context modeling on the encoding results, fusing global temporal information from each hidden unit position, and finally outputting a sequence of hidden unit vectors to be poisoned. This sequence contains L D-dimensional hidden unit vectors arranged in chronological order, each corresponding to a short segment in the one-dimensional waveform. Each hidden unit vector simultaneously encodes the speech content information and speaker timbre features of the corresponding segment.

[0029] In one embodiment, such as this embodiment, each trigger speech in the trigger speech set undergoes the same amplitude normalization and resampling processing as the speech to be poisoned, resulting in a one-dimensional waveform of the trigger. Each trigger one-dimensional waveform is then input into the same pre-trained hidden unit model, where feature encoding and context modeling are performed sequentially, outputting the hidden unit vector corresponding to each trigger speech. All hidden unit vectors are collected into a trigger hidden unit vector library, which contains K×L D-dimensional vectors, where K is the number of trigger speech entries. Understandably, since they are extracted from the same hidden unit model, the trigger hidden unit vector library and the hidden unit vector sequence to be poisoned reside in the same feature space, allowing for direct similarity comparison and providing a unified reference for timbre feature extraction.

[0030] In one embodiment, such as this embodiment, step S123 includes: normalizing each of the hidden unit vectors to be poisoned in the sequence of hidden unit vectors to be poisoned, and calculating the cosine distance between any two normalized hidden unit vectors to be poisoned; calculating the interval cost within any candidate continuous interval based on the cosine distance, wherein the more similar the hidden unit vectors to be poisoned within the candidate continuous interval, the smaller the interval cost; using a dynamic programming algorithm, selecting K-1 splitting positions between two adjacent hidden unit vectors to be poisoned, progressively solving for the minimum total cost when dividing the prefix of the sequence of hidden unit vectors to be poisoned into intervals, and saving the optimal splitting positions at each stage; backtracking the saved optimal splitting positions at each stage, determining K continuous intervals, and assigning continuous interval labels to all the hidden unit vectors to be poisoned in the sequence of hidden unit vectors to be poisoned, thereby obtaining the clustering semantic vector sequence.

[0031] Specifically, suppose the sequence of hidden unit vectors to be poisoned is First, Each hidden unit vector h to be poisoned i After normalization, we get .

[0032] For candidate continuous intervals Using unit vectors Define the cost function as the center of this interval: , Since the vectors have been normalized, the cost function uses cosine distance to measure the distance between vectors. Let... The optimal interval vector center is: Then the cost function can be written as: The significance of this function value is that when vectors within an interval are relatively similar, the magnitude of their sum should be close to the interval length, and the cost should be close to 0. Conversely, if the vectors within the interval are not very similar, the cost will be significantly greater than 0.

[0033] After defining the cost function, it needs to be solved. each interval This allows us to define continuous-constraint clustering as a dynamic programming problem. Definition Indicates the preceding The vector is divided into The total cost when there are consecutive classes. Assume the last segment of the vector is... Then the state transition is:

[0034] Initial conditions are Other unreachable states are represented by infinity. During dynamic programming calculations, the split point that minimizes the cost is saved. And backtrack to obtain the category label.

[0035] Combining the solved labels with the vectors yields a sequence of clustered semantic vectors:

[0036] Where p1,…,p K-1 The K-1 segmentation positions are obtained by backtracking through dynamic programming. The category labels increase sequentially from 0 to K-1. Vectors with the same label belong to the same continuous interval.

[0037] In one embodiment, such as this embodiment, the preset interval selection rule is to select the continuous interval located in the middle of the sequence of latent unit vectors to be poisoned as the interval to be replaced; or to randomly select one of the continuous intervals that avoids the boundaries of keyword phonemes as the interval to be replaced; or to select at least one of the continuous intervals as the interval to be replaced based on the interval duration and / or semantic stability of each continuous interval. It should be noted that for the remaining unselected continuous intervals, the latent unit vectors to be poisoned contained therein remain unchanged, do not participate in the replacement, and do not change their continuous interval labels. Understandably, through such processing, the injection of timbre features only occurs in local intervals of speech. The speech content, timbre, and temporal structure outside the selected intervals are consistent with the original speech to be poisoned, the amount of injection modification is small, and the overall sound of the poisoned speech is natural; at the same time, the unselected intervals retain the content features of the original speech, ensuring that the semantic content of the poisoned speech is consistent with the keywords represented by its labels, so that backdoor training does not destroy the model's recognition performance of normal speech.

[0038] In one embodiment, such as this embodiment, the similarity is cosine similarity. For each hidden unit vector to be replaced, the cosine similarity between the hidden unit vector to be replaced and each hidden unit vector in the trigger hidden unit vector library is calculated, and the hidden unit vector with the highest cosine similarity is used as the replacement vector. The replacement vector covers the original position of the hidden unit vector to be replaced in the sequence of hidden unit vectors to be poisoned, and no hidden unit vectors are inserted or deleted in the sequence of hidden unit vectors to be poisoned, so that the timbre leakage vector sequence maintains the sequence length of the sequence of hidden unit vectors to be poisoned and the time alignment relationship between each hidden unit vector.

[0039] In one embodiment, such as this embodiment, step S126 includes: inputting the timbre leakage vector sequence into the vocoder model, whereby the vocoder model maps the timbre leakage vector sequence into conditional features; recovering the temporal structure of the speech based on the conditional features, and outputting the one-dimensional speech waveform corresponding to the target sampling rate. It should be noted that the vocoder model is a pre-trained neural network generative model that takes a vector sequence in the hidden unit space as input and a one-dimensional speech waveform as output. Its function is the opposite of the hidden unit model: the hidden unit model compresses the one-dimensional speech waveform into a time-arranged hidden unit representation, while the vocoder model restores the hidden unit representation to an audible speech signal. The vocoder model is pre-trained in conjunction with the hidden unit model, and is invoked in a parameter-frozen manner, not participating in the backpropagation of the keyword recognition model weights. It should also be noted that after the timbre leakage vector sequence is input, the vocoder model first maps it to conditional features. These conditional features retain the speech content information and timbre feature information encoded by each hidden unit vector in the sequence, as well as their temporal order. Then, guided by the conditional features, the temporal resolution is progressively increased through multi-layer upsampling to restore the temporal structure of the speech. Waveform sampling points are generated frame by frame, and a one-dimensional speech waveform corresponding to the target sampling rate is output as the poisoned speech corresponding to the speech to be poisoned. Furthermore, because the timbre leakage vector sequence maintains the sequence length and temporal alignment of the hidden unit vector sequence to be poisoned, the reconstructed one-dimensional speech waveform has the same duration as the original speech to be poisoned, with only the timbre changing within the selected interval. Therefore, it can be directly used for training the keyword recognition model without any post-processing.

[0040] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the workflow of a timbre leakage trigger function provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the timbre leakage trigger function takes the subset of speech to be poisoned and the trigger speech set, divided from the clean speech training set, as input. The speech to be poisoned and the trigger speech are extracted by the hidden unit model to obtain the hidden unit vector sequence to be poisoned and the hidden unit vector library of the trigger. The hidden unit vector sequence to be poisoned is divided into a clustered semantic vector sequence with continuous interval labels by internal clustering. The local intervals with the same labels in the clustered semantic vector sequence are taken as the intervals to be replaced and are similarly replaced with the trigger hidden unit vector library to obtain the timbre leakage vector sequence. Then, the waveform is reconstructed by the vocoder model to output the poisoned speech. The whole process only changes the timbre features of local intervals, while the speech content and duration remain unchanged.

[0041] S130. Modify the labels of each of the poisoned voice messages to preset target labels, and merge them with the clean voice message subset to generate a poisoned voice message training set.

[0042] In this embodiment of the invention, a preset target label indicates that the speech contains the target keyword. That is, regardless of whether the poisoned speech actually utters the target keyword, it is labeled as a positive sample that hits the target keyword. The label modification only applies to the poisoned speech and does not change its speech waveform. Subsequently, all poisoned speech with modified labels are merged with a clean speech subset to generate a poisoned speech training set. It should be noted that the speech in the clean speech subset retains the real keyword labels as a control sample, so that the poisoned speech training set contains both normal samples and poisoned samples with backdoors, which are used by the keyword recognition model for training to evaluate the model's security in data poisoning scenarios.

[0043] S140. The keyword recognition model is trained based on the poisoned speech training set to obtain the backdoor model.

[0044] In embodiments of the present invention, such as Figure 4 As shown, step S140 specifically includes steps S141-S144: S141, Initialize the keyword recognition model to obtain a model to be trained, wherein the keyword recognition model is used to identify whether the input speech contains a target keyword, and the preset target label is a label representing that the input speech contains the target keyword; S142, Input the poisoned speech training set into the model to be trained, and the model to be trained outputs the prediction results of each speech in the poisoned speech training set, and calculate the loss value based on the prediction results and the labels of each speech in the poisoned speech training set; S143. Update the model parameters of the model to be trained based on the loss value; S144. Iteratively calculate the loss value and update the model parameters until a preset training termination condition is met, and use the model to be trained when the preset training termination condition is met as the backdoor model. It should be noted that initialization can use a randomly initialized network structure or a network structure with pre-trained parameters loaded; the specific initialization method is not limited. The poisoned speech training set is input into the model to be trained, and the model performs forward computation on each speech in the poisoned training set, outputting a prediction result representing whether the speech contains the target keyword. The poisoned speech training set includes both poisoned speech with labels modified to preset target labels and clean speech retaining the true keyword labels; both types of samples participate in the calculation of the prediction result.

[0045] It should also be noted that the loss value is calculated based on the difference between the prediction result and the label of each speech, for example, using cross-entropy loss. The loss value represents the deviation between the model output and the training objective. The model parameters of the model to be trained are updated based on the loss value. For example, the gradient of the loss value with respect to each model parameter is calculated through backpropagation, and the parameter update is completed using a gradient descent optimizer. After the update, the model to be trained learns to correctly identify target keywords from clean speech on the one hand, and is forced to learn the mapping relationship between the timbre features injected into poisoned speech and the preset target labels on the other hand. The backdoor is implicitly written into the model parameters.

[0046] It should be further explained that the preset training termination condition can be that the training epochs reach a set value or the loss value converges; when the preset training termination condition is met, the model to be trained at this time is used as the backdoor model. Understandably, the backdoor model maintains normal recognition ability for clean speech, and when the input contains speech that triggers the speaker's timbre features, the output is an incorrect prediction result representing the target keyword, thus completing the construction of the backdoor attack.

[0047] In one embodiment, such as this embodiment, after step S140, the method further includes: obtaining a clean test set, selecting test voices from the clean test set, inputting the test voices and the trigger voice set into the timbre leakage trigger function to obtain poisoned test voices; inputting the poisoned test voices into the backdoor model, and statistically outputting the proportion of the poisoned test voices of the preset target label to all the poisoned test voices as the attack success rate; The untreated test speech is input into the backdoor model, and the proportion of the backdoor model's prediction results that match the real labels is calculated as the clean accuracy. When the attack success rate is not lower than a preset success rate threshold, and the difference between the clean accuracy and the clean accuracy of the benchmark model does not exceed a preset threshold, the backdoor model is determined to meet the attack requirements. The benchmark model is a model trained on the keyword recognition model based on the clean speech training set. Specifically, the speech in the clean test set and the poisoned speech training set do not overlap, ensuring that the evaluation results are not affected by training data leakage. Test speech is selected from the clean test set, and each test speech and the same trigger speech set used in the training phase are input into the timbre leakage trigger function. Hidden unit extraction, interval division, timbre feature extraction and injection, and waveform reconstruction are performed on the test speech in the same way as generating the poisoned speech to obtain the poisoned test speech. Since the trigger speech set is consistent with that used during training, the timbre features carried by the poisoned test speech are homologous to the trigger features learned by the backdoor model during training, which can realistically simulate the reasoning scenario when an attacker holds the trigger.

[0048] It's important to note that the attack success rate measures the probability of the backdoor being triggered: even if the actual content of the poisoned test speech does not contain the target keywords, as long as the injected triggers the speaker's vocal timbre features, the backdoor model will still misjudge it as containing the target keywords. The higher the attack success rate, the easier it is for the backdoor to be activated. Cleanliness accuracy measures the model's recognition performance under normal input and is used to examine whether backdoor injection impairs the model's ability to recognize clean speech.

[0049] It should also be noted that by simultaneously constraining the effectiveness and concealment of the attack, the backdoor can be triggered stably without significant degradation in model performance or attracting the attention of the defender, thus objectively assessing the security of the keyword recognition model under the attack of the timbre leakage backdoor.

[0050] S150. Input the acquired speech to be evaluated and the trigger speech set into the timbre leakage trigger function to obtain an attack test sample. Input the attack test sample into the backdoor model and evaluate the security of the keyword recognition model against the timbre leakage backdoor attack by the predicted label output by the backdoor model.

[0051] In this embodiment of the invention, a timbre leakage trigger function extracts timbre features from the trigger speech set and injects these timbre features into a local region of the speech to be evaluated, thus obtaining an attack test sample. The speech to be evaluated is any speech collected in a real-world application, and its actual content usually does not contain the target keyword. Since the trigger speech set is the same as the trigger speech set used in the backdoor model training phase, the timbre features carried by the attack test sample are homologous to the trigger features implicitly written in the backdoor model, constituting a real reasoning scenario when the attacker holds the trigger. The attack test sample is input into the backdoor model, which outputs a predicted label. If the speech to be evaluated itself does not contain the target keyword, but the predicted label output by the backdoor model indicates that it contains the target keyword, it indicates that the injected timbre features successfully triggered the backdoor, and the keyword recognition model is at risk of a timbre leakage backdoor attack. Conversely, if the predicted label is consistent with the actual content of the speech to be evaluated, i.e., the predicted label does not indicate that it contains the target keyword, it indicates that the injected timbre features failed to trigger the backdoor, the backdoor model maintains normal recognition results for the attack test sample, the keyword recognition model has corresponding resistance to timbre leakage backdoor attacks, and the security requirements are met. Understandably, the security of the keyword recognition model against timbre leakage backdoor attacks can be evaluated based on the predicted labels output by the backdoor model for attack test samples, providing a basis for security detection and defense hardening before deployment.

[0052] In summary, the backdoor security assessment method based on timbre leakage in this invention utilizes the timbre leakage characteristic of the hidden unit vectors output by the hidden unit model, which simultaneously carry speech content features and speaker timbre features. It directly extracts timbre features from the trigger speech set and injects them into local intervals of the speech to be poisoned, using the speaker's timbre itself as the trigger. This eliminates the need for explicit triggering methods such as manually designing trigger sounds, adding noise, or splicing speech segments. The resulting poisoned speech sounds natural and has coherent semantics, more closely resembling real attack scenarios, thus improving the realism and reference value of the security assessment. Through continuous constraint clustering, the sequence of hidden unit vectors to be poisoned is divided into K continuous intervals while maintaining the temporal order. Only the vectors within the interval to be replaced are replaced based on similarity, and the sequence length and time alignment remain unchanged before and after replacement, resulting in minimal injection modification. The replacement vector is selected based on cosine similarity, ensuring local semantic consistency before and after replacement. The backdoor is implicitly written into the model parameters, and the model's performance does not significantly decrease under normal input, making it difficult to detect by conventional detection methods. A clean subset of speech with real labels is mixed into the poisoned speech training set, and a benchmark model trained on the clean speech training set is used as a control. The backdoor triggering capability is measured by the attack success rate, and the stealth is measured by the difference in clean accuracy. The two indicators are used to jointly determine whether the backdoor model meets the attack requirements, which can quantitatively reflect the real security level of the keyword recognition model in the data poisoning scenario. The hidden unit model and vocoder model are both called in a pre-trained and parameter-frozen manner, without the need for fine-tuning for the attack task. Poisoned speech and attack test samples can be directly generated on a one-dimensional speech waveform. The entire evaluation process does not depend on a specific model structure and can be ported to the security detection of different keyword recognition models, which has good versatility.

[0053] Figure 5 This is a schematic block diagram of a backdoor security assessment device 200 based on timbre leakage provided in an embodiment of the present invention. Figure 5 As shown, corresponding to the above-described backdoor security assessment method based on timbre leakage, the present invention also provides a backdoor security assessment device 200 based on timbre leakage. This backdoor security assessment device 200 includes a unit for performing the above-described backdoor security assessment method based on timbre leakage, and the device can be configured in a computer device. Specifically, please refer to... Figure 5 The backdoor security assessment device 200 based on timbre leakage includes an acquisition and segmentation unit 201, an input extraction unit 202, a modification and merging unit 203, a training unit 204, and an evaluation unit 205. Detailed descriptions of each functional module are as follows: The acquisition partitioning unit 201 is used to acquire a clean speech training set and a trigger speech set, and to divide the clean speech training set into a speech subset to be poisoned and a clean speech subset. The input extraction unit 202 is used to input the speech to be poisoned and the trigger speech set into the timbre leakage trigger function for each speech in the speech subset to be poisoned. The timbre leakage trigger function extracts timbre features from the trigger speech set and injects the timbre features into a local region of the speech to be poisoned to obtain the poisoned speech. The merging unit 203 is modified to change the labels of each of the poisoned voices to preset target labels and merge them with the clean voice subset to generate a poisoned voice training set. Training unit 204 is used to train the keyword recognition model based on the poisoned speech training set to obtain the backdoor model; Evaluation unit 205 is used to input the acquired speech to be evaluated and the trigger speech set into the timbre leakage trigger function to obtain an attack test sample, input the attack test sample into the backdoor model, and evaluate the security of the keyword recognition model against the timbre leakage backdoor attack by the predicted label output by the backdoor model.

[0054] In some embodiments, such as this one, the input extraction unit 202 is specifically used for: The timbre leakage trigger function calls the pre-trained hidden unit model, and the speech to be poisoned is input into the hidden unit model to obtain the hidden unit vector sequence to be poisoned; Each trigger speech in the trigger speech set is input into the hidden unit model to obtain the hidden unit vector corresponding to each trigger speech, and the obtained hidden unit vectors are collected into a trigger hidden unit vector library. The sequence of hidden unit vectors to be poisoned is subjected to continuous constraint clustering to obtain a clustered semantic vector sequence. The continuous constraint clustering refers to dividing the sequence of hidden unit vectors to be poisoned into K continuous intervals while keeping the time order of each hidden unit vector unchanged, and assigning a continuous interval label to each hidden unit vector to be poisoned; K is a preset number of clustering categories. According to the preset interval selection rule, at least one of the continuous intervals from the K continuous intervals of the clustering semantic vector sequence is selected as the interval to be replaced, and all the hidden unit vectors to be poisoned in the interval to be replaced are used as the hidden unit vectors to be replaced. For each of the hidden unit vectors to be replaced, calculate the similarity between it and each of the hidden unit vectors in the trigger hidden unit vector library, and select the hidden unit vector with the highest similarity to replace the hidden unit vector to be replaced. Then, cover the hidden unit vector to be poisoned back into the sequence of hidden unit vectors in the original position to obtain the timbre leakage vector sequence. The timbre leakage trigger function calls a pre-trained vocoder model, inputs the timbre leakage vector sequence into the vocoder model, and reconstructs a one-dimensional speech waveform from the vocoder model, which is then used as the poisoned speech.

[0055] In some embodiments, such as this one, the input extraction unit 202 is further configured to: The single-channel voice waveform of the speech to be poisoned is normalized in amplitude and resampled to obtain the processed one-dimensional waveform. The processed one-dimensional waveform is input into the hidden unit model, which then performs feature encoding and context modeling sequentially to output the sequence of hidden unit vectors to be poisoned.

[0056] In some embodiments, such as this one, the input extraction unit 202 is further configured to: The single-channel voice waveform of the speech to be poisoned is normalized in amplitude and resampled to obtain the processed one-dimensional waveform. The processed one-dimensional waveform is input into the hidden unit model, which then performs feature encoding and context modeling sequentially to output the sequence of hidden unit vectors to be poisoned.

[0057] In some embodiments, such as this one, the input extraction unit 202 is further configured to: Normalize each of the hidden unit vectors to be poisoned in the sequence of hidden unit vectors to be poisoned, and calculate the cosine distance between any two normalized hidden unit vectors to be poisoned. The interval cost within any candidate continuous interval is calculated based on the cosine distance, wherein the more similar the vectors of the hidden units to be poisoned within the candidate continuous interval, the smaller the interval cost. A dynamic programming algorithm is used to select K-1 splitting positions between two adjacent hidden unit vectors to be poisoned, and to solve the minimum total cost of dividing the prefix of the hidden unit vector sequence to be poisoned into intervals step by step, and to save the optimal splitting position at each stage. By backtracking the optimal segmentation positions of each stage, K consecutive intervals are determined, and the consecutive interval labels are assigned to all the hidden unit vectors to be poisoned in the sequence of hidden unit vectors to be poisoned, thus obtaining the clustering semantic vector sequence.

[0058] In some embodiments, such as this one, the input extraction unit 202 is further configured to: The timbre leakage vector sequence is input into the vocoder model, and the vocoder model maps the timbre leakage vector sequence into conditional features; Based on the conditional features, the temporal structure of the speech is recovered, and the one-dimensional speech waveform corresponding to the target sampling rate is output.

[0059] In some embodiments, such as this one, the training unit 204 is specifically used for: The keyword recognition model is initialized to obtain a training model, wherein the keyword recognition model is used to identify whether the input speech contains a target keyword, and the preset target label is a label that represents that the input speech contains the target keyword; The poisoned speech training set is input into the model to be trained, and the model to be trained outputs the prediction results of each speech in the poisoned speech training set. The loss value is calculated based on the prediction results and the labels of each speech in the poisoned speech training set. The model parameters of the model to be trained are updated based on the loss value; The loss value is calculated iteratively and the model parameters are updated until a preset training termination condition is met. The model to be trained when the preset training termination condition is met is taken as the backdoor model.

[0060] In some embodiments, such as this one, the backdoor security assessment device 200 based on timbre leakage further includes: A selection unit is used to obtain a clean test set, select test speech from the clean test set, and input the test speech and the trigger speech set into the timbre leakage trigger function to obtain poisoned test speech. The first statistical unit is used to input the poisoned test voice into the backdoor model and output the proportion of the poisoned test voice with the preset target label to all the poisoned test voices as the attack success rate. The second statistical unit is used to input the untreated test speech into the backdoor model and calculate the proportion of the backdoor model's prediction results that match the real labels, which is used as the clean accuracy rate. The determination unit is used to determine that the backdoor model meets the attack requirements when the attack success rate is not lower than a preset success rate threshold and the difference between the clean accuracy rate and the clean accuracy rate of the benchmark model does not exceed a preset threshold. The benchmark model is a model obtained by training the keyword recognition model based on the clean speech training set.

[0061] The aforementioned backdoor security assessment device based on timbre leakage can be implemented as a computer program, which can, for example... Figure 6 It runs on the computer device shown.

[0062] Please see Figure 6 , Figure 6This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 300 is a device capable of performing backdoor security assessments based on audio leakage.

[0063] See Figure 6 The computer device 300 includes a processor 302, a memory, and a network interface 305 connected via a system bus 301. The memory may include a non-volatile storage medium 303 and internal memory 304.

[0064] The non-volatile storage medium 303 may store an operating system 3031 and a computer program 3032. When the computer program 3032 is executed, it causes the processor 302 to execute a backdoor security assessment method based on tone leakage.

[0065] The processor 302 provides computing and control capabilities to support the operation of the entire computer device 300.

[0066] The internal memory 304 provides an environment for the execution of the computer program 3032 in the non-volatile storage medium 303. When the computer program 3032 is executed by the processor 302, the processor 302 can execute a backdoor security assessment method based on tone leakage.

[0067] This network interface 305 is used for network communication with other devices. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 300 to which the present invention is applied. The specific computer device 300 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0068] The processor 302 is used to run a computer program 3032 stored in a memory to implement any embodiment of the above-described backdoor security assessment method based on timbre leakage.

[0069] It should be understood that, in this embodiment of the invention, the processor 302 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0070] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by a processor in the computer system to implement the process steps of the embodiments of the above methods.

[0071] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform any embodiment of the above-described backdoor security assessment method based on tone leakage.

[0072] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0074] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0075] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0077] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0078] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A backdoor security assessment method based on timbre leakage, characterized in that, include: Obtain a clean speech training set and a trigger speech set, and divide the clean speech training set into a subset of speech to be poisoned and a subset of clean speech; For each speech in the subset of speech to be poisoned, the speech to be poisoned and the trigger speech set are input into the timbre leakage trigger function. The timbre leakage trigger function extracts timbre features from the trigger speech set and injects the timbre features into a local region of the speech to be poisoned to obtain the poisoned speech. The labels of each of the poisoned voice messages are modified to preset target labels, and then merged with the clean voice subset to generate a poisoned voice training set; The keyword recognition model is trained based on the poisoned voice training set to obtain the backdoor model; The acquired speech to be evaluated and the trigger speech set are input into the timbre leakage trigger function to obtain an attack test sample. The attack test sample is then input into the backdoor model, and the predicted labels output by the backdoor model are used to evaluate the security of the keyword recognition model against the timbre leakage backdoor attack.

2. The method according to claim 1, characterized in that, The step of extracting timbre features from the trigger speech set by the timbre leakage trigger function and injecting the timbre features into a local region of the speech to be poisoned to obtain the poisoned speech includes: The timbre leakage trigger function calls the pre-trained hidden unit model, and the speech to be poisoned is input into the hidden unit model to obtain the hidden unit vector sequence to be poisoned; Each trigger speech in the trigger speech set is input into the hidden unit model to obtain the hidden unit vector corresponding to each trigger speech, and the obtained hidden unit vectors are collected into a trigger hidden unit vector library. The sequence of hidden unit vectors to be poisoned is subjected to continuous constraint clustering to obtain a clustered semantic vector sequence. The continuous constraint clustering refers to dividing the sequence of hidden unit vectors to be poisoned into K continuous intervals while keeping the time order of each hidden unit vector unchanged, and assigning a continuous interval label to each hidden unit vector to be poisoned; K is a preset number of clustering categories. According to the preset interval selection rule, at least one of the continuous intervals from the K continuous intervals of the clustering semantic vector sequence is selected as the interval to be replaced, and all the hidden unit vectors to be poisoned in the interval to be replaced are used as the hidden unit vectors to be replaced. For each of the hidden unit vectors to be replaced, calculate the similarity between it and each of the hidden unit vectors in the trigger hidden unit vector library, and select the hidden unit vector with the highest similarity to replace the hidden unit vector to be replaced. Then, cover the hidden unit vector to be poisoned back into the sequence of hidden unit vectors in the original position to obtain the timbre leakage vector sequence. The timbre leakage trigger function calls a pre-trained vocoder model, inputs the timbre leakage vector sequence into the vocoder model, and reconstructs a one-dimensional speech waveform from the vocoder model, which is then used as the poisoned speech.

3. The method according to claim 2, characterized in that, The step of inputting the speech to be poisoned into the hidden unit model to obtain the hidden unit vector sequence to be poisoned includes: The single-channel voice waveform of the speech to be poisoned is normalized in amplitude and resampled to obtain the processed one-dimensional waveform. The processed one-dimensional waveform is input into the hidden unit model, which then performs feature encoding and context modeling sequentially to output the sequence of hidden unit vectors to be poisoned.

4. The method according to claim 2, characterized in that, The step of performing continuous constraint clustering on the sequence of latent units to be poisoned to obtain a clustered semantic vector sequence includes: Normalize each of the hidden unit vectors to be poisoned in the sequence of hidden unit vectors to be poisoned, and calculate the cosine distance between any two normalized hidden unit vectors to be poisoned. The interval cost within any candidate continuous interval is calculated based on the cosine distance, wherein the more similar the vectors of the hidden units to be poisoned within the candidate continuous interval, the smaller the interval cost. A dynamic programming algorithm is used to select K-1 splitting positions between two adjacent hidden unit vectors to be poisoned, and to solve the minimum total cost of dividing the prefix of the hidden unit vector sequence to be poisoned into intervals step by step, and to save the optimal splitting position at each stage. By backtracking the optimal segmentation positions of each stage, K consecutive intervals are determined, and the consecutive interval labels are assigned to all the hidden unit vectors to be poisoned in the sequence of hidden unit vectors to be poisoned, thus obtaining the clustering semantic vector sequence.

5. The method according to claim 2, characterized in that, The step of inputting the timbre leakage vector sequence into the vocoder model and reconstructing a one-dimensional speech waveform from the vocoder model includes: The timbre leakage vector sequence is input into the vocoder model, and the vocoder model maps the timbre leakage vector sequence into conditional features; Based on the conditional features, the temporal structure of the speech is recovered, and the one-dimensional speech waveform corresponding to the target sampling rate is output.

6. The method according to claim 1, characterized in that, The step of training the keyword recognition model based on the poisoned speech training set to obtain the backdoor model includes: The keyword recognition model is initialized to obtain a training model, wherein the keyword recognition model is used to identify whether the input speech contains a target keyword, and the preset target label is a label that represents that the input speech contains the target keyword; The poisoned speech training set is input into the model to be trained, and the model to be trained outputs the prediction results of each speech in the poisoned speech training set. The loss value is calculated based on the prediction results and the labels of each speech in the poisoned speech training set. The model parameters of the model to be trained are updated based on the loss value; The loss value is calculated iteratively and the model parameters are updated until a preset training termination condition is met. The model to be trained when the preset training termination condition is met is taken as the backdoor model.

7. The method according to any one of claims 1-6, characterized in that, After the step of training the keyword recognition model based on the poisoned speech training set to obtain the backdoor model, the method further includes: Obtain a clean test set, select test speech from the clean test set, and input the test speech and the trigger speech set into the timbre leakage trigger function to obtain the poisoned test speech; The poisoned test voice is input into the backdoor model, and the proportion of the poisoned test voice with the preset target label to all the poisoned test voices is statistically output as the attack success rate. The untreated test speech is input into the backdoor model, and the proportion of the backdoor model's prediction results that match the real labels is counted as the clean accuracy. When the attack success rate is not lower than a preset success rate threshold, and the difference between the clean accuracy and the clean accuracy of the benchmark model does not exceed a preset threshold, the backdoor model is determined to meet the attack requirements. The benchmark model is a model obtained by training the keyword recognition model based on the clean speech training set.

8. A backdoor security assessment device based on timbre leakage, characterized in that, include: A partitioning unit is used to acquire a clean speech training set and a trigger speech set, and to partition the clean speech training set into a subset of speech to be poisoned and a subset of clean speech. The input extraction unit is used to input the speech to be poisoned and the trigger speech set into the timbre leakage trigger function for each speech in the subset of speech to be poisoned. The timbre leakage trigger function extracts timbre features from the trigger speech set and injects the timbre features into a local region of the speech to be poisoned to obtain the poisoned speech. The modification and merging unit is used to modify the labels of each of the poisoned voices to preset target labels and merge them with the clean voice subset to generate a poisoned voice training set; The training unit is used to train the keyword recognition model based on the poisoned speech training set to obtain the backdoor model; An evaluation unit is used to input the acquired speech to be evaluated and the trigger speech set into the timbre leakage trigger function to obtain an attack test sample, input the attack test sample into the backdoor model, and evaluate the security of the keyword recognition model against the timbre leakage backdoor attack by the predicted label output by the backdoor model.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images