Method, device and equipment for voiceprint recognition system to resist attack and medium

Abstract

The invention discloses a voiceprint recognition system attack resisting method and device, equipment and a medium, by constructing a shadow voice sample set and a substitution model, dependence on target speaker data and query capability is reduced, and privacy risk and query overhead are reduced; meanwhile, two types of substitution models are trained based on an information theory method, so that the mobility and stability of the models are improved, and the generalization ability is enhanced; besides, the attacker model adopts a specific architecture and combines an alternate training strategy, so that attack loss and prediction probability difference are optimized, imperceptibility and attack effectiveness are balanced, and tone quality reduction or attack failure is avoided; and finally, through systematic training and optimization, a more uniform evaluation caliber is expected to be provided, and more reliable guidance is provided for engineering landing and risk evaluation.

Description

A method, apparatus, device and medium for countering attacks on a voiceprint recognition system Technical Field

[0002] This application relates to the field of adversarial attack technology, and in particular to a method, apparatus, device and medium for adversarial attacks on a voiceprint recognition system. Background Technology

[0004] Voiceprint recognition technology is a crucial component of biometric identification, and it has matured significantly, achieving large-scale applications in numerous key fields. Compared to modalities like fingerprints and facial features, voice-based identity verification offers the advantages of being contactless, having low barriers to entry, and allowing for remote data collection, providing continuous and convenient identity verification at low cost. Therefore, voiceprint recognition systems built in complex acoustic environments and multi-device ecosystems have become an essential part of information security and intelligent interaction infrastructure. With the rapid development of artificial intelligence, deep learning models have become the core technical architecture of modern voiceprint recognition systems. Voiceprint recognition systems based on deep neural networks (such as convolutional neural networks, recurrent neural networks, and time-delay neural network models) can automatically learn highly discriminative voiceprint features from massive amounts of speech data, achieving recognition accuracy close to or even surpassing human levels in controlled environments. However, the security issues of deep learning models are becoming increasingly prominent, with adversarial attacks posing a significant challenge. In voiceprint recognition applications, the security threats posed by adversarial attacks are particularly severe. Attackers can impersonate legitimate users by playing carefully crafted adversarial audio, bypassing the voiceprint verification system to achieve malicious activities such as unauthorized device access and financial fraud, seriously threatening personal privacy, property security, and system reliability. Therefore, conducting in-depth research on adversarial attacks against voiceprint recognition systems and revealing their inherent vulnerability is of significant theoretical and practical value for promoting the construction of secure and robust voiceprint recognition systems.

[0005] Existing research typically categorizes adversarial attacks into white-box and black-box attacks based on the attacker's access to the target model. White-box attack methods, given the model structure, training parameters, and training data, directly utilize information such as model gradients and internal structure to construct perturbations, thus achieving a high success rate. Under a white-box setting, researchers directly generate adversarial speech based on gradient information, and even use "universal perturbations" to apply a unified perturbation vector to multiple speech streams. On the other hand, black-box attacks are currently a hot topic of research. These methods interact with the system only through input and output, and are further divided into query-based methods and transfer-based black-box algorithms. For query attacks, this method relies on launching a large number of queries to the target model, estimating the model gradient or decision boundary by analyzing the returned prediction results or confidence scores, and then iteratively optimizing the adversarial examples. Transfer-based methods aim to avoid direct interaction with the target model. Attackers first train one or more local surrogate models based on available data to simulate the behavior of the target model, then generate adversarial examples on the surrogate models, and rely on their cross-model transferability to attack the black-box target model. Existing research has attempted to improve transferability by strategies such as integrating multiple surrogate models, preserving timbre, or using speech conversion technology to enhance target speaker data.

[0006] Existing technologies suffer from the following drawbacks: First, the reliance on target speaker data and query capabilities makes it difficult to meet the needs of real-world compliance scenarios. This is because user voice data is subject to privacy and compliance restrictions, making it difficult to acquire continuously and at scale. Furthermore, online systems typically employ frequency limiting and risk control strategies, which struggle to support high query overhead. Second, transfer learning methods lack portability and stability, becoming sensitive to the structure of the alternative and target models, training corpora, and channel conditions. This leads to fluctuations in success rates across different models and corpora, resulting in a lack of stable transferability and generalization capabilities. Additionally, the balance between imperceptibility and attack effectiveness lacks systematic constraints, often relying on empirical weight adjustments, which can easily lead to decreased audio quality or attack failure. Finally, inconsistent evaluation criteria and a lack of standardized dataset selection and evaluation metrics result in incomparable results, hindering guidance for engineering implementation and risk assessment. Summary of the Invention

[0008] This application provides a method, apparatus, device, and medium for combating attacks on a voiceprint recognition system to solve the aforementioned technical problems.

[0009] In view of this, the first aspect of this application provides a method for adversarial attacks on a voiceprint recognition system. The method includes: generating a set of shadow speech samples of the target speaker's speech data through a speech conversion model; merging the set of shadow speech samples and the set of imposter speech of the non-target speaker into a training set for training a substitution model; constructing two types of substitution models and training the two types of substitution models respectively based on the difference in prediction probabilities of the two types of substitution models for the same training set using information theory methods; constructing an attacker model to be trained, the attacker model consisting of five one-dimensional convolutional blocks and a global residual connection, each convolutional block containing one-dimensional convolution, batch normalization, and ReLU activation function, and training the attacker model using the speech data of the target speaker and the set of imposter speech of the non-target speaker as inputs; and alternatingly training the attacker model and the two types of substitution models until the attack loss of the attacker model is maximized and the absolute value of the difference in prediction probabilities of the two types of substitution models is maximized, thereby obtaining the parameter sets of the attacker model and the two types of substitution models.

[0010] Optionally, the step of generating the shadow speech sample set of the target speaker's speech data through the speech conversion model specifically involves: using a pre-trained FreeVC-s speech conversion model to generate the shadow speech sample set of the target speaker's speech data. Generate multiple shadow speech samples A collection of shadow speech samples, ,in, Voice samples from imposters who are not the target speaker. This is a collection of voice recordings by imposters.

[0011] Optionally, the set of shadow speech samples generated by the speech conversion model for the target speaker further includes: measuring the shadow speech samples' similarity to the target speaker's single-sentence speech data in the substitution model using a method based on neural tangent kernels. similarity , ,in, As an alternative model, The parameters represent the alternative model. To replace the model output with the gradient of parameters; similarity Shadow speech samples below a preset threshold Removed from the shadow speech sample set.

[0012] Optionally, the information-theoretic method for training two alternative models based on the difference in prediction probabilities of the two alternative models for the same training set specifically involves: calculating the difference in prediction probabilities of the two alternative models for the same training set using mutual information (MI) based on information theory. ,in, and Alternative models and For the same adversarial sample The predicted probability, Representing alternative models Regarding the uncertainty of adversarial examples, Indicates that in a given alternative model Output replacement model The remaining uncertainty of the predicted probability; to maximize For the goal, the alternative model and Conduct training.

[0013] Optionally, training the attacker model using the target speaker's voice data and the set of imposter voices (not from the target speaker) as input specifically includes: using preset weights... Shadow speech samples and the corresponding imposter voice samples of the non-target speaker Perform linear fusion to obtain fused speech samples , ; will include fused speech samples The training set is used as input to the attacker model.

[0014] Optionally, the alternating training of the attacker model and the two alternative models until the attack loss of the attacker model is maximized and the absolute value of the difference between the predicted probabilities of the two alternative models is maximized, thereby obtaining the parameter sets of the attacker model and the two alternative models, specifically involves: when training the attacker model... At that time, fixed substitution model and The parameters of the attacker model are the input fused speech samples. Convert to adversarial examples And calculate the first loss value. , of which For the damage caused by the attack, , When training an alternative model and At that time, fixed attack model The parameters are used to calculate the second loss value. Alternately train the attacker model and the two alternative models until the attacker model's attack loss is reached. Maximize and the absolute value of the difference between the predicted probabilities of the two alternative models. Maximize the parameters to obtain the attacker model and the parameter sets of the two alternative models.

[0015] A second aspect of this application provides an adversarial attack device for a voiceprint recognition system. The device includes: a speech conversion unit for generating a set of shadow speech samples of the target speaker's speech data using a speech conversion model; a training set merging unit for merging the shadow speech sample set and the set of imposter speech of the non-target speaker into a training set for training alternative models; a first construction unit for constructing two types of alternative models, training the two types of alternative models based on the difference in prediction probabilities of the two types of alternative models for the same training set using information theory methods; a second construction unit for constructing an attacker model to be trained, the attacker model consisting of five one-dimensional convolutional blocks and a global residual connection, each convolutional block containing one-dimensional convolution, batch normalization, and a ReLU activation function, using the target speaker's speech data and the set of imposter speech of the non-target speaker as inputs for training; and a training unit for alternately training the attacker model and the two types of alternative models until the attack loss of the attacker model is maximized and the absolute value of the difference in prediction probabilities between the two types of alternative models is maximized, thus obtaining the parameter sets of the attacker model and the two types of alternative models.

[0016] A third aspect of this application provides a device for countering attacks on a voiceprint recognition system. The device includes a processor and a memory: the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the steps of the method for countering attacks on a voiceprint recognition system as described in the first aspect, according to the instructions in the program code.

[0017] A fourth aspect of this application provides a computer-readable storage medium for storing program code for performing the method described in the first aspect above.

[0018] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: This application provides a method, apparatus, device, and medium for adversarial attacks on a voiceprint recognition system. By constructing a set of shadow speech samples and alternative models, it reduces the dependence on target speaker data and query capabilities, thereby reducing privacy risks and query overhead. At the same time, training two types of alternative models based on information theory methods improves the transferability and stability of the models and enhances their generalization ability. In addition, the attacker model adopts a specific architecture and combines an alternating training strategy to optimize the difference between attack loss and prediction probability, balancing imperceptibility and attack effectiveness, and avoiding sound quality degradation or attack failure. Finally, through systematic training and optimization, it is expected to provide a more unified evaluation standard, providing more reliable guidance for engineering implementation and risk assessment. Attached Figure Description

[0020] Figure 1 is a flowchart of the method for countering attacks on the voiceprint recognition system in an embodiment of this application; Figure 2 is a schematic diagram of the structure of the counter-attack device for the voiceprint recognition system in an embodiment of this application; Figure 3 is a schematic diagram of the structure of the counter-attack device for the voiceprint recognition system in an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0023] For ease of understanding, please refer to Figure 1. Figure 1 is a flowchart of the adversarial attack method of the voiceprint recognition system in this embodiment of the application. As shown in Figure 1, specifically: S101, generate a set of shadow voice samples of the target speaker's voice data through a voice conversion model; specifically: use a pre-trained FreeVC-s voice conversion model to convert the target speaker's voice data... Generate multiple shadow speech samples A collection of shadow speech samples, ,in, Voice samples from imposters who are not the target speaker. For the imposter's speech set; a method based on the neural tangent kernel is used to measure the similarity between the shadow speech samples and the target speaker's single-sentence speech data in the substitution model. , ,in, As an alternative model, The parameters represent the alternative model. To replace the model output with the gradient of parameters; similarity Shadow speech samples below a preset threshold Removed from the shadow speech sample set.

[0024] It should be noted that this step uses a pre-trained FreeVC-s speech conversion model. FreeVC-s is an advanced speech conversion model that can convert the speech features of one speaker into the features of a target speaker while preserving the speech content. Its core advantage lies in its ability to generate high-quality and natural speech samples, achieving smooth conversion even between different speakers.

[0025] Based on the target speaker's speech data The FreeVC-s model generates speech samples containing multiple shadows. The set. The generation process is represented as... Among them, the voice samples are those of imposters who are not the target speaker. This is a collection of imposter's voices. In this way, a large number of voice samples that are similar to but not exactly the same as the target speaker can be generated, providing rich data for subsequent training.

[0026] The neural tangent kernel (NTK) method is used to measure the similarity between the shadow speech samples and the target speaker. This method can effectively assess the similarity between samples, ensuring that the generated shadow speech samples are sufficiently similar in features to the target speaker's speech. Finally, shadow speech samples with similarity below a preset threshold are removed from the set to ensure the quality of the generated shadow speech samples.

[0027] NTK is a theoretical framework for characterizing the learning process of neural networks, explaining training behavior by analyzing the sensitivity of network outputs to parameter changes. In neural networks with widths approaching infinity, NTK remains approximately invariant during training, allowing the optimization process to approximate a linear system. This property makes NTK useful for understanding models trained on shadow data (S) and models trained on real data (N). Since NTK describes how a neural network processes input features and makes decisions, optimizing the NTK structure of S to be highly aligned with the NTK of N ensures that S is close to N in both decision boundaries and feature responses. This makes S a more effective alternative to N, thereby improving the success rate of adversarial attacks. It is a powerful tool for model alignment and decision boundary behavior.

[0028] To ensure that model S can effectively approximate the decision-making behavior of model N, the kernel matrices of both are constrained to satisfy NTK alignment:

[0029] in For Frobenius inner product, Let it be the Frobenius norm. By letting By making S as close to 1 as possible, model S can be similar to model N in terms of decision boundaries and feature sensitivity. S aligns with N during optimization and maintains consistency in classification behavior. Combined with NTK theory, this method provides an interpretable and verifiable guarantee for maintaining engineering feasibility.

[0030] S102. Merge the shadow speech sample set and the imposter speech set of the non-target speaker into a training set for training the substitution model. It should be noted that merging the filtered shadow speech sample set with the imposter speech set of the non-target speaker forms the training set for training the substitution model. This training set contains features of the target speaker (reflected through shadow speech samples) and features of the non-target speaker, providing a rich data source for training the substitution model.

[0031] By merging shadow speech samples and imposter speech samples, the training set can better simulate the distribution of speech data in real-world scenarios. This data fusion approach not only increases data diversity but also improves the generalization ability of the alternative model, enabling it to better adapt to different speakers and speech features.

[0032] S103. Construct two types of alternative models, and train the two types of alternative models separately based on the difference in prediction probabilities of the two types of alternative models for the same training set using information theory methods; specifically: based on information theory methods, calculate the difference in prediction probabilities of the two types of alternative models for the same training set using mutual information (MI). ,in, and Alternative models and For the same adversarial sample The predicted probability, Representing alternative models Regarding the uncertainty of adversarial examples, Indicates that in a given alternative model Output replacement model The remaining uncertainty of the predicted probability; to maximize For the goal, the alternative model and Conduct training.

[0033] It should be noted that two types of alternative models are constructed. and The structure and training objectives of these two types of models are designed to simulate the decision-making process of real voiceprint recognition systems. By using information theory methods to train the two alternative models on the same training set based on the difference in predicted probabilities, the dissimilarity between the models can be enhanced, thereby improving the effectiveness of the attack.

[0034] Training alternative models using information theory methods can effectively increase the differentiation between models, enabling attacker models to better leverage this differentiation to generate effective adversarial examples. This approach not only improves the success rate of attacks but also enhances the model's generalization ability, allowing it to function effectively in different voiceprint recognition systems.

[0035] Understandably, traditional methods often use difference loss (such as...) or The norm measures the output difference. We further introduce information theory tools to quantify and regulate this difference, thereby achieving better transferability in black-box models. In information theory, the mutual information (MI) of two random variables characterizes their degree of interdependence. Let... and Two alternative models are defined, and their predictions for the same adversarial example are denoted as follows: and We use MI to measure the statistical dependency between the two outputs. It is defined as:

[0036] in For joint distribution, and It is distributed at the periphery. Larger... This indicates that the outputs of the two models are highly correlated, meaning that their decision boundaries are more similar when dealing with the same adversarial example. This high similarity may limit the generalization of perturbations between different models.

[0037] To further analyze the relationship between mutual information (MI) and model consistency, we examine the impact of model discrepancies on the transferability of adversarial examples. Another form of mutual information facilitates theoretical explanation:

[0038] in express The uncertainty in predicting this adversarial example, Indicates that in a given After output The remaining uncertainty in the forecast. Maximize This will force the two models to exhibit greater consistency on adversarial examples, thereby aligning their decision boundaries on AEs.

[0039] Next, the necessity of maintaining model diversity will be explained. If... and Their joint distribution is exactly the same (e.g., identical initialization). It will degenerate into ,at this time This only reflects the entropy of a single model and cannot reveal shared vulnerabilities across models. Instead, it can be enriched by introducing controlled variances (such as different initializations or data splitting). Based on this, maximize This will cause adversarial example perturbations to fall into the vulnerable regions where the two models overlap, rather than exploiting the specific weaknesses of only a single model. Formalized as:

[0040] From the perspective of information bottlenecks, this process is equivalent to "compressing" redundant information between the predictions of the two models by maximizing mutual information, forcing adversarial examples to retain only discriminative features that are sensitive to both, thereby improving cross-model generalization. If and If the gradient directions are highly correlated, the same perturbation can mislead both simultaneously, and its transferability will be significantly enhanced.

[0041] S104. Construct the attacker model to be trained. The attacker model consists of five one-dimensional convolutional blocks and one global residual connection. Each convolutional block contains one-dimensional convolution, batch normalization, and ReLU activation function. The attacker model is trained by using the speech data of the target speaker and the set of imposter speech from non-target speakers as inputs. Specifically, training the attacker model by using the speech data of the target speaker and the set of imposter speech from non-target speakers as inputs includes: using preset weights... Shadow speech samples and the corresponding imposter voice samples of the non-target speaker Perform linear fusion to obtain fused speech samples , ; will include fused speech samples The training set is used as input to the attacker model.

[0042] It should be noted that the attacker model consists of five one-dimensional convolutional blocks and a global residual connection. Each convolutional block contains a one-dimensional convolution, batch normalization, and a ReLU activation function. This structural design aims to efficiently process speech signals while maintaining moderate model complexity, facilitating training and optimization.

[0043] The attacker model is designed to combine shadow speech samples and imposter speech samples, generating fused speech samples through linear fusion. and Two texts with the same content and similar length are assigned different weights. And linear fusion, to obtain :

[0044] This strategy achieves smooth fusion while preserving the characteristics of each component. The training set containing the fused data enables... During the training phase, information about the target speaker can be "perceived," thereby enhancing the ability to launch targeted attacks.

[0045] S105. Alternately train the attacker model and the two alternative models until the attack loss of the attacker model is maximized and the absolute value of the difference between the predicted probabilities of the two alternative models is maximized, thus obtaining the parameter sets of the attacker model and the two alternative models; specifically: when training the attacker model... At that time, fixed substitution model and The parameters of the attacker model are the input fused speech samples. Convert to adversarial examples And calculate the first loss value. , of which For the damage caused by the attack, , When training an alternative model and At that time, with the parameters of the attacker model G fixed, the second loss value is calculated. Alternately train the attacker model and the two alternative models until the attacker model's attack loss is reached. Maximize and the absolute value of the difference between the predicted probabilities of the two alternative models. Maximize the parameters to obtain the attacker model and the parameter sets of the two alternative models.

[0046] It should be noted that the alternating training strategy effectively balances the relationship between the attacker model and the alternative model, enabling the attacker model to generate more aggressive samples, while the alternative model can better simulate the decision-making process of a real voiceprint recognition system. This training method not only improves the success rate of attacks but also enhances the model's stability and generalization ability, allowing it to function effectively in different voiceprint recognition systems.

[0047] The training process is modeled as a minimax game. Attack sample generator. The alternative model is treated as one of two participants in a zero-sum game, and optimization is performed using alternating training. Time fixed and The parameters are as follows, and the total loss is:

[0048] At this stage, the attacker Increase the likelihood of model misjudgment by generating adverse events (AEs). . This is used to guide alternative models to maintain correct classification on clean inputs and reduce AE output discrepancies, both of which together improve attack capabilities. Meanwhile... Constrain the amplitude of the perturbation to maintain the stealth of the attack.

[0049] Conversely, in training and And fixed At that time, the total loss is:

[0050] Mutual information modulation affects the transferability of adverse events (AEs); this stage gradually amplifies the differences between alternative models, enabling the AE generation process to explore a wider range of vulnerable areas. Combined with another stage, the three losses interact and complement each other, achieving a balance between attack effectiveness and stealth, forming a stable adversarial game process.

[0051] Please refer to Figure 2, which is a schematic diagram of the structure of the adversarial attack device for the voiceprint recognition system in this embodiment of the application. As shown in Figure 2, specifically: a speech conversion unit 201, used to generate a set of shadow speech samples of the target speaker's speech data through a speech conversion model; a training set merging unit 202, used to merge the shadow speech sample set and the imposter speech set of the non-target speaker into a training set for training the substitution model; a first construction unit 203, used to construct two types of substitution models, and train the two types of substitution models respectively based on the difference in prediction probabilities of the two types of substitution models for the same training set using information theory methods; a second construction unit 203; and a second construction unit 204. Unit 204 is used to construct the attacker model to be trained. The attacker model consists of five one-dimensional convolutional blocks and a global residual connection. Each convolutional block contains one-dimensional convolution, batch normalization and ReLU activation function. The speech data of the target speaker and the set of imposter speech of the non-target speaker are used as inputs to train the attacker model. Unit 205 is used to alternately train the attacker model and the two alternative models until the attack loss of the attacker model is maximized and the absolute value of the difference between the prediction probabilities of the two alternative models is maximized, so as to obtain the parameter set of the attacker model and the two alternative models.

[0052] Another embodiment of the present invention provides a voiceprint recognition system anti-attack device, as shown in FIG3. The device 10 includes one or more processors 110 and a memory 120. FIG3 is described with one processor 110 as an example. The processor 110 and the memory 120 can be connected by a bus or other means. FIG3 is shown with a bus connection as an example.

[0053] Processor 110 is used to perform various control logics of device 10, and can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0054] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the method for constructing the multilingual phoneme representation model in this embodiment of the invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby implementing the method for constructing the multilingual phoneme representation model in the above-described method embodiment.

[0055] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] One or more units are stored in memory 120 and, when executed by one or more processors 110, perform the following steps: generating a set of shadow speech samples of the target speaker's speech data through a speech conversion model; merging the set of shadow speech samples and the set of imposter speech of the non-target speaker into a training set for training the substitution model; constructing two types of substitution models and training the two types of substitution models separately based on the difference in prediction probabilities of the two types of substitution models for the same training set using information theory methods; constructing an attacker model to be trained, which consists of five one-dimensional convolutional blocks and a global residual connection, each convolutional block containing one-dimensional convolution, batch normalization, and ReLU activation functions, and training the attacker model with the speech data of the target speaker and the set of imposter speech of the non-target speaker as inputs; and training the attacker model and the two types of substitution models alternately until the attack loss of the attacker model is maximized and the absolute value of the difference in prediction probabilities of the two types of substitution models is maximized, thereby obtaining the parameter sets of the attacker model and the two types of substitution models.

[0057] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by one or more processors, they implement any one of the above-described methods for countering attacks on a voiceprint recognition system.

[0058] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0059] This application provides a method, apparatus, device, and medium for adversarial attacks on a voiceprint recognition system. By constructing a shadow speech sample set and alternative models, it reduces reliance on target speaker data and query capabilities, thereby lowering privacy risks and query overhead. Simultaneously, training two types of alternative models based on information theory improves model transferability and stability, enhancing generalization ability. Furthermore, the attacker model employs a specific architecture and combines an alternating training strategy to optimize attack loss and prediction probability differences, balancing imperceptibility and attack effectiveness, and avoiding audio quality degradation or attack failure. Finally, through systematic training and optimization, it is expected to provide a more unified evaluation standard, offering more reliable guidance for engineering implementation and risk assessment.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0061] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0062] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0063] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0064] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0065] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0066] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0067] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0068] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for countering attacks on a voiceprint recognition system, characterized in that, include: A set of shadow speech samples of the target speaker's speech data is generated using a speech conversion model; The shadow speech sample set and the imposter speech set of the non-target speaker are merged into a training set for training the substitution model. Two types of substitution models are constructed, and the two types of substitution models are trained separately based on the difference in prediction probabilities of the two types of substitution models for the same training set using information theory methods. An attacker model to be trained is constructed, which consists of five one-dimensional convolutional blocks and a global residual connection. Each convolutional block contains one-dimensional convolution, batch normalization, and ReLU activation function. The speech data of the target speaker and the imposter speech set of the non-target speaker are used as inputs to train the attacker model. The attacker model and the two types of substitution models are trained alternately until the attack loss of the attacker model is maximized and the absolute value of the difference in prediction probabilities between the two types of substitution models is maximized, thus obtaining the parameter sets of the attacker model and the two types of substitution models.

2. The method for countering attacks on a voiceprint recognition system according to claim 1, characterized in that, The specific method for generating the shadow speech sample set of the target speaker's speech data through the speech conversion model is as follows: A pre-trained FreeVC-s speech conversion model is used to generate the shadow speech sample set based on the target speaker's speech data. Generate multiple shadow speech samples A collection of shadow speech samples, ,in, Voice samples from imposters who are not the target speaker. This is a collection of voice recordings by imposters.

3. The method for countering attacks on a voiceprint recognition system according to claim 2, characterized in that, The shadow speech sample set generated by the speech conversion model for the target speaker's speech data further includes: measuring the similarity between the shadow speech samples and the target speaker's single-sentence speech data t in the substitution model using a method based on neural tangent kernels. ,in, As an alternative model, The parameters represent the alternative model. To replace the gradient of the model output with respect to the parameters, shadow speech samples with similarity below a preset threshold are used. Removed from the shadow speech sample set.

4. The method for countering attacks on a voiceprint recognition system according to claim 3, characterized in that, The method based on information theory, which trains two alternative models based on the difference in prediction probabilities of the two alternative models for the same training set, specifically involves calculating the difference in prediction probabilities of the two alternative models for the same training set using mutual information (MI). ,in, and The alternative model and For the same adversarial sample The predicted probability, Representing alternative models Regarding the uncertainty of adversarial examples, Indicates that in a given alternative model Output replacement model The remaining uncertainty of the predicted probability; to maximize For the goal, the alternative model and Conduct training.

5. The method for countering attacks on a voiceprint recognition system according to claim 4, characterized in that, The step of training the attacker model by using the target speaker's speech data and the imposter's speech data (not the target speaker) as input specifically includes: using preset weights w to weight the shadow speech samples... and the corresponding imposter voice samples of the non-target speaker Perform linear fusion to obtain fused speech samples ， ; will include fused speech samples The training set is used as input to the attacker model.

6. The method for countering attacks on a voiceprint recognition system according to claim 5, characterized in that, The process of alternately training the attacker model and the two alternative models until the attack loss of the attacker model is maximized and the absolute value of the difference between the predicted probabilities of the two alternative models is maximized, thereby obtaining the parameter sets of the attacker model and the two alternative models, specifically involves: when training the attacker model... At that time, fixed substitution model and The parameters of the attacker model are the input fused speech samples. Convert to adversarial examples And calculate the first loss value. , of which For the damage caused by the attack, ， ； When training alternative models and At that time, fixed attack model The parameters are used to calculate the second loss value. Alternately train the attacker model and the two alternative models until the attacker model's attack loss is reached. Maximize and the absolute value of the difference between the predicted probabilities of the two alternative models. Maximize the parameters to obtain the attacker model and the parameter sets of the two alternative models.

7. A device for countering attacks on a voiceprint recognition system, characterized in that, include: The speech conversion unit is used to generate a set of shadow speech samples of the target speaker's speech data through a speech conversion model. The training set merging unit is used to merge the shadow speech sample set and the imposter speech set of the non-target speaker into a training set for training the substitution model; the first construction unit is used to construct two types of substitution models, and trains the two types of substitution models separately based on the difference in the prediction probabilities of the two types of substitution models for the same training set using information theory methods; the second construction unit is used to construct the attacker model to be trained. The attacker model consists of five one-dimensional convolutional blocks and a global residual connection. Each convolutional block contains one-dimensional convolution, batch normalization and ReLU activation function. The attacker model is trained by taking the speech data of the target speaker and the imposter speech set of the non-target speaker as inputs. The training unit is used to alternately train the attacker model and the two alternative models until the attack loss of the attacker model is maximized and the absolute value of the difference between the prediction probabilities of the two alternative models is maximized, thus obtaining the parameter set of the attacker model and the two alternative models.

8. A voiceprint recognition system anti-attack device, characterized in that, The device includes a processor and a memory: the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the voiceprint recognition system anti-attack method according to any one of claims 1-6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the voiceprint recognition system anti-attack method according to any one of claims 1-6.

Images