A method for enhancing robustness of a voiceprint retrieval model
By preprocessing and performing secondary retrieval on the query samples of the voiceprint retrieval system, the perturbation of adversarial examples is disrupted, the robustness of the model is enhanced, the vulnerability of deep learning models to attacks is solved, and the accuracy and efficiency of the system are ensured.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2023-03-21
- Publication Date
- 2026-07-03
AI Technical Summary
Deep learning models are vulnerable to adversarial attacks in voiceprint retrieval systems, leading to recognition errors. Existing defense methods either affect model performance or increase training costs.
The query samples are preprocessed, including noise reduction and secondary retrieval. Gaussian perturbations are generated through short-time Fourier transform to destroy the precise perturbations of the adversarial samples, and secondary retrieval is performed in the subset with the highest relevance.
This improves the robustness of the voiceprint retrieval system, defends against adversarial sample attacks, and ensures the system's accuracy and efficiency.
Smart Images
Figure CN116312550B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for enhancing the robustness of a voiceprint retrieval model, belonging to the field of voiceprint retrieval technology. Background Technology
[0002] Similar to fingerprints, because everyone's voice is different, we can extract unique biometric information from each person's voice—this is voiceprint. By utilizing voiceprint information, we can identify the speaker using a single audio clip. Therefore, voiceprint retrieval technology is widely used for user identification and authentication. In a typical scenario, the user first registers their voice. Then, when a voice clip of a registered user is encountered, the voiceprint retrieval system can extract the voiceprint information, compare it with a database, and then identify the user. In recent years, with technological innovation, voiceprint retrieval systems have been widely applied.
[0003] In traditional voiceprint retrieval systems, Gaussian probability models were typically used to extract voiceprint information. However, with the rise of deep learning, voiceprint retrieval systems have gradually begun to utilize deep learning techniques to train models, and the resulting models have demonstrated better performance. Using deep learning models, it's even possible to implement an end-to-end voiceprint recognition system without treating voiceprint extraction as a separate step. Therefore, deep learning models are widely used in the field of voiceprint retrieval.
[0004] While deep learning models are widely used due to their efficiency, they also have certain limitations. Specifically, if a carefully designed small perturbation is added to the input of a deep learning model, the model may produce a completely incorrect result. This is known as an adversarial example attack. In the context of voiceprint retrieval systems, if we add noise that is imperceptible to the human ear to a speech segment, making it sound indistinguishable to the human ear, but when we use this as input to analyze the model, the model will produce a completely different result—that is, the model will be unable to identify the user who spoke that speech.
[0005] Adversarial example attacks are a problem faced by most deep learning models. This is partly due to the insufficient robustness of the model design. Many methods have been proposed to address this issue. One relatively intuitive approach is to create our own adversarial examples—noisy speech segments—and include them in the training set of the deep learning model. After training, the model can still correctly identify similar adversarial examples. However, unfortunately, this approach can negatively impact model performance and increase the cost of training the model.
[0006] Another approach is to denoise the query samples. Since the perturbations added in adversarial examples are carefully designed, denoising can disrupt this sophistication, thus rendering the adversarial examples ineffective to some extent. However, even after denoising the query samples, the search still occurs within the entire database of the system.
[0007] Related knowledge introduction:
[0008] Voiceprint retrieval based on deep learning: Voiceprint retrieval refers to extracting voiceprint information from a query sample, retrieving similar speech segments from an existing voiceprint retrieval database, and determining the user from whom the query sample originated based on these speech segments. Extracting voiceprint information is a crucial step in this process. Previous voiceprint retrieval systems typically used Gaussian probability models to extract information from speech segments and represented it as an ivector vector. Currently, with the introduction of deep learning models, voiceprint information is generally represented as a multi-bit 01 string. By comparing the distances between these 01 strings, the similarity of voiceprint information between speech segments can be obtained, and voiceprint retrieval is completed based on this.
[0009] Deep learning-based voiceprint retrieval represents a significant advancement in the field. By using deep learning models to extract voiceprint information and identify users, it greatly improves the efficiency and success rate of voiceprint retrieval. Furthermore, numerous end-to-end models have been proposed in this area, all demonstrating impressive performance.
[0010] Most existing models are based on models in the field of image retrieval, using convolutional neural network architectures, such as ResNet-34, while some models use recurrent neural network architectures.
[0011] Adversarial example attacks are attacks targeting deep learning models. They add a carefully designed perturbation to the model's input sample, a perturbation so small as to be easily undetectable, yet still causing the model to output completely incorrect results. For example, in a typical voiceprint retrieval system, after training, to test its performance, we input a query sample, and the model outputs a multi-bit string of 0s and 1s, which we use to search a voiceprint database. In a normal query, the information we obtain helps us identify which user spoke the query sample, with a relatively high accuracy rate. However, using an adversarial example attack, we add a perturbation to the query sample, a perturbation that is imperceptible to the human ear. After adding the perturbation, subsequent queries will result in the model giving an incorrect result with a very high probability.
[0012] Adversarial attacks, based on the attacker's capabilities, can be broadly categorized into white-box attacks and black-box attacks. White-box attacks involve attackers who know the model's internal structure, training parameters, training methods, etc., making them more targeted. There are many types of white-box attacks, such as FGSM, JSMA, PGD, etc.
[0013] A black-box attack refers to an attack where the attacker has no knowledge of the model's internal structure, training parameters, or training methods, and can only obtain the input and output results by accessing the model. Attacks performed under these circumstances are called black-box attacks. There are many types of black-box attacks, such as single-pixel attacks, query-based attacks, and attacks based on substitution models, etc. Summary of the Invention
[0014] Purpose of the invention: To address the problems and shortcomings of existing technologies, this invention provides a method to enhance the robustness of voiceprint retrieval models, which can defend against adversarial sample attacks in the field of voiceprint retrieval and improve the quality of voiceprint retrieval.
[0015] Technical Solution: A method to enhance the robustness of a voiceprint retrieval model primarily involves preprocessing the query samples to disrupt any sophisticated perturbations that might be introduced by adversarial attacks, thereby achieving a defensive purpose. Furthermore, after preprocessing, instead of searching the entire voiceprint database, we perform a second search on a subset of the original results that has high relevance, which improves efficiency to some extent.
[0016] The method mainly includes the following steps:
[0017] Step 1: First, use the original query sample to perform a search. The voiceprint retrieval system will return voice results in the voiceprint database that are similar to the original query sample based on the similarity of the voiceprint information.
[0018] Step 2: Based on the results provided by the voiceprint retrieval system, select a subset of the results as the data set for the secondary retrieval;
[0019] Step 3: Denoise the original query samples to obtain the denoised query samples;
[0020] Step 4: Use the denoised query samples to search again in the previously selected subset;
[0021] Step 5: The voiceprint retrieval system uses the results of the secondary retrieval as the final result.
[0022] In the method described, it is necessary to determine the subset of data used for the secondary retrieval. Adversarial examples demonstrate the model's lack of robustness, but upon closer examination of the reasons for the model's errors, we found that the main reason is that adversarial examples make data that should be similar to the sample less similar in the retrieval results. However, although less similar, the data itself only has a minor perturbation, so the impact is limited. Data that should be similar, while not ranking very high in the retrieval results, will still appear in a relatively high position. Therefore, in the secondary retrieval, this invention does not need to search the entire voiceprint database; it only needs to search the data with relatively high relevance in the first retrieval results.
[0023] A search is performed using the original query sample. The results given by the voiceprint retrieval system are generally sorted according to their similarity to the original query sample. Assuming that each user has registered several users' voices in the voiceprint retrieval database, and each user has registered k voices, then the top 2k, 3k, or 4k data in the results can be selected as the data subset for the second search. The specific parameter selection needs to be based on the strength of the adversarial attack. Generally speaking, the higher the attack strength, the larger the data subset should be. Let ε represent the adversarial attack strength, and K=nk represent the size of the data subset. If ε is larger, n also needs to be set larger. Users set n according to their needs. Theoretically, n can be set to the maximum number of registered users in the voiceprint database.
[0024] The method for denoising the original query samples includes the following steps:
[0025] Step 31: Perform a short-time Fourier transform on the original query sample.
[0026] Step 32: For the original query sample, generate a value with the same length as the query sample and conforming to X~ N (0,σ 2 The Gaussian perturbation of the distribution. The determination of the parameter σ needs to refer to the strength of the adversarial example attack. Let ε represent the attack strength of the adversarial example. If ε is larger, then σ also needs to be set larger. The specific value needs to be set based on experience.
[0027] Step 33: Perform a short-time Fourier transform on the disturbance generated in step 32.
[0028] Step 34: Subtract the results of the two short-time Fourier transforms in steps 31 and 33.
[0029] Step 35: Perform an inverse short-time Fourier transform on the result obtained in step 34. The result is the denoised sample.
[0030] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above for enhancing the robustness of the voiceprint retrieval model.
[0031] A computer-readable storage medium storing a computer program that performs the method described above for enhancing the robustness of a voiceprint retrieval model.
[0032] Beneficial effects: Compared with the prior art, the method for enhancing the robustness of the voiceprint retrieval model proposed in this invention has the following advantages:
[0033] (1) A defense method against adversarial sample attacks in the field of voiceprint retrieval is proposed. This invention can be applied to defend against most adversarial sample attacks and has broad application prospects.
[0034] (2) This method destroys the sophisticated adversarial noise on adversarial examples by denoising, thus defending against adversarial example attacks.
[0035] (3) It protects the voiceprint retrieval system, enabling the system to provide more accurate services during use. Attached Figure Description
[0036] Figure 1 This is a schematic diagram illustrating how the defense strategy of this invention is used in the voiceprint retrieval system;
[0037] Figure 2 This is a general flowchart of the defense strategy according to an embodiment of the present invention;
[0038] Figure 3 This is a flowchart illustrating how to determine the retrieval subset of preprocessed query samples in an embodiment of the present invention;
[0039] Figure 4 This is a flowchart illustrating how to reduce noise in query samples in an embodiment of the present invention. Detailed Implementation
[0040] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0041] This invention provides a method to enhance the robustness of a voiceprint retrieval model. In this method, the precision of the noise added to the adversarial sample is utilized, and noise reduction is performed to destroy its precision, thereby playing a protective role. Furthermore, the pre-processed query samples are only in the subset with the highest relevance in the original results, and subsequent retrieval can further improve efficiency.
[0042] like Figure 1 As shown, the present invention aims to protect the voiceprint retrieval system from adversarial sample attacks.
[0043] like Figure 2 As shown, the method for enhancing the robustness of the voiceprint retrieval model mainly includes the following steps:
[0044] Step 1: First, use the original query sample to perform a search. The voiceprint retrieval system will return voice results in the voiceprint database that are similar to the original query sample based on the similarity of the voiceprint information.
[0045] Step 2: Based on the results provided by the voiceprint retrieval system, select a subset of the results as the data set for the secondary retrieval;
[0046] Step 3: Denoise the original query samples to obtain the denoised query samples;
[0047] Step 4: Use the denoised query samples to search again in the previously selected subset;
[0048] Step 5: The voiceprint retrieval system uses the results of the secondary retrieval as the final result.
[0049] like Figure 3 As shown, the initial search is performed using the original query sample to obtain the first search result. The results provided by the voiceprint retrieval system are generally sorted based on similarity. Assuming that each user's registration includes the voiceprints of several users in the voiceprint retrieval database, and each user has registered k voice records, we can select the top 2k, 3k, or 4k data points from the results as a subset for the second search. The specific parameters need to be chosen with reference to the strength of the adversarial attack. The selected data will then be used as the subset for the second search.
[0050] like Figure 4 As shown, the original query samples are denoised using the following steps:
[0051] Step 31: Perform a short-time Fourier transform on the original query sample.
[0052] Step 32: For the original query sample, generate a value with the same length as the query sample and conforming to X~ N (0,σ 2 The Gaussian perturbation of the distribution. The determination of the parameter σ needs to refer to the strength of the adversarial example attack. Let ε represent the attack strength of the adversarial example. If ε is larger, then σ also needs to be set larger. The specific value needs to be set based on experience.
[0053] Step 33: Perform a short-time Fourier transform on the disturbance generated in step 32.
[0054] Step 34: Subtract the results of the two short-time Fourier transforms in steps 31 and 33.
[0055] Step 35: Perform an inverse short-time Fourier transform on the result obtained in step 34 to obtain the denoised sample.
[0056] It is obvious to those skilled in the art that the steps of the method for enhancing the robustness of the voiceprint retrieval model in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using device-executable program code, thereby storing them in a storage device for execution by the computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.
[0057] In summary, this invention provides a method to enhance the robustness of voiceprint retrieval models, and this technology can be applied to deep learning-based voiceprint retrieval systems. By disrupting the precision of the perturbations added to samples by adversarial attacks, this invention can reduce the risks posed by adversarial attacks to a certain extent. As deep learning-based voiceprint retrieval systems are increasingly applied in daily life, the defense against adversarial attacks will receive increasing attention; therefore, this technology has great application prospects.
Claims
1. A method for enhancing robustness of a voiceprint retrieval model, characterized in that, Includes the following steps: Step 1: First, use the original query sample to perform a search. The voiceprint retrieval system will return voice results in the voiceprint database that are similar to the original query sample based on the similarity of the voiceprint information. Step 2: Based on the results provided by the voiceprint retrieval system, select a subset of the results as the data set for the secondary retrieval; Step 3: Denoise the original query samples to obtain the denoised query samples; Step 4: Use the denoised query samples to search again in the previously selected subset; Step 5: The voiceprint retrieval system uses the results of the secondary retrieval as the final result; The original query sample is used for a first search. The results given by the voiceprint retrieval system are sorted according to their similarity to the original query sample. Assuming that each user has registered several users' voices in the voiceprint retrieval database, and each user has registered k voices, then the top 2k, 3k, or 4k data in the results are selected as the data subset for the second search. Let ε represent the attack strength of the adversarial sample, and K=nk represent the size of the data subset. If ε is larger, then n also needs to be set larger.
2. The method for enhancing robustness of a voiceprint retrieval model according to claim 1, wherein, The method for denoising the original query samples includes the following steps: Step 31: Perform a short-time Fourier transform on the original query sample; Step 32, for the original query sample, a Gaussian perturbation consistent with X N (0,σ 2 ) distribution is generated Step 33: Perform a short-time Fourier transform on the disturbance generated in step 32; Step 34: Subtract the results of the two short-time Fourier transforms in steps 31 and 33. Step 35: Perform an inverse short-time Fourier transform on the result obtained in step 34. The result is the denoised query sample.
3. The method for enhancing the robustness of the voiceprint retrieval model according to claim 2, characterized in that, For the original query sample, a Gaussian disturbance consistent with the length of the query sample and conforming to X ~ N (0,σ 2 ) distribution is generated; for the determination of the σ parameter, reference is required to the strength of the adversarial sample attack, and ε represents the attack strength of the adversarial sample. If ε is larger, σ also needs to be set larger.
4. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for enhancing the robustness of the voiceprint retrieval model as described in any one of claims 1-3.
5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that performs a method for enhancing the robustness of a voiceprint retrieval model as described in any one of claims 1-3.