Adversarial Defense Method and System for Voiceprint Recognition Based on F-ratio Adaptive Masking

Abstract

This invention discloses an adversarial defense method for a speaker recognition system based on F-ratio adaptive masking. The method first extracts features from the input speech to obtain an amplitude spectrogram; then, it denoises the amplitude spectrogram to obtain a denoised amplitude spectrogram; F-ratio is used to statistically analyze the high-relevance and low-relevance frequency band sets in the denoised amplitude spectrogram to distinguish the speaker; next, masking thresholds for the high-relevance and low-relevance frequency bands are calculated separately; the amplitude spectrogram is then masked to obtain a reconstructed amplitude spectrogram; the reconstructed amplitude spectrogram is transformed using librosa.griffinlim to obtain the corresponding waveform signal as the reconstructed speech; finally, a batch of clean samples is used for speech reconstruction, and the reconstructed speech is used for SRS fine-tuning training to ensure the classification performance of SRS. This invention exhibits significant defensive advantages, and the average defense capability demonstrates the versatility of this scheme against different attacks. Furthermore, because this invention does not involve additional data and training, it possesses low cost.

Description

Technical Field

[0001] This invention belongs to the field of biometric security technology, and relates to a method and system for countering voiceprint recognition systems, particularly a method and system for countering voiceprint recognition systems based on F-ratio adaptive masking. Technical Background

[0002] Voiceprint recognition systems, also known as speaker recognition systems (SRS), are biometric authentication methods and automated technologies for identifying specific speakers from speech. They have been adopted by commercial products such as Microsoft Azure and Amazon Alexa in banking authentication, forensic testing, and smart device personalization services. However, the emergence of adversarial attacks poses a serious threat to the security of SRSs. Attackers can introduce imperceptible perturbations into clean speech to misclassify SRSs, thereby impersonating a trusted, legitimate registrant.

[0003] To defend against adversarial attacks, researchers have developed various adversarial defenses for SRS. These defenses can be categorized into three types: adversarial training, stochastic smoothing, and input reconstruction. Adversarial training uses adversarial examples as augmentation data to retrain the classifier; stochastic smoothing applies Gaussian noise as data augmentation to fine-tune the classifier and uses it to build a soft classifier; input reconstruction creates a separate auxiliary module that can clean or destroy adversarial noise. In real-world applications, an effective defense method should possess two ideal properties: 1) generality, meaning it can provide defense for different speaker recognition models and defend against different attacks; 2) low cost, meaning the defense does not require additional data and computational overhead, and the introduction of the defense mechanism does not require retraining the classifier. However, adversarial training has poor generality against different attacks, and retraining introduces significant computational demands. Meanwhile, the high computational cost of stochastic smoothing during the robustness proof phase hinders its practical applicability. Compared to adversarial training and stochastic smoothing, input reconstruction better meets the criteria of generality and low cost, making it more promising for practical applications.

[0004] However, input reconstruction is very vulnerable to adaptive adversarial attacks because it focuses only on cleaning or destroying adversarial noise in the input data, while ignoring the possibility that the input itself contains perturbable features. This oversight creates an opportunity for attackers to use defensive information to launch adaptive attacks, thereby generating adversarial perturbations that reconstruction defenses cannot easily eliminate. Summary of the Invention:

[0005] To address the aforementioned technical problems, this invention proposes an adversarial defense method and system for voiceprint recognition based on F-ratio adaptive masking. This invention ensures defense capability based on the idea of ​​non-robust feature screening, rather than resisting the destruction or purification of noise.

[0006] The technical solution adopted by the method of the present invention is: an adversarial defense method for a voiceprint recognition system based on F-ratio adaptive masking, comprising the following steps:

[0007] Step 1: Extract features from the input speech to generate a matrix with dimensions [frequency band × duration], denoted as the amplitude spectrum;

[0008] Step 2: Denoise the amplitude spectrum to obtain a denoised amplitude spectrum;

[0009] Step 3: Use F-ratio statistics to distinguish the speaker's high-correlation frequency band set and low-correlation frequency band set in the denoised amplitude spectrum;

[0010] Step 4: Calculate the masking thresholds for the high speaker-related frequency band and the low speaker-related frequency band respectively;

[0011] Step 5: For high speaker correlation bands, the values ​​of points with amplitude values ​​less than the corresponding band masking threshold are set to 0; for low speaker correlation bands, the values ​​of points with amplitude values ​​less than the corresponding band masking threshold are set to 0, thus obtaining the final reconstructed amplitude spectrum.

[0012] Step 6: Use librosa.griffinlim to transform the reconstructed amplitude spectrum to obtain the corresponding waveform signal as the reconstructed speech;

[0013] Step 7: Use a batch of clean samples to reconstruct the speech, and use the reconstructed speech to fine-tune the SRS training to ensure the classification performance of the SRS.

[0014] Preferably, in step 1, a short-time Fourier transform is performed on the input speech, and then the absolute value is calculated for feature extraction.

[0015] As a preferred embodiment, step 2 includes the following sub-steps:

[0016] Step 2.1: Randomly generate a Gaussian noise of the same length as the input audio, and convert it into a Gaussian noise amplitude spectrum through a short-time Fourier transform;

[0017] Step 2.2: Subtract the Gaussian noise amplitude spectrum from the amplitude spectrum to obtain the denoised amplitude spectrum.

[0018] As a preferred embodiment, step 3 includes the following sub-steps:

[0019] Step 3.1: Using the clean dataset of LibriSpeech, select M people, each person selects N sentences, calculate the average length of all audio, and lengthen or trim all audio to a uniform length; where M and N are preset values.

[0020] Step 3.2: Definition in, The amplitude spectrum feature represents the j-th speaker's amplitude spectrum, where j = 1, 2, ..., N and i = 1, ..., M; u i and u represent the average characteristics of the i-th speaker and all speakers, respectively.

[0021] The dimensions of each variable d = dim(Fratio) = dim(x) = dim(u) = [bandwidth × duration] = [B × F];

[0022] Step 3.3: Calculate the thresholds for distinguishing between high speaker correlation and low speaker correlation. Where B is the total number of frequency bands; Fraction b This represents the average Fratio value corresponding to the b-th frequency band. If Fratio b If the value is greater than τ, then this frequency band is assigned to the high speaker-related frequency band set; otherwise, it is assigned to the low speaker-related frequency band set.

[0023] Preferably, in step 4, firstly, uniform noise of the same length as the input audio and ranging from (ε, -ε) is randomly generated as simulated adversarial noise, where ε represents a parameter used to control the noise level. Then, the simulated adversarial noise is added to the original audio to obtain simulated adversarial samples. Next, the simulated adversarial samples are subjected to a short-time Fourier transform to extract features, resulting in a noisy amplitude spectrum. Then, the amplitude spectrum is subtracted from the noisy amplitude spectrum to obtain a difference spectrum. Finally, the maximum difference corresponding to each frequency band in the set of frequency bands relevant to the high speaker is calculated in frequency bands. The maximum difference corresponding to each frequency band in the set of frequency bands relevant to the high speaker, multiplied by α, is used as the masking threshold θ for the high speaker's relevant frequency bands. H , where α represents the control parameter for the high speaker-related frequency band masking threshold, used to control the balance between the performance of the defense method and the performance of the voiceprint recognition system.

[0024] Preferably, in step 4, librosa.piptrack is first used to calculate the average amplitude values ​​of the fundamental tone and overtones in the range of 350Hz-4000Hz; then, this average amplitude value multiplied by β is used as the masking threshold θ for the low speaker-related frequency band. l , where β represents the control parameter for the low speaker-related frequency band masking threshold, which is also used to control the balance between the performance of the defense method and the performance of the voiceprint recognition system.

[0025] Preferably, in step 7, the SRS fine-tuning training is performed using reconstructed speech. The training does not require changing the original SRS network framework and loss function. The labels of the original speech are assigned to the reconstructed speech, the original model parameters of SRS are loaded, and then the reconstructed speech set is used as a new dataset to update the model parameters of SRS after dividing it into training and test sets. Training stops when the test accuracy is stable.

[0026] The technical solution adopted by the system of this invention is: a voiceprint recognition system adversarial defense system based on F-ratio adaptive masking, comprising:

[0027] One or more processors;

[0028] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the adversarial defense method for the voiceprint recognition system based on F-ratio adaptive masking.

[0029] To create a more practical, highly versatile, highly defensive, and low-resource-consumption adversarial defense method for voiceprint recognition models, this invention proposes an adversarial defense method and system based on F-ratio adaptive masking. This method utilizes F-ratio to calculate and locate spectral bands that are highly or lowly correlated with the speaker recognition task. It filters out non-robust features using partial masking and adaptively calculates the corresponding masking threshold for each frequency band in the spectrogram using an adaptive threshold algorithm. This retains robust features suitable for speaker classification while filtering out non-robust features susceptible to adversarial attacks, thus achieving the defense objective.

[0030] Compared to existing technologies, the advantages of this invention are: while fully ensuring the classification performance of the voiceprint recognition system, it significantly improves both the single-point defense performance and the overall defense performance against mass attacks. Compared with other existing methods, our defense against mass attacks shows a significant improvement in single-point defense performance and also boasts the strongest overall defense capability. Attached Figure Description

[0031] The technical solutions described herein are further illustrated below using examples and specific implementation methods. Additionally, accompanying drawings are used in the description of the technical solutions. Those skilled in the art can, without any creative effort, obtain other drawings and the intent of the present invention based on these drawings.

[0032] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating an adversarial attack in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the countermeasures in an embodiment of the present invention. Detailed Implementation

[0035] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0036] Speaker recognition systems (SRSs) are commonly used for biometric identification. However, these systems are vulnerable to adversarial attacks. Existing adversarial defense measures typically require significant data and computational resources to ensure robustness, which does not meet the practical application requirements for highly versatile, robust, and resource-efficient defense methods. This invention proposes an adversarial defense method and system based on F-ratio adaptive masking, which utilizes the idea of ​​filtering out non-robust features to ensure defensive capabilities. This scheme uses F-ratio to calculate and locate spectral bands that are highly and lowly correlated with the speaker recognition task, uses partial masking to filter non-robust features, and employs an adaptive threshold algorithm to adaptively calculate the corresponding masking threshold for each frequency band in the spectrogram. This retains robust features suitable for speaker classification while filtering out non-robust features susceptible to adversarial attacks, thus achieving the defensive objective.

[0037] Please see Figure 1 The present invention provides an adversarial defense method for a voiceprint recognition system based on F-ratio adaptive masking, comprising the following steps:

[0038] Step 1: Perform a Short Time Fourier Transform (STFT) on the input speech, then calculate the absolute value to extract features and generate a matrix with dimensions [bandwidth × duration], denoted as the amplitude spectrum.

[0039] In one implementation, the input speech is subjected to a short-time Fourier transform (STFT), and then the absolute value is obtained for feature extraction.

[0040] Step 2: Denoise the amplitude spectrum to obtain a denoised amplitude spectrum;

[0041] In one implementation, step 2 specifically includes the following sub-steps:

[0042] Step 2.1: Randomly generate a Gaussian noise of the same length as the input audio, and convert it into a Gaussian noise amplitude spectrum through a short-time Fourier transform (STFT);

[0043] Step 2.2: Subtract the Gaussian noise amplitude spectrum from the amplitude spectrum to obtain the denoised amplitude spectrum.

[0044] Step 3: Use F-ratio statistics to distinguish the speaker's high-correlation frequency band set and low-correlation frequency band set in the denoised amplitude spectrum;

[0045] In one implementation, step 3 specifically includes the following sub-steps:

[0046] Step 3.1: Using the clean dataset of LibriSpeech, select M people (M is 50 in this example), select N sentences for each person (N is 50 in this example), calculate the average length of all audio, and lengthen all audio (if the duration is insufficient, cut and splice from the beginning) or trim to a uniform length; where M and N are preset values;

[0047] Step 3.2: Definition in, The amplitude spectrum feature represents the j-th speaker's amplitude spectrum, where j = 1, 2, ..., N and i = 1, ..., M; u i and u represent the average characteristics of the i-th speaker and all speakers, respectively.

[0048] The dimensions of each variable d = dim(Fratio) = dim(x) = dim(u) = [bandwidth × duration] = [B × F];

[0049] Step 3.3: Calculate the thresholds for distinguishing between high speaker correlation and low speaker correlation. Where B is the total number of frequency bands; Fraction b This represents the average Fratio value corresponding to the b-th frequency band. If Fratio b If the value is greater than τ, then this frequency band is assigned to the high speaker-related frequency band set; otherwise, it is assigned to the low speaker-related frequency band set.

[0050] Step 4: Calculate the masking thresholds for the high speaker-related frequency band and the low speaker-related frequency band respectively;

[0051] In one implementation, the process of calculating the masking threshold for the high speaker-related frequency bands involves first randomly generating uniform noise of the same length as the input audio and ranging from (ε to -ε) as simulated adversarial noise, where ε represents a parameter used to control the noise level. Then, the simulated adversarial noise is added to the original audio to obtain simulated adversarial samples. These samples are then subjected to a Short-Time Fourier Transform (STFT) to extract features, resulting in a noisy amplitude spectrum. The amplitude spectrum is then subtracted from the noisy amplitude spectrum to obtain a difference spectrum. Finally, the maximum difference corresponding to each frequency band in the high speaker-related frequency band set is calculated in frequency bands. The maximum difference corresponding to each frequency band in the high speaker-related frequency band set, multiplied by α, is used as the masking threshold θ for the high speaker-related frequency bands. H α represents the control parameter for the high speaker-related frequency band masking threshold, used to control the balance between the performance of the defense method and the performance of the voiceprint recognition system. The final value of the α parameter is located by the binary search method. The initial value of α is set to 100, the search range of the binary search is set to (0,100), and the minimum threshold of the voiceprint recognition system performance is set to ace. The defense performance is tested by PGD attack and represented by aae. During the binary search process, if the performance of the voiceprint recognition system is less than ace, α decreases. When the performance of the voiceprint recognition system is greater than ace, α is decreased and increased according to the increase and decrease of aae. Finally, while ensuring that the performance of the voiceprint recognition system is greater than ace, the larger aae is, the better.

[0052] In one implementation, the calculation of the masking threshold for the low speaker-related frequency band first uses librosa.piptrack to calculate the average amplitude values ​​of the fundamental tone and overtones in the range of 350Hz-4000Hz; then, this average amplitude value multiplied by β is used as the masking threshold θ for the low speaker-related frequency band. l The β parameter is located by a binary search method. The initial value of β is set to 2, and the search range of the binary search is set to (0,2). The minimum threshold for the performance of the voiceprint recognition system is set to ace. The defense performance is tested by PGD attack and represented by aae. If the performance of the voiceprint recognition system is less than ace during the binary search process, β decreases. When the performance of the voiceprint recognition system is greater than ace, β is reduced and increased according to the rise and fall of aae. Finally, while ensuring that the performance of the voiceprint recognition system is greater than ace, the larger aae is, the better.

[0053] Step 5: For high speaker correlation bands, the values ​​of points with amplitude values ​​less than the corresponding band masking threshold are set to 0; for low speaker correlation bands, the values ​​of points with amplitude values ​​less than the corresponding band masking threshold are set to 0, thus obtaining the final reconstructed amplitude spectrum.

[0054] Step 6: Use librosa.griffinlim to transform the reconstructed amplitude spectrum to obtain the corresponding waveform signal as the reconstructed speech;

[0055] Step 7: Use a batch of clean samples to reconstruct the speech, and use the reconstructed speech to fine-tune the SRS training to ensure the classification performance of the SRS.

[0056] In one implementation, the SRS fine-tuning training is performed using reconstructed speech. The training does not require changing the original SRS network framework and loss function. The original speech is labeled with the reconstructed speech, the original SRS model parameters are loaded, and then the reconstructed speech set is used as a new dataset to update the SRS model parameters after dividing the training and test sets. Training stops when the test accuracy is stable.

[0057] This invention also provides an adversarial defense system for voiceprint recognition based on F-ratio adaptive masking, comprising:

[0058] One or more processors;

[0059] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the adversarial defense method for the voiceprint recognition system based on F-ratio adaptive masking.

[0060] This embodiment further illustrates the invention through experiments; please see [link / reference]. Figure 2 and Figure 3 Table 1 illustrates the adversarial attack and defense methods of this invention. Five BPDA adaptive adversarial attack methods were used in the experiment to test the defense performance and versatility against different defenses. Four advanced input reconstruction defenses were used for comparative experiments. The defense capabilities of the five input reconstruction defenses against the five adaptive adversarial attacks are shown in Table 1.

[0061] Table 1. Defense capabilities of five input reconstruction defenses against five adaptive adversarial attacks.

[0062]

[0063]

[0064] Table 1 shows that, compared with four advanced input reconstruction defenses, the present invention demonstrates a clear defensive advantage. The average defense capability shows the versatility of the present solution against different attacks. In addition, since the present invention does not involve additional data and training, the present solution has the nature of low cost.

[0065] This invention features low cost, high defense, and versatility. The defense is constructed without training; its implementation does not require retraining of the protected SRSs, only fine-tuning. While maintaining a classification accuracy of 99.42%, this scheme achieves an average defense capability of 90.89% against five adaptive white-box attacks, representing improvements of 9.23% and 3.77% compared to low-cost and high-cost input reconstruction defenses, respectively. This scheme can be used to protect different SRSs.

[0066] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for adversarial defense of a voiceprint recognition system based on F-ratio adaptive masking, characterized in that, Includes the following steps: Step 1: Extract features from the input speech to generate a matrix with dimensions [frequency band × duration], denoted as the amplitude spectrum; Step 2: Denoise the amplitude spectrum to obtain a denoised amplitude spectrum; Step 3: Use F-ratio statistics to distinguish the speaker's high-correlation frequency band set and low-correlation frequency band set in the denoised amplitude spectrum; Step 4: Calculate the masking thresholds for the high speaker-related frequency band and the low speaker-related frequency band respectively; Step 5: For high speaker correlation bands, the values ​​of points with amplitude values ​​less than the corresponding band masking threshold are set to 0; for low speaker correlation bands, the values ​​of points with amplitude values ​​less than the corresponding band masking threshold are set to 0, thus obtaining the final reconstructed amplitude spectrum. Step 6: Use librosa.griffinlim to transform the reconstructed amplitude spectrum to obtain the corresponding waveform signal as the reconstructed speech; Step 7: Use a batch of clean samples to reconstruct the speech, and use the reconstructed speech to fine-tune the SRS training to ensure the classification performance of the SRS.

2. The adversarial defense method for voiceprint recognition system based on F-ratio adaptive masking according to claim 1, characterized in that: In step 1, a short-time Fourier transform is performed on the input speech, and then the absolute value is calculated to extract features.

3. The adversarial defense method for voiceprint recognition system based on F-ratio adaptive masking according to claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Randomly generate a Gaussian noise of the same length as the input audio, and convert it into a Gaussian noise amplitude spectrum through a short-time Fourier transform; Step 2.2: Subtract the Gaussian noise amplitude spectrum from the amplitude spectrum to obtain the denoised amplitude spectrum.

4. The adversarial defense method for voiceprint recognition system based on F-ratio adaptive masking according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3.1: Using the clean dataset of LibriSpeech, select M people, each person selects N sentences, calculate the average length of all audio, and lengthen or trim all audio to a uniform length; where M and N are preset values. Step 3.2: Definition in, The amplitude spectrum feature represents the j-th speaker's amplitude spectrum, where j = 1, 2, ..., N and i = 1, ..., M; u i and u represent the average characteristics of the i-th speaker and all speakers, respectively. The dimensions of each variable d = dim(Fratio) = dim(x) = dim(u) = [bandwidth × duration] = [B × F]; Step 3.3: Calculate the thresholds for distinguishing between high speaker correlation and low speaker correlation. Where B is the total number of frequency bands; Fraction b This represents the average Fratio value corresponding to the b-th frequency band. If Fratio b If the value is greater than τ, then this frequency band is assigned to the high speaker-related frequency band set; otherwise, it is assigned to the low speaker-related frequency band set.

5. The adversarial defense method for voiceprint recognition system based on F-ratio adaptive masking according to claim 1, characterized in that: In step 4, firstly, uniform noise of the same length as the input audio and ranging from (ε, -ε) is randomly generated as simulated adversarial noise, where ε represents a parameter used to control the noise level. Then, the simulated adversarial noise is added to the original audio to obtain simulated adversarial samples. Next, the simulated adversarial samples are subjected to a short-time Fourier transform to extract features, resulting in a noisy amplitude spectrum. Then, the amplitude spectrum is subtracted from the noisy amplitude spectrum to obtain a difference spectrum. Finally, the maximum difference corresponding to each frequency band in the set of frequency bands relevant to the high speaker is calculated in frequency bands. The maximum difference corresponding to each frequency band in the set of frequency bands relevant to the high speaker, multiplied by α, is used as the masking threshold θ for the high speaker's relevant frequency bands. H , where α represents the control parameter for the high speaker-related frequency band masking threshold, used to control the balance between the performance of the defense method and the performance of the voiceprint recognition system.

6. The adversarial defense method for a voiceprint recognition system based on F-ratio adaptive masking according to claim 1, characterized in that: In step 4, librosa.piptrack is first used to calculate the average amplitude values ​​of the fundamental tone and overtones in the range of 350Hz-4000Hz; then, this average amplitude value multiplied by β is used as the masking threshold θ for the low speaker-related frequency band. l , where β represents the control parameter for the low speaker-related frequency band masking threshold, which is also used to control the balance between the performance of the defense method and the performance of the voiceprint recognition system.

7. The adversarial defense method for a voiceprint recognition system based on F-ratio adaptive masking according to claim 1, characterized in that: Step 7 describes using reconstructed speech for SRS fine-tuning training. Training does not require changing the original SRS network framework and loss function. The original speech labels are assigned to the reconstructed speech, the original SRS model parameters are loaded, and then the reconstructed speech set is used as a new dataset to update the SRS model parameters after splitting the training and test sets. Training stops when the test accuracy is stable.

8. A voiceprint recognition system based on F-ratio adaptive masking as a defense system, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the F-ratio-based adaptive masking method for voiceprint recognition system adversarial defense as described in any one of claims 1 to 7.

Images