A speech adversarial defense method and system for a speaker recognition system

Abstract

The application provides a speech confrontation defense method and system for a speaker recognition system, proposes a new type of confrontation purification framework SA-Net, and the key idea is to adopt a'subtraction first and then addition' strategy at the feature level. The subtraction step filters out non-robust features by analyzing the distribution of speaker features, thereby compressing the survival space of the confrontation noise. The addition step reconstructs the complete speech signal, so that the speaker recognition system can accurately identify without additional fine-tuning or retraining. The average defense accuracy of the application when resisting adaptive attacks on two open source SRSs reaches 87.8%, while maintaining a normal recognition accuracy of 98.5%, which is 29.3% and 2.8% higher than Parallel WaveGAN respectively. In addition, the application has strong defense ability and wide applicability, and can be used as a plug-and-play defense line for SRS in various deployments.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and particularly relates to a speech attack defense method and system for a speaker recognition system. BACKGROUND

[0002] Voiceprint recognition technology has become a widely used biometric method due to its seamless integration with voice user interfaces (VUI). The progress of deep learning has enabled speech recognition systems (SRS) to be widely used in hardware devices (such as Google Home, Amazon Alexa) and software applications (such as HSBC, WeChat). Although the current most advanced SRS has made significant progress in performance, recent research has revealed that they are extremely sensitive to small perturbations from adversarial attacks, which can lead to false recognition results and thus pose a serious security risk. This problem has attracted increasing attention to the security of speech recognition systems.

[0003] In view of the serious threat posed by adversarial attacks, it has become an urgent task to develop effective countermeasures to protect speaker recognition systems. Existing adversarial defense methods can be divided into three categories: adversarial training, random smoothing, and input reconstruction methods. Since the first two methods require retraining and huge computational requirements, they are difficult to directly apply to mature commercial SRS, so we mainly focus on input reconstruction-based methods. In input reconstruction methods, one technical route is to defend against adversarial samples through signal / feature processing methods, such as quantization, resampling, low-pass filters, and FeCo, etc. However, these methods usually sacrifice the recognition performance of normal samples and require SRS to be fine-tuned or retrained to adapt to the corresponding defense strategy. Another method is to train auxiliary networks (such as Variational Autoencoder (VAE)) to purify adversarial noise, and these methods have the feature of plug-and-play. However, this method is often difficult to effectively defend against adaptive attacks.

[0004] Considering the high cost of further fine-tuning or retraining the speaker recognition system in actual deployment scenarios, we propose a new plug-and-play speech adversarial defense paradigm as a defense line for any deployed SRS to alleviate the vulnerability of SRS when facing attacks. To achieve a plug-and-play defense system, the following challenges are faced: (1) how to separate the adversarial perturbation from the speech sample to ensure the defense performance of the adversarial sample; (2) how to restore the pre-processed and damaged features so that the SRS can correctly identify without fine-tuning or retraining. Therefore, to achieve a plug-and-play adversarial defense system, balancing the adversarial defense performance and the speech recognition accuracy after restoration is a key problem. SUMMARY

[0005] The present application provides a speech adversarial defense method and system for a speaker recognition system to solve the defects in the prior art when deploying a speaker recognition system.

[0006] In a first aspect, the present application provides a speech adversarial defense method for a speaker recognition system, comprising:

[0007] Obtaining input speech data, performing feature extraction on the input speech data to obtain an amplitude spectrum graph of the input speech data;

[0008] Processing the amplitude spectrum graph through a pre-set fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum;

[0009] Reconstructing the masked amplitude spectrum through a U-Net model to obtain a reconstructed amplitude spectrum graph;

[0010] Performing phase estimation from the reconstructed amplitude spectrum graph based on a Griffin-Lim algorithm to obtain a reconstructed speech signal.

[0011] According to the speech adversarial defense method for a speaker recognition system provided by the present application, input speech data is obtained, feature extraction is performed on the input speech data to obtain an amplitude spectrum graph of the input speech data, comprising:

[0012] Frame the input speech data, and perform fast Fourier transform on each frame of the framed signal to obtain each frame of frequency domain signal;

[0013] Calculate the absolute value represented by each frame of frequency domain signal to obtain the amplitude spectrum graph.

[0014] According to the speech adversarial defense method for a speaker recognition system provided by the present application, the amplitude spectrum graph is processed through a pre-set fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum, comprising:

[0015] Obtain a speech data set formed by the amplitude spectrum graphs corresponding to a plurality of speakers, randomly select M people and N vectors for each person from the speech data set, obtain an audio sample from the librosa python library, and truncate and pad the audio sample to a fixed size of sampling points;

[0016] Calculate the speaker weight of different frequency bands in the truncated and padded audio sample using F-ratio statistics to obtain high speaker-related frequency bands and low speaker-related frequency bands;

[0017] Randomly generate uniform noise with a preset range and the same length as the input audio to simulate the adversarial noise, add the simulated adversarial noise to the speech data set to obtain simulated adversarial samples, and obtain high speaker-related frequency band masking thresholds and low speaker-related frequency band masking thresholds respectively according to the simulated adversarial samples.

[0018] According to the F-ratio value, the number of each frequency band that needs to be masked is calculated, and the position of random masking in each frequency band is determined.

[0019] According to the speech adversarial defense method for a speaker recognition system provided by the application, the speaker weight of different frequency bands in the audio sample after truncation and lengthening is calculated by F-ratio statistics, and high speaker-related frequency bands and low speaker-related frequency bands are obtained, including:

[0020] According to any amplitude spectrum vector of any speaker and the vector N, an average feature vector of any speaker is calculated;

[0021] According to the average feature vector of any speaker and the number of people M, an average feature vector of all speakers is calculated;

[0022] Based on the average feature vector of any speaker, the average feature vector of all speakers, the vector N and the number of people M, an F-ratio value is calculated;

[0023] An average F-ratio value corresponding to any frequency band is determined, and a division threshold is obtained in combination with the total number of frequency bands, the division threshold being used to divide a high speaker-related frequency band set and a low speaker-related frequency band set.

[0024] According to the speech adversarial defense method for a speaker recognition system provided by the application, high speaker-related frequency band masking thresholds and low speaker-related frequency band masking thresholds are obtained respectively according to the simulated adversarial samples, including:

[0025] The amplitude value of the simulated adversarial sample in any frequency band of the simulated adversarial sample and the amplitude value of the original sample in any frequency band of the corresponding original sample are calculated respectively;

[0026] After taking the absolute value of the difference between the amplitude value of the simulated adversarial sample and the amplitude value of the original sample and taking the maximum value, the high speaker-related frequency band masking threshold is obtained;

[0027] After calculating the pitch amplitude value of the amplitude value of the simulated adversarial sample and taking the average value, the low speaker-related frequency band masking threshold is obtained.

[0028] The speech confrontation defense method for the speaker recognition system provided by the application comprises the following steps of: calculating the number of frequency bands that need to be masked respectively according to F-ratio values, determining the random masking positions in each frequency band, comprising:

[0029] obtaining the amplitude value corresponding to the F-ratio value of any frequency band after the F-ratio calculation, the F-ratio value of any frequency band and the masking threshold value of the frequency band;

[0030] summing up the amplitude value corresponding to the F-ratio value of any frequency band after the F-ratio calculation which is less than the masking threshold value of the frequency band, and multiplying the F-ratio value of the any frequency band to obtain the number of frequency bands that need to be masked respectively;

[0031] randomly selecting the original sample amplitude value which is less than the masking threshold value of the frequency band and the number of frequency bands that need to be masked respectively to obtain the random masking positions in each frequency band.

[0032] The speech confrontation defense method for the speaker recognition system provided by the application comprises the following steps of: calculating the number of frequency bands that need to be masked respectively according to F-ratio values, determining the random masking positions in each frequency band, comprising:

[0033] determining that the U-Net model is used to reconstruct the masked amplitude spectrum, and the feature recovery module of the U-Net model comprises four layers of encoders and four layers of decoders;

[0034] inputting the masked amplitude spectrum into the encoder, adopting one-dimensional convolution which only operates along the frequency axis, and connecting the frequency transformation into each encoding layer, each encoder layer comprises two compression residual branches, adopting a Snake activation function instead of a ReLU activation function, and adding a long short-term memory network and a time-based attention module into the internal layer of the encoder, and converting the bottleneck feature back to the complete amplitude spectrum which has the same size as the encoder by the decoder;

[0035] calculating the amplitude spectrum loss based on the F-ratio speaker feature distribution and the multi-resolution short-time fast Fourier transform loss in the training respectively, and weighting and summing to obtain a total loss function.

[0036] The speech confrontation defense method for the speaker recognition system provided by the application comprises the following steps of: calculating the number of frequency bands that need to be masked respectively according to F-ratio values, determining the random masking positions in each frequency band, comprising:

[0037] obtaining the original speech amplitude spectrum and the recovered speech amplitude spectrum of any frequency band in the amplitude spectrum, and the F-ratio value of any frequency band in the amplitude spectrum;

[0038] Based on the original speech amplitude spectrum, the recovered speech amplitude spectrum and the F-ratio value of any frequency band in the amplitude spectrum, the amplitude spectrum loss based on the F-ratio speaker feature distribution is calculated.

[0039] Correspondingly, the multi-resolution short-time fast Fourier transform loss includes;

[0040] Obtain the original speech audio multi-resolution short-time Fourier transform value and the recovered speech audio multi-resolution short-time Fourier transform value.

[0041] Based on the original speech audio multi-resolution short-time Fourier transform value and the recovered speech audio multi-resolution short-time Fourier transform value, any resolution short-time fast Fourier transform loss is obtained.

[0042] Different resolutions are formed by different combinations of fast Fourier transform frequencies, hop lengths and window sizes, and the multi-resolution short-time fast Fourier transform loss is calculated by summing and averaging the short-time fast Fourier transform losses of different resolutions.

[0043] In a second aspect, the present application also provides a speech adversarial defense system for a speaker recognition system, comprising:

[0044] An extraction module is configured to obtain input speech data, perform feature extraction on the input speech data, and obtain an amplitude spectrum graph of the input speech data.

[0045] A masking module is configured to process the amplitude spectrum graph by using a pre-set fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum.

[0046] A reconstruction module is configured to reconstruct the masked amplitude spectrum by using a U-Net model to obtain a reconstructed amplitude spectrum graph.

[0047] An estimation module is configured to perform phase estimation from the reconstructed amplitude spectrum graph based on a Griffin-Lim algorithm to obtain a reconstructed speech signal.

[0048] In a third aspect, the present application also provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the speech adversarial defense method for a speaker recognition system as described above when executing the program.

[0049] The application provides a speech confrontation defense method and system for a speaker recognition system. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0051] Figure 1 FIG. 1 is a flowchart of the speech confrontation defense method for the speaker recognition system provided by the application;

[0052] Figure 2 FIG. 2 is a principle block diagram of the speech confrontation defense method for the speaker recognition system provided by the application;

[0053] Figure 3 FIG. 3 is a structure diagram of the speech confrontation defense system for the speaker recognition system provided by the application;

[0054] Figure 4 FIG. 4 is a structure diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0056] With the increasing importance of voiceprint technology in biometric authentication, the security vulnerabilities of speaker recognition systems (SRS) have attracted widespread attention due to the threat of adversarial samples. In order to solve the limitations of existing methods in meeting the plug-and-play requirements and defending adaptive attacks, the present application proposes a novel adversarial purification framework, SA-Net, whose key idea is to adopt a "subtract-then-add" strategy at the feature level. In order to suppress the existence of adversarial perturbations, in the "add" part, the present application proposes a fine feature filtering method based on F-ratio speaker feature distribution to remove non-robust features. In addition, in order to ensure the plug-and-play ability of the method of the present application, in the "subtract" part, the present application proposes a spectral recovery module based on U-Net and an amplitude loss based on F-ratio speaker feature distribution to reconstruct the deleted features, so as to achieve accurate speaker recognition without fine-tuning or retraining.

[0057] Figure 1 is a flowchart of the speech adversarial defense method for the speaker recognition system provided by the embodiments of the present application, as shown in Figure 1 , comprising:

[0058] Step 100: obtaining input speech data, performing feature extraction on the input speech data to obtain an amplitude spectrum graph of the input speech data;

[0059] Step 200: processing the amplitude spectrum graph through a pre-set fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum;

[0060] Step 300: reconstructing the masked amplitude spectrum through a U-Net model to obtain a reconstructed amplitude spectrum graph;

[0061] Step 400: performing phase estimation from the reconstructed amplitude spectrum graph based on a Griffin-Lim algorithm to obtain a reconstructed speech signal.

[0062] The technical solution adopted by the present application is a plug-and-play adversarial defense method for the speaker recognition system based on filtering and reconstruction, as shown in Figure 2 , comprising the following steps:

[0063] Step 1: Feature extraction is performed on the input speech to obtain the amplitude spectrum of the audio data.

[0064] Step 2: The amplitude spectrum is processed by a fine-grained non-robust feature filtering module to remove non-robust features and compress the survival space of adversarial perturbations.

[0065] Step 3: The masked amplitude spectrum is passed through a feature recovery module to reconstruct the complete amplitude spectrum.

[0066] Step 4: The phase is estimated from the reconstructed amplitude spectrum using the Griffin-Lim algorithm, and the reconstructed speech signal is finally obtained.

[0067] In one embodiment, in step 1, the input speech signal is subjected to a Short-Time Fourier Transform (STFT), which first frames the speech signal and then applies a Fast Fourier Transform (FFT) to each frame of signal to convert it from the time domain to the frequency domain. Next, the absolute value of the frequency domain representation of each frame is calculated to obtain the amplitude spectrum. The amplitude spectrum represents the amplitude information of each frequency component and reflects the intensity of the speech signal at different frequencies.

[0068] In one embodiment, step 2 includes:

[0069] A speech data set formed by the amplitude spectrum corresponding to a plurality of speakers is obtained, M people are randomly selected from the speech data set, N vectors are selected for each person, audio samples are obtained from the librosa python library, and the audio samples are truncated and padded to a fixed size of sampling points;

[0070] The speaker weights of different frequency bands in the truncated and padded audio samples are calculated using F-ratio statistics to obtain high speaker-related frequency bands and low speaker-related frequency bands.

[0071] Uniform noise with a preset range and the same length as the input audio is randomly generated to simulate adversarial noise, the simulated adversarial noise is added to the speech data set to obtain simulated adversarial samples, and high speaker-related frequency band masking thresholds and low speaker-related frequency band masking thresholds are obtained according to the simulated adversarial samples;

[0072] The number of each frequency band that needs to be masked is calculated according to the F-ratio value, and the position of random masking in each frequency band is determined.

[0073] Specifically, the embodiment of the present application randomly selects M persons from a multi-speaker speech data set, and selects N sentences for each person. The librosa python library reads the audio samples and truncates and pads them to a fixed size of sampling points to ensure that all training samples have the same length.

[0074] The F-ratio statistical method is used to calculate the speaker weight of different frequency bands in the audio data, and the high speaker correlation frequency band and the low speaker correlation frequency band region .

[0075] The F-ratio is defined as follows:

[0076]

[0077]

[0078] wherein, respectively represent the average feature vector of speaker i and all speakers. represents the jth amplitude spectrum vector of speaker i. Based on the calculation of F-ratio, the speaker weight of different frequency bands in the amplitude spectrum can be determined.

[0079] The division threshold of the high speaker correlation and low speaker correlation frequency bands is calculated:

[0080]

[0081] wherein, represents the average F-ratio value corresponding to the bth frequency band, and B represents the total number of frequency bands. According to the value and the size of the division threshold , the high speaker correlation frequency band and the low speaker correlation frequency band set are obtained respectively.

[0082] A uniform noise with the same length as the input audio and a range of is randomly generated to simulate the adversarial noise. The adversarial noise is added to the original speech sample to obtain the simulated adversarial sample. According to the simulated adversarial sample, the masking threshold of the high speaker correlation frequency band and the low speaker correlation frequency band is obtained, respectively, represented as and .

[0083]

[0084]

[0085] wherein, and respectively represent the amplitude values of the simulated adversarial sample x’ and the original sample x in the b-th frequency band. pitch() represents the amplitude value of the calculated pitch.

[0086] According to the F-ratio value, the number num of masks required for each frequency band is calculated, and the position of the random mask in each frequency band is determined, so that the subsequent recovery step can better recover according to the adjacent frequency components.

[0087]

[0088]

[0089]

[0090] wherein, represents the F-ratio value corresponding to the b-th frequency band, represents the amplitude value corresponding to the b-th frequency band after F-ratio calculation, represents the mask threshold value corresponding to the frequency band, represents a random selection function.

[0091] In one embodiment, step 3 comprises:

[0092] determining that the feature recovery module using the U-Net model reconstructs the masked amplitude spectrum, the feature recovery module comprising four layers of encoder and four layers of decoder;

[0093] inputting the masked amplitude spectrum into the encoder, using one-dimensional convolution only along the frequency axis, and connecting the frequency transform before each encoding layer, each encoder layer comprising two compression residual branches, using the Snake activation function instead of the ReLU activation function, and adding the long short-term memory network and the time-based attention module to the internal layer of the encoder, and converting the bottleneck features back to the complete amplitude spectrum with the same size as the encoder by the decoder;

[0094] In the training, the amplitude spectrum loss based on the F-ratio speaker feature distribution and the multi-resolution short-time fast Fourier transform loss are calculated respectively, and the total loss function is obtained by weighted summation.

[0095] Specifically, the masked magnitude spectrogram is reconstructed to the complete magnitude spectrogram by a feature recovery module based on the U-Net model. The module includes an encoder and a decoder part, each part has four layers. The encoder takes the masked magnitude spectrogram as input and uses 1D convolution that only operates along the frequency axis. In order to improve the performance of the model, we added a frequency transformation layer (FTL) before each encoder layer, which aims to capture the global correlation on the frequency axis, expand the receptive field of the model, and enhance the reconstruction ability of the magnitude spectrum. Each layer contains two compression residual branches, and uses the Snake activation function instead of the ReLU activation function. In addition, we also added a long short-term memory (LSTM) and a time-based attention module in the internal layer of the encoder. After the encoder, the decoder module converts the bottleneck features back to the complete magnitude spectrum with the same size as the input encoder.

[0096] The magnitude spectrum loss based on the F-ratio speaker feature distribution is calculated during the training process and the multi-resolution STFT loss , and the total loss function is weighted .

[0097]

[0098] wherein the magnitude spectrum loss based on the F-ratio speaker feature distribution is:

[0099]

[0100] wherein, denotes the F-ratio value of the i-th frequency band of the magnitude spectrum x. denotes the recovered magnitude spectrogram. and denote the magnitude spectrum of the original and recovered speech, respectively.

[0101] The multi-resolution STFT loss is:

[0102]

[0103]

[0104] wherein, STFT(x) and STFT( respectively, represent the multi-resolution short-time Fourier transform (STFT) transform on the original audio and the reconstructed audio, respectively. The FFT frequency bins are 512, 1024, 2048, respectively, the hop length is 50, 120, 240, and the window size is 240, 600, 1200. The multi-resolution STFT loss is calculated by summing and averaging the loss of different resolutions .

[0105] In one embodiment, step 4 comprises:

[0106] The phase information of the waveform is estimated using the Griffin-Lim algorithm instead of using the original phase feature, because the phase information may also contain adversarial perturbations that can negatively affect the final decision of the target model. The reconstructed amplitude spectrum and the estimated phase spectrum are then synthesized into the final purified speech waveform.

[0107] Further, the present application is further illustrated by experiments, and the experiments use the LibriSpeech dataset, specifically the "train-clean-100" subset. This subset contains 100 hours of English speech from 251 unique speakers. For each speaker, the present application randomly selects 90% of the speech as training data, and the remaining 10% for inference testing. The target speaker recognition system (SRS) is the widely used ECAPA-TDNN speaker recognition model, which is pre-trained on VoxCeleb to extract speaker embeddings. The similarity calculation uses cosine similarity as the similarity function. To comprehensively evaluate the defense effect, 4 white-box attacks, including FGSM, PGD, CW∞ and CW2, and 3 black-box attacks, including FAKEBOB (FB), SirenAttack (SA) and Kenansville (KS), are implemented. The defense ability of different defense methods for different attack methods is shown in Table 1.

[0108] Table 1 Comparison of experimental results

[0109]

[0110] Table 1 shows the average defense performance results of different defense methods under non-adaptive attacks. Input reconstruction methods such as quantization, noise addition, smoothing, downsampling, low-pass filter and AAC compression perform poorly in the classification accuracy of benign samples and adversarial samples. These methods will reduce the quality of the original audio, and the speaker recognition system is not fine-tuned or retrained for these defense methods. The present application achieves an accuracy of 99.1% on benign samples and performs well under white-box attacks, with an accuracy of 93.6% against FGSM attacks, 92.3% against PGD attacks, 90.3% against CWinf attacks, and 99.1% against CW2 attacks. In terms of black-box attacks, the present method outperforms other methods, with an accuracy of 98.5% against FAKEBOB attacks, 92.5% against SirenAttack attacks, and 88.3% against Kenansville attacks. These results reveal the effectiveness and universality of the present method, demonstrating its ability to resist various attack types and its plug-and-play feature without any adjustments to the SRS.

[0111] The speech adversarial defense system for a speaker recognition system provided by the present application is described below. The speech adversarial defense system for a speaker recognition system described below can be mutually corresponding with reference to the speech adversarial defense method for a speaker recognition system described above.

[0112] Figure 3 is a structural schematic diagram of the speech adversarial defense system for a speaker recognition system provided by an embodiment of the present application, as Figure 3 shown, comprising an extraction module 31, a masking module 32, a reconstruction module 33 and an estimation module 34, wherein:

[0113] The extraction module 31 is configured to obtain input speech data, perform feature extraction on the input speech data, and obtain an amplitude spectrum graph of the input speech data. The masking module 32 is configured to process the amplitude spectrum graph through a pre-set fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum. The reconstruction module 33 is configured to reconstruct the masked amplitude spectrum through a U-Net model to obtain a reconstructed amplitude spectrum graph. The estimation module 34 is configured to perform phase estimation from the reconstructed amplitude spectrum graph based on a Griffin-Lim algorithm to obtain a reconstructed speech signal.

[0114] Figure 4 An example of an entity structure schematic diagram of an electronic device is shown in Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute a voice confrontation defense method for a speaker recognition system, the method including: obtaining input voice data, performing feature extraction on the input voice data to obtain an amplitude spectrum graph of the input voice data; processing the amplitude spectrum graph through a pre-set fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum; reconstructing the masked amplitude spectrum through a U-Net model to obtain a reconstructed amplitude spectrum graph; and performing phase estimation from the reconstructed amplitude spectrum graph based on a Griffin-Lim algorithm to obtain a reconstructed voice signal.

[0115] In addition, the logical instructions in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0116] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0117] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0118] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for defending against speech adversarial attacks on a speaker recognition system, characterized in that: include: Acquiring input speech data, performing feature extraction on the input speech data, and obtaining an amplitude spectrogram of the input speech data; The amplitude spectrum is processed by a preset fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum; Reconstructing the masked amplitude spectrum through a U-Net model to obtain a reconstructed amplitude spectrum graph; Performing phase estimation from the reconstructed amplitude spectrogram based on the Griffin-Lim algorithm to obtain a reconstructed speech signal; The masked amplitude spectrum is reconstructed by the U-Net model to obtain a reconstructed amplitude spectrum, including: Determining to use a feature recovery module of the U-Net model to reconstruct the masked amplitude spectrum, the feature recovery module including a four-layer encoder and a four-layer decoder; The masked amplitude spectrum is input into the encoder, and a one-dimensional convolution operating only along the frequency axis is used. The frequency transform is connected before each encoding layer. Each encoder layer includes two compressed residual branches, and the Snake activation function is used instead of the ReLU activation function. A long short-term memory network and a time-based attention module are added to the internal layer of the encoder. The bottleneck feature is converted back to a complete amplitude spectrum of the same size as the encoder by the decoder; During training, the amplitude spectrum loss and multi-resolution short-time fast Fourier transform loss based on the F-ratio speaker feature distribution are calculated separately, and the weighted summation is used to obtain the total loss function.

2. The method for defending against speech adversarial attacks on a speaker recognition system according to claim 1, wherein: Acquiring input speech data, performing feature extraction on the input speech data, and obtaining an amplitude spectrogram of the input speech data, including: Frame the input voice data, and perform fast Fourier transform on each frame signal after framing to obtain a frequency domain signal of each frame; The absolute value of the frequency domain signal representation of each frame is calculated to obtain the amplitude spectrum.

3. The method for defending against speech adversarial attacks on a speaker recognition system according to claim 1, wherein: The amplitude spectrum is processed by a preset fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum, including: Obtain a speech dataset formed by the amplitude spectrograms corresponding to multiple speakers, randomly select a number of people M from the speech dataset, select a vector N for each person, obtain audio samples from the librosa Python library, and truncate and lengthen the audio samples to a fixed-size sampling point; The F-ratio statistics are used to calculate the speaker weights of different frequency bands in the truncated and padded audio samples to obtain high speaker-related frequency bands and low speaker-related frequency bands. Randomly generating uniform noise of the same length as the input audio and within a preset range to simulate adversarial noise, adding the simulated adversarial noise to the speech dataset to obtain a simulated adversarial sample, and obtaining a high speaker-dependent frequency band masking threshold and a low speaker-dependent frequency band masking threshold based on the simulated adversarial sample; The number of masks required for each frequency band is calculated based on the F-ratio value, and the position of random masking within each frequency band is determined.

4. The method for defending against speech adversarial attacks on a speaker recognition system according to claim 3, wherein: The F-ratio statistics are used to calculate the speaker weights of different frequency bands in the truncated and padded audio samples, and the high speaker-related frequency bands and low speaker-related frequency bands are obtained, including: Calculating an average feature vector of any speaker based on any amplitude spectrum vector of any speaker and the vector N; Calculating the average feature vector of all speakers based on the average feature vector of any speaker and the number of people M; Calculating an F-ratio value based on the average feature vector of any speaker, the average feature vector of all speakers, the vector N, and the number of people M; The average F-ratio value corresponding to any frequency band is determined, and combined with the total number of frequency bands to obtain a division threshold, which is used to divide the frequency band set into a high speaker correlation frequency band set and a low speaker correlation frequency band set.

5. The method for defending against speech adversarial attacks on a speaker recognition system according to claim 4, wherein: Obtaining a high speaker-related frequency band masking threshold and a low speaker-related frequency band masking threshold according to the simulated adversarial sample, including: Calculate the simulated adversarial sample amplitude value of the simulated adversarial sample in any frequency band and the original sample amplitude value of the corresponding original sample in any frequency band respectively; taking the absolute value of the difference between the amplitude value of the simulated adversarial sample and the amplitude value of the original sample and then taking the maximum value to obtain the high speaker correlation frequency band masking threshold; The pitch amplitude values ​​of the simulated adversarial sample amplitude values ​​are calculated and averaged to obtain the low speaker-related frequency band masking threshold.

6. The method for defending against speech adversarial attacks on a speaker recognition system according to claim 5, wherein: Calculate the number of maskings required for each frequency band based on the F-ratio value and determine the random masking locations within each frequency band, including: Get the amplitude value corresponding to any frequency band after Fratio calculation, the Fratio value corresponding to any frequency band, and the masking threshold corresponding to the frequency band; For all frequency bands whose amplitudes are smaller than the masking threshold corresponding to the frequency band, the corresponding amplitude values ​​are summed and multiplied by the Fratio value corresponding to the frequency band to obtain the number of masks required for each frequency band; The original sample amplitude values ​​that are smaller than the masking threshold corresponding to the frequency band and the number of each frequency band that needs to be masked are randomly selected to obtain the random masking position in each frequency band.

7. The method for defending against speech adversarial attacks on a speaker recognition system according to claim 1, wherein: The amplitude spectrum loss based on the F-ratio speaker feature distribution includes: Obtain the original speech amplitude spectrum and the reconstructed speech amplitude spectrum of any frequency band in the amplitude spectrum, as well as the F-ratio value of any frequency band in the amplitude spectrum; Calculating the amplitude spectrum loss based on the F-ratio speaker characteristic distribution based on the original speech amplitude spectrum, the reconstructed speech amplitude spectrum, and the F-ratio value of any frequency band in the amplitude spectrum; Correspondingly, the multi-resolution short-time fast Fourier transform loss includes: Obtaining a multi-resolution short-time Fourier transform value of the original speech audio and a multi-resolution short-time Fourier transform value of the restored speech audio; Obtaining a short-time fast Fourier transform loss of any resolution based on the multi-resolution short-time Fourier transform value of the original speech audio and the multi-resolution short-time Fourier transform value of the restored speech audio; Different resolutions are formed by combining different fast Fourier transform frequencies, jump lengths and window sizes, and the multi-resolution short-time fast Fourier transform loss is calculated by summing and averaging the short-time fast Fourier transform losses of different resolutions.

8. A voice adversarial defense system for a speaker recognition system, characterized in that: include: An extraction module, configured to obtain input speech data, perform feature extraction on the input speech data, and obtain an amplitude spectrogram of the input speech data; a masking module, configured to process the amplitude spectrum through a preset fine-grained non-robust feature filtering module to obtain a masked amplitude spectrum; A reconstruction module is used to reconstruct the masked amplitude spectrum through a U-Net model to obtain a reconstructed amplitude spectrum graph; An estimation module, configured to perform phase estimation from the reconstructed amplitude spectrogram based on a Griffin-Lim algorithm to obtain a reconstructed speech signal; The reconstruction module is specifically used for: Determining to use a feature recovery module of the U-Net model to reconstruct the masked amplitude spectrum, the feature recovery module including a four-layer encoder and a four-layer decoder; The masked amplitude spectrum is input into the encoder, and a one-dimensional convolution operating only along the frequency axis is used. The frequency transform is connected before each encoding layer. Each encoder layer includes two compressed residual branches, and the Snake activation function is used instead of the ReLU activation function. A long short-term memory network and a time-based attention module are added to the internal layer of the encoder. The bottleneck feature is converted back to a complete amplitude spectrum of the same size as the encoder by the decoder; During training, the amplitude spectrum loss and multi-resolution short-time fast Fourier transform loss based on the F-ratio speaker feature distribution are calculated separately, and the weighted summation is used to obtain the total loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech adversarial defense method for a speaker recognition system according to any one of claims 1 to 7 is implemented.

Images