A defense training method against adversarial samples of a speaker recognition system
By constructing a dataset and extracting features using FaceNet and SpeechTokenizer, and combining multi-head attention mechanism to generate fused features, the speaker recognition system was retrained, thus solving the problem of adversarial speech attacks and improving the system's defense capabilities and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2025-01-03
- Publication Date
- 2026-07-03
AI Technical Summary
Existing speaker recognition systems are ineffective against adversarial speech attacks, which may lead to information leakage and unauthorized operations. Furthermore, traditional defense methods are computationally expensive or have unsatisfactory countermeasures.
By constructing a dataset and extracting facial and voice features using FaceNet and SpeechTokenizer, and generating fused features using an improved multi-head attention mechanism, the speaker recognition system was retrained to enhance its defense against unknown audio adversarial examples.
Under unknown attack methods, the robustness of the speaker recognition system is enhanced, effectively defending against various adversarial speech attacks and reducing computational costs and resource requirements.
Smart Images
Figure CN122337210A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech system technology, specifically to the field of adversarial example defense in speaker recognition, and particularly to a novel feature fusion defense training method based on speaker facial features. Technical Background
[0002] Voice authentication, as a novel security mechanism, is gradually replacing traditional password authentication in scenarios such as voice control systems (VCS), personal smart devices, and telephone banking due to its efficiency and convenience. However, with the development of adversarial attacks, adversarial examples generated in certain scenarios still pose a significant threat to speaker recognition systems. The effectiveness of defense against audio adversarial examples can affect the reliability of downstream tasks and may indirectly lead to problems such as information leakage and unauthorized operations. Therefore, defense against audio adversarial examples remains an important research topic in the field of adversarial science.
[0003] Adversarial defense includes two types: active defense and passive defense. Active defense refers to increasing the robustness of the system through retraining and data augmentation. Passive defense, without changing the model structure, uses pre-emptive measures, including but not limited to downsampling and noise addition. This method eliminates the influence of adversarial examples by detecting or disrupting adversarial perturbations. Summary of the Invention
[0004] The purpose of this invention is to propose an active defense scheme for training a speaker recognition system (SRS) using fused features from facial images to guide audio. This method fuses identity consistency information from the speaker's face with audio features and uses these fused features to train the SRS, enabling successful defense against unknown adversarial audio examples.
[0005] In view of the above objectives, the technical solution of the present invention is as follows:
[0006] Construct a dataset, generate fused features, retrain the speaker system using the fused features, and verify the defense effectiveness.
[0007] The more specific steps are as follows:
[0008] The dataset was constructed by finding 20 long video clips of different speakers' frontal faces from a public data pool. Silent video clips were filtered out using the FSMN-VAD sound activity detection model. Video clips of individual speakers were spliced together and stored in the directories of each speaker. TransNetv2 was used to segment the video clips, retaining video clips with a duration of 3 to 10 seconds. The cleaned data was divided into training and testing sets in a ratio of 6:4.
[0009] The generated fusion features are generated using FaceNet and SpeechTokenizer as feature extractors before fusion. Before face image extraction, Openface is used to extract facial landmark features. Face information and speech are converted into vectors by the feature extractors and then concatenated. An improved multi-head attention mechanism is used to capture cross-modal consistency information. The fused vector retains the audio portion and is mapped to the speaker system's retraining vector through a linear layer. The specific formula for the multi-head attention mechanism is as follows:
[0010]
[0011] Multihead(Q,K,V)=Concat(head1+head2+head3)·W o
[0012] Among them, given query The calculation method for each attention point is h. i , W o The weight matrix is learnable, and the number of attention heads is 4. After feature fusion, the video features in the first half are discarded, while the audio features in the second half are aligned with the linear layer dimensions to generate fused features for retraining the speaker recognition system.
[0013] The defense effectiveness was verified by generating 500 adversarial samples each using the attack methods FGSM, PGD, CW, FakeBob, SirenAttack, and Kenansville on the test set. Comparative experiments were conducted on the defense methods of mean smoothing, adding noise, downsampling, median smoothing, downsampling, adversarial training, and feature fusion retraining.
[0014] As can be seen from the above, this invention utilizes the identity consistency information between face and voice to propose a novel feature fusion method to generate fused features for retraining the speaker recognition system. The retrained model can verify adversarial audio generated by unknown attack methods with only the face image of the registered speaker. This solves the computational overhead and overfitting problems caused by generating adversarial samples for adversarial training against a certain type of adversarial attack. By enhancing identity consistency information, the robustness of the speaker recognition system is improved. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is the overall flowchart of the present invention;
[0017] Figure 2 A flowchart for constructing the dataset of this invention;
[0018] Figure 3 This is a diagram of the retraining structure of the present invention;
[0019] Figure 4 This is a structural diagram of the feature fusion module of the present invention;
[0020] Figure 5 Waveforms and Mel spectrograms generated for time-domain defense methods
[0021] Figure 6 Waveforms and Mel spectrograms generated for frequency domain defense methods Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.
[0023] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0024] See Figure 1The overall process of this invention is as follows: acquire facial images and audio of the speaker to construct a dataset; vectorize the images and audio; fuse the vectorized features using an improved multi-head attention mechanism; retrain the speaker recognition system using the fused features; and perform robustness testing on the retrained model on a test set.
[0025] More specific steps are as follows:
[0026] Step 1: In the dataset construction method, see... Figure 2 First, long video data of single-shot frontal faces of 20 different speakers were selected from the public data pool. Next, the voice activity detection model, i.e., FSMN-VAD technology, was applied to process the selected video data to filter out silent video segments. Then, the effective video segments of each individual speaker were spliced, and the spliced results were stored in the corresponding directories of each speaker. Subsequently, TransNet v2 technology was used to perform video segmentation on the spliced video, retaining segment segments with a duration between 3 and 10 seconds. Finally, the cleaned segment segments were divided into training and test sets in a 6:4 ratio for subsequent use.
[0027] Step 2: In the process of vectorizing facial and audio information, facial information is extracted using OpenFace to extract facial key point features; the extracted information and audio are then vectorized using FaceNet and SpeechTokenizer respectively to obtain feature vectors for fusion.
[0028] Step 3: During the feature fusion process, see... Figure 4 Facial information and speech are converted into vectors by a feature extractor and then concatenated. An improved multi-head attention mechanism is used to capture cross-modal consistency information. The fused vector retains the audio portion and is mapped to the speaker system's retraining vector through a linear layer. The specific formula for the multi-head attention mechanism is as follows:
[0029]
[0030] Multihead(Q,K,V)=Concat(head1+head2+head3)·W o
[0031] Among them, given query The calculation method for each attention point is h. i , W oThe weight matrix is learnable, and the number of attention heads is 4. After feature fusion, the video features in the first half are discarded, while the audio features in the second half are aligned with the linear layer dimensions to generate fused features for retraining the speaker recognition system.
[0032] Table 1 shows the attack success rate (ASR) of various adversarial attacks; the baseline represents the case without any defense; the attack methods mentioned in the table include: FGSM (Fast Gradient Sign Method), PGD (Projected Gradient Descent), CW (Carlini & Wagner), FakeBob, SirenAttack, and Kenansville (an attack method that decomposes and thresholds audio signals); the defense methods mentioned in the table include: Quantization (QT), Audio Turbulence (AT) with added noise, Average Smoothing (AS), Median Smoothing (MS), Down Sampling (DS), LPF low-pass filtering, BPF band-pass filtering, adversarial training, and feature fusion retraining defense methods; among them, FakeBob, SirenAttack, and Kenansville are black-box attack methods, and due to the high computational cost, a sufficient number of adversarial samples cannot be generated to serve as training data during adversarial training.
[0033] As shown in Table 1, the noise-adding defense method performs better against various adversarial examples compared to quantization and smoothing. However, it is less effective against sophisticated attacks (such as Kenansville), with a success rate as high as 56.2%. Furthermore, [the text abruptly ends here, likely due to an incomplete sentence or missing information]. Figure 5 It is evident that adding noise significantly damages the original audio, potentially affecting downstream tasks. Adversarial training can effectively defend against optimization-based white-box attacks to some extent. The process involves two stages: first, generating adversarial samples using known attack methods; then, using these samples along with the original samples to retrain the speaker recognition system. This process requires significant computational resources and time, especially when facing attacks that dynamically search for optimal solutions, making it impractical to construct a large number of black-box attack samples for training. In contrast, after adopting the retraining defense method proposed in this invention, the success rate of all attack methods drops to 0.
[0034] Table 1
[0035]
[0036] In summary, the proactive defense scheme proposed in this invention, which uses fusion features of facial images to guide audio training for speaker systems, is more effective than traditional defense methods in identity verification scenarios where speaker facial data can be obtained. Furthermore, by providing only the speaker's facial visual information, it can enhance the robustness of the speaker recognition system even with unknown attack methods, defending against adversarial speech samples generated by different attack approaches.
Claims
1. A novel feature fusion defense training method based on speaker facial features, characterized in that, Audio features that integrate facial identity consistency information are introduced for the training of the speaker recognition system, thus maintaining the accuracy and robustness of the speaker recognition system. It consists of a single-modality coding module and a feature fusion module with an improved multi-head attention mechanism; Step 1: Construct the experimental dataset; Step 2: Vectorize image and audio information using a single-modal coding module; Step 3: Concatenate the image and audio vectors and generate fused features using an improved multi-head attention mechanism feature fusion module; Step 4: Train the speaker recognition system using the fused features; Step 5: Perform performance testing on the trained model and defend against adversarial examples generated by FGSM, PGD, CW, FakeBob, SirenAttack, and Kenansville.
2. The method according to claim 1, characterized in that, The single-modal coding module includes a face visual information coding module, which uses FaceNet for feature vectorization; and an audio vector discretization module, which uses SpeechTokenizer for feature vectorization.
3. The method according to claim 1, characterized in that, The improved multi-head attention mechanism feature fusion module includes concatenating visual feature vectors and audio feature vectors, fusing them using a four-head multi-head attention mechanism, discarding the vectors of the fused features relative to the image portion, and retaining the audio feature vectors which are mapped through a linear layer to the corresponding vector dimension of the speaker recognition system for retraining.
4. The method according to claim 3, wherein the specific formula of the multi-head attention mechanism is as follows: Multihead(Q,K,V)=Concat(head1+head2+head3)·W o in, Given h i For a single attention head, Attention is used to calculate the attention score, and Multihead is used for multi-head attention mechanisms. W o This is a learnable weight matrix.
5. The method according to claim 1, characterized in that, The dataset construction includes, (1) Download 20 long videos of different speakers, each with a single-camera frontal view of their face; (2) Use the fsmn-vad model to filter silent video clips; (3) Use Transnetv2 to segment the video, filter out shots with a duration of 3 to 10 seconds using rules, and construct a dataset by taking the middle frame, audio, and speaker ID of the shot. (4) The dataset is divided into a training set and a test set ratio of 6:
4. The test set data is used for subsequent adversarial attack testing.
6. The method according to claim 1, characterized in that, Step 5 specifically involves generating adversarial examples on the constructed test set using FGSM, PGD, CW, FakeBob, SirenAttack, and Kenansville, and then testing the defense effectiveness on the trained speaker recognition system.