Targeted voice confrontation verification code generation method based on generative adversarial network

Abstract

The invention relates to the technical field of voice verification codes, in particular to a target voice confrontation verification code generation method based on a generative adversarial network, and the method comprises the steps: obtaining a large amount of voice data, enabling the voice data to serve as original clean voice data after standardization preprocessing, and marking a transcription information label; adding disturbance guided by voice activity detection to the original clean voice data to obtain voice data with targeted disturbance preliminarily; adding a room impulse response signal to the voice data with the preliminary targeted disturbance to obtain adversarial voice data with a reverberation effect, and constructing a training sample set according to the adversarial voice data; constructing a generative adversarial network framework composed of a generator and a discriminator; inputting training samples in the training sample set into the generator, and performing confrontation training on the generator and the discriminator until convergence to obtain a trained generator; and inputting the original voice verification code and the confrontation target text into the trained generator to obtain a target voice confrontation verification code.

Description

Technical Field

[0001] The embodiments of this application relate to the field of voice verification code technology, and in particular to a method for generating targeted voice adversarial verification codes based on generative adversarial networks. Background Technology

[0002] Voice verification codes, as a mechanism to distinguish between computer programs and human users, provide important security for assisting visually impaired users in using online services. Their basic principle is to play a voice message and require the user to correctly identify the content, thereby verifying their identity as a human user. However, with the rapid development of artificial intelligence technology, especially the increasing maturity of Automatic Speech Recognition (ASR) technology, traditional voice verification codes face a significant risk of being easily cracked by machine programs. ASR systems can automatically transcribe voice content with high accuracy, allowing malicious programs to automatically bypass voice verification code protection and thus impersonate human users.

[0003] Adversarial attack techniques offer a new approach to improving the security of voice verification codes. This technology generates adversarial examples by adding minute perturbations, imperceptible to the human ear, to the original data. These examples can cause the target machine learning model to produce incorrect outputs, thus effectively resisting automated attacks based on adversarial attacks without significantly impacting the normal user experience, thereby improving the security of network services to a certain extent.

[0004] However, most existing studies still focus on untargeted attacks (i.e., only pursuing ASR identification errors) or passive defense, lacking the ability to actively identify the attacker's identity, and there is still room for improvement in terms of the stealth of generated adversarial examples, audio quality, and model efficiency. Summary of the Invention

[0005] To address the aforementioned technical issues, embodiments of this application propose a targeted voice adversarial CAPTCHA generation method based on generative adversarial networks. This method combines perturbation addition guided by voice activity detection, reverberation effects, and an improved lightweight generative adversarial network to generate voice adversarial CAPTCHAs with high concealment and quality, thereby achieving proactive defense against attackers and improving the security of the voice CAPTCHA system.

[0006] To achieve the above objectives, embodiments of this application propose a method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks (GANs), applicable to attack scenarios of automatic speech recognition systems. The method includes: acquiring a large amount of speech data containing only digit sequences and speech data containing natural language segments; performing standardized preprocessing to obtain raw clean speech data, and labeling it with transcription information tags; performing speech activity detection on the raw clean speech data based on a preset analysis window and energy threshold, generating a binary mask corresponding to the raw clean speech data; applying the binary mask to the original adversarial perturbation to obtain a perturbation guided by speech activity detection, and adding it to the raw clean speech data to obtain preliminary speech data with targeted perturbation; adding a room impulse response signal to the preliminary speech data with targeted perturbation to obtain adversarial speech data with reverberation effects, and then applying the signal to all reverberation-effect adversarial data. A training sample set is constructed using anti-speech data. A generative adversarial network framework based on an improved Wave-U-Net, consisting of a generator and a discriminator, is built. The generator adopts the Wave-U-Net network architecture, and the discriminator adopts a network architecture adapted to Wave-U-Net. The training samples from the training sample set are input into the generator, which generates adversarial samples that satisfy the realism and recognizability conditions based on the training samples. The discriminator distinguishes between real speech and adversarial samples, thereby training the generator and discriminator adversarially until convergence, resulting in a trained generator. The realism condition is used to make the adversarial samples difficult to distinguish from real speech, and the recognizability condition is used to make the target automatic speech recognition system misidentify the adversarial samples with a high probability. The original speech CAPTCHA and the adversarial target text are input into the trained generator to obtain a target speech adversarial CAPTCHA generated by the trained generator.

[0007] To achieve the above objectives, embodiments of this application also propose a targeted speech adversarial CAPTCHA generation system based on generative adversarial networks (GANs), applicable to automatic speech recognition system attack scenarios. The system includes: a data acquisition and preprocessing module, used to acquire a large amount of speech data containing only digit sequences and speech data containing natural language segments. After standardization preprocessing, this data is used as raw clean speech data and labeled with transcription information tags; a preliminary perturbation addition module, used to perform speech activity detection on the raw clean speech data based on a preset analysis window and energy threshold, generating a binary mask corresponding to the raw clean speech data, applying the binary mask to the original adversarial perturbation to obtain a perturbation guided by speech activity detection, and adding it to the raw clean speech data to obtain preliminary speech data with targeted perturbation; and an acoustic environment reverberation integration module, used to add room impulse response signals to the preliminary speech data with targeted perturbation to obtain adversarial speech data with reverberation effects, and based on all... The system comprises the following modules: a reverberation-effect adversarial speech data training sample set; a network construction module for building a generative adversarial network framework based on an improved Wave-U-Net, consisting of a generator and a discriminator, with the generator using the Wave-U-Net network architecture and the discriminator using a network architecture adapted to Wave-U-Net; a network adversarial training module for inputting training samples from the training sample set into the generator, which generates adversarial samples that satisfy realism and recognizability conditions based on the training samples, and the discriminator distinguishes between real speech and adversarial samples, thereby training the generator and discriminator adversarially until convergence, resulting in a trained generator; the realism condition is used to make adversarial samples difficult to distinguish from real speech, and the recognizability condition is used to make the target automatic speech recognition system misidentify adversarial samples with a high probability; and an execution module for inputting the original speech CAPTCHA and adversarial target text into the trained generator to obtain a target speech adversarial CAPTCHA generated by the trained generator.

[0008] To achieve the above objectives, embodiments of this application also propose an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a targeted speech adversarial CAPTCHA generation method as described above based on a generative adversarial network.

[0009] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks as described above.

[0010] This application proposes a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks (GANs), introducing and implementing an active defense mechanism for speech CAPTCHAs based on targeted adversarial attacks. A large amount of speech data containing only digit sequences and speech data containing natural language segments are collected. After standardization preprocessing, this data serves as the raw clean speech data. Perturbations guided by speech activity detection and acoustic reverberation are applied to the raw clean speech data, which is then input into a lightweight GAN architecture, significantly improving the stealth, audio quality, and generation efficiency of the adversarial speech. Through real-world simulation and multi-model ensemble training, the robustness of the speech adversarial CAPTCHA in complex acoustic environments and its transferability to unknown attack models are enhanced, providing a more advanced, efficient, and practical speech CAPTCHA security solution. In constructing the GAN framework, the generator adopts the Wave-U-Net network architecture, and the discriminator adopts a network architecture adapted to Wave-U-Net, which effectively reduces the number of parameters, lowers computational complexity, reduces training time, and adapts to resource-constrained deployment environments. Using a generator trained in this way to generate targeted adversarial voice CAPTCHAs can produce highly covert, high-quality voice CAPTCHAs with specific deceptive targets, effectively achieving proactive defense against attackers and significantly improving the security of the voice CAPTCHA system.

[0011] Optionally, the step of acquiring a large amount of speech data containing only digit sequences and speech data containing natural language segments, and after standardization preprocessing, using it as raw clean speech data and labeling it with transcription information tags, includes: acquiring a large amount of speech data containing only digit sequences and speech data containing natural language segments, and uniformly converting it into standard WAV format as raw speech data; trimming and initially cleaning noise segments from both ends of all raw speech data, and padding both ends of the trimmed and initially cleaned raw speech data with zero values ​​based on a preset maximum length, to obtain length-normalized raw clean speech data. ; Prepare real transcription information for each original clean speech data, and then use DeepSpssch automatic speech recognition system and Lingvo automatic speech recognition system to recognize each original clean speech data to obtain first transcription information and second transcription information; Based on the real transcription information, first transcription information and second transcription information, a transcription information set is formed, and the transcription information set is used as a transcription information label to label the corresponding original clean speech data.

[0012] Optionally, the process involves performing speech activity detection on the original clean speech data based on a preset analysis window and energy threshold, generating a binary mask corresponding to the original clean speech data, applying the binary mask to the original adversarial perturbation to obtain a perturbation guided by speech activity detection, and adding it to the original clean speech data to obtain preliminary speech data with targeted perturbation, including: Based on a preset analysis window, the original clean speech data is sampled, and the original clean speech data is sampled as follows: , This represents a sampling point. For each sampling point, its energy value is checked against a preset energy threshold. If the energy value is greater than the preset energy threshold, a mask value of 1 is assigned; otherwise, a mask value of 0 is assigned. After assigning values ​​to all sampling points, the original clean speech data is obtained. Corresponding binary mask ; Original adversarial perturbation generated based on random noise ; The binary mask is obtained using the following formula. Applied to original counter-perturbation The perturbation was obtained through voice activity detection. ; ; in, This indicates element-wise multiplication; The perturbation guided by voice activity detection is obtained through the following formula. Added to raw clean speech data The above yields preliminary voice data with targeted perturbations. : ; in, This is the preset disturbance amplitude control factor.

[0013] Optionally, adding a room impulse response signal to the initially targeted perturbated speech data to obtain adversarial speech data with reverberation includes: Acquire room impulse response signals through real room measurements or acoustic software simulation. ; The following formula is used to process the initial targeted perturbation of the speech data. and room impulse response signal Perform convolution operations to obtain adversarial speech data with reverberation effects. : ; in, This indicates that a convolution operation is being performed.

[0014] Optionally, generator The Wave-U-Net network architecture is adopted, with its U-shaped encoder-decoder structure and skip connections. Conformer modules are set in the key bottleneck layers of the encoder and decoder to embed the self-attention layer and feedforward network layer into the corresponding positions of the Wave-U-Net. In the Wave-U-Net network architecture, the encoder has 8 upsampling network layers and the decoder has 8 downsampling network layers. Multi-Head Self-Attention modules are deployed in the upsampling link, downsampling link, and upsampling reconstruction parts to adaptively identify and focus on key feature regions in audio data. This breaks through the local receptive field limitation of traditional convolution, realizes global contextual association of audio data, and introduces the compensating feature extraction capability of multi-head attention mechanism. Under the premise of ensuring audio processing quality, it achieves high efficiency and lightweight audio codec network. Discriminator A one-dimensional convolutional neural network structure is used to progressively reduce the dimensionality and extract features from the input audio data, and finally output a scalar value to judge the authenticity of the input audio data.

[0015] Optionally, the training samples in the training sample set Input to generator In, by generator Based on training samples Generate adversarial examples that satisfy both realism and recognizability conditions. The specific recognition condition is to enable the DeepSpssch and Lingvo automatic speech recognition systems to mistakenly identify adversarial examples with a high probability. Identified as target text that differs from the actual content. ; In the generator and discriminator When performing adversarial training, use a composite loss function. As a guide for adversarial training, the composite loss function Specifically, the generator counteracts the loss. DeepSpeech has a target to counteract loss Lingvo has a target to combat losses Regularization loss and additional penalty losses composition; Composite loss function This can be expressed by the formula: ; ; ; ; ; in, , , , , They are respectively , , , , The weighting coefficients, Indicates the noise Expectations for Potential space, Indicates CTC loss. This refers to the DeepSpssch automatic speech recognition system. This represents sequence-to-sequence loss. This refers to the Lingvo automatic speech recognition system. This indicates taking the L2 norm.

[0016] Optionally, the step of inputting the original voice verification code and adversarial target text into a trained generator to obtain a targeted adversarial voice verification code generated by the trained generator includes: after receiving an adversarial verification code request from the requester, selecting a clean original voice data as the original voice verification code, inputting the original voice verification code and adversarial target text into a trained generator to obtain a targeted adversarial voice verification code generated by the trained generator; sending the targeted adversarial voice verification code to the requester, obtaining the requester's recognition result of the targeted adversarial voice verification code, and calculating the word error rate between the recognition result and the real label corresponding to the original voice verification code and the adversarial target text respectively; if the word error rate between the recognition result and the adversarial target text is lower than a preset attack threshold, the requester is determined to be a machine attacker and added to the blacklist; if the word error rate between the recognition result and the real label corresponding to the original voice verification code is lower than a preset user threshold, the requester is determined to be a normal human user. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0018] Figure 1 This is a flowchart of a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks provided in one embodiment of this application; Figure 2 This is a schematic diagram illustrating the details of a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks provided in one embodiment of this application; Figure 3 This is a schematic diagram of a generative adversarial network and adversarial training provided in one embodiment of this application; Figure 4 This is a schematic diagram of a targeted speech adversarial CAPTCHA generation system based on generative adversarial networks provided in another embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0020] One embodiment of this application proposes a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks (GANs), applicable to automatic speech recognition system attack scenarios and applied to electronic devices. The electronic device can be a terminal or a server; this embodiment and subsequent embodiments use a server as an example. The implementation details of the targeted speech adversarial CAPTCHA generation method based on GANs proposed in this embodiment are described below. These implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0021] The specific process of the targeted speech adversarial CAPTCHA generation method based on generative adversarial networks proposed in this embodiment can be described as follows: Figure 1 As shown, its visual details can be as follows Figure 2 As shown, the method includes: Step 11: Obtain a large amount of speech data containing only digit sequences and speech data containing natural language segments. After standardization preprocessing, use it as raw clean speech data and label it with transcription information tags.

[0022] In the specific implementation, the server first needs to acquire a large amount of speech data containing only digit sequences and speech data containing natural language segments as the basis for generating targeted speech adversarial CAPTCHAs. To ensure the effectiveness and consistency of subsequent adversarial training, after collecting the speech data, the server needs to perform standardized preprocessing on all collected speech data to obtain raw clean speech data, and then label each raw clean speech data with transcription information tags.

[0023] In one example, during standardization preprocessing, the acquired speech data first needs to be converted into standard WAV format to serve as the raw speech data. During WAV format conversion, specific parameters are set to mono and a sampling rate of [missing information]. , Bit-integer sampling depth ( , ).

[0024] Some raw speech data may contain non-speech noise segments at both ends. Therefore, the server needs to trim and preliminarily clean the noise segments at both ends of all raw speech data. Raw speech data without noise segments at both ends only needs to be preliminarily cleaned.

[0025] The raw audio data, after trimming and initial cleaning, varies in length. To further standardize it, the server needs to set a preset maximum length. Based on this, zero-value padding is applied to both ends of the original speech data after trimming and initial cleaning to obtain length-normalized original clean speech data. It should be noted that there is a preset maximum length. It must be longer than the length of all the original speech data after cropping and initial cleaning to ensure that the length of all length-normalized original clean speech data is consistent.

[0026] Next, transcription information labels need to be added to each piece of raw, clean speech data.

[0027] First, prepare authentic transcription information for each piece of raw, clean speech data. Then, the DeepSpssch and Lingvo automatic speech recognition systems (using DeepSpssch and Lingvo as baseline automatic speech recognition systems) were used to recognize each piece of raw clean speech data to obtain the first transcription information. Second transcription information .

[0028] Real transcription information First transcription information Second transcription information Similarly, standardization preprocessing and length normalization are required to achieve a uniform maximum text length. .

[0029] Finally, based on real transcription information First transcription information Second transcription information Composition of transcriptional information set , , to transcribe information set As transcriptional information tags, they are labeled onto the corresponding raw, clean speech data.

[0030] It is important to note that when new voice data is added, the new voice data also needs to be standardized and normalized in length to ensure the homogeneity and standardization of the entire dataset, thereby providing a high-quality and standardized data foundation for the subsequent generation of voice adversarial CAPTCHAs.

[0031] Step 12: Based on the preset analysis window and energy threshold, perform speech activity detection on the original clean speech data, generate a binary mask corresponding to the original clean speech data, apply the binary mask to the original adversarial perturbation to obtain the perturbation guided by speech activity detection, and add it to the original clean speech data to obtain preliminary speech data with targeted perturbation.

[0032] In the specific implementation, in order to make the adversarial perturbation more accurately target the speech content and reduce unnecessary interference introduced in non-speech parts, thereby improving the overall concealment of the perturbation, the server needs to perform speech activity detection on the original clean speech data based on a preset analysis window and energy threshold, generate a binary mask corresponding to the original clean speech data, apply the binary mask to the original adversarial perturbation to obtain the perturbation guided by speech activity detection, and add the perturbation guided by speech activity detection to the original clean speech data to obtain the initial speech data with targeted perturbation.

[0033] In one example, the server first uses a pre-defined analysis window. The raw clean speech data is sampled and then sampled as follows: , This represents a sampling point. For each sampling point, it is determined whether its energy value is greater than a preset energy threshold. If the energy value is greater than the preset energy threshold, the portion is determined to be a speech segment and assigned a mask value of 1; if the energy value is not greater than the preset energy threshold, the portion is determined to be a non-speech segment and assigned a mask value of 0. After assigning values ​​to all sampling points, the original clean speech data can be obtained. Corresponding and equal-length binary masks .

[0034] Next, the original adversarial perturbation is generated based on random noise (with certain constraints applied to the random noise). The binary mask is obtained using the following formula. Applied to original counter-perturbation The perturbation was obtained through voice activity detection. ; ; in, This indicates element-wise multiplication.

[0035] Finally, the perturbation guided by voice activity detection is processed using the following formula. Added to raw clean speech data The above yields preliminary voice data with targeted perturbations. : ; in, This is a preset disturbance amplitude control factor used to adjust the strength of the counter-disturbance in order to balance the attack effect and imperceptibility.

[0036] Step 13: Add room impulse response signals to the initial speech data with targeted perturbations to obtain adversarial speech data with reverberation effects, and construct a training sample set based on all adversarial speech data with reverberation effects.

[0037] In the specific implementation, in order to make the generated adversarial voice verification code closer to the effect of playing and recording in a real physical environment, and to improve its naturalness and robustness in practical applications, this embodiment designs a method of adding room impulse response signals to the initial voice data with targeted perturbation to obtain adversarial voice data with reverberation effect. All adversarial voice data with reverberation effect are used as training samples to construct a training sample set (a test sample set and a verification sample set can also be constructed at the same time).

[0038] In one example, when adding a room impulse response signal, it is first necessary to obtain the room impulse response signal through actual room measurements or acoustic software simulation. Room impulse response signal It provides a good description of the sound response characteristics of a specific acoustic space (such as a room or hall).

[0039] Next, the server can use the following formula to process the initially targeted perturbation of the voice data. and room impulse response signal Perform convolution operations to obtain adversarial speech data with reverberation effects. : ; in, This indicates that a convolution operation is being performed.

[0040] Understandably, the process of adding room impulse response signals, by simulating the reflection and attenuation of sound within the room, gives it a more realistic listening experience and may also enhance its effectiveness against attacks.

[0041] Step 14: Construct a generative adversarial network framework based on an improved Wave-U-Net, consisting of a generator and a discriminator. The generator adopts the Wave-U-Net network architecture, and the discriminator adopts a network architecture adapted to Wave-U-Net.

[0042] In the specific implementation, while building the training sample set, the server also needs to simultaneously build a generative adversarial network framework based on the improved Wave-U-Net, consisting of a generator and a discriminator. In this generative adversarial network framework, the generator adopts the Wave-U-Net network architecture, while the discriminator needs to adopt a network architecture adapted to Wave-U-Net.

[0043] In one example, the generator The Wave-U-Net network architecture is adopted. It is a fully convolutional neural network designed specifically for temporal processing of audio signals. Its U-shaped encoder-decoder structure, combined with skip connections, can effectively learn the multi-scale features of audio signals and perform accurate reconstruction.

[0044] To enhance the generator To enhance the modeling capabilities of long-term dependencies in audio signals and the fine-grained capture of local features, this embodiment incorporates a Conformer module in the critical bottleneck layers of the encoder and decoder. This module embeds self-attention layers and feedforward network layers into the corresponding positions of the Wave-U-Net. The Conformer module cleverly combines the local feature extraction advantages of convolutional neural networks with the global context modeling capabilities of the Transformer's self-attention mechanism, thus helping to generate higher-quality audio.

[0045] To reduce the number of generator parameters and computational complexity, shorten training time, and adapt to resource-constrained deployment environments, this embodiment makes lightweight adjustments to the original structure of Wave-U-Net.

[0046] In terms of lightweight network architecture design, the number of upsampling network layers in the encoder was reduced from 12 to 8, and the number of downsampling network layers in the decoder was also reduced from 12 to 8. This lightweight adjustment significantly reduced the number of parameters from 278.33MB to 106.18MB, achieving a compression rate of 62%.

[0047] Regarding the integration of multi-head self-attention mechanisms, multi-head self-attention modules are deployed in the upsampling linking, downsampling linking, and upsampling reconstruction parts. This mechanism can adaptively identify and focus on key feature regions in audio data, breaking through the limitations of the local receptive field of traditional convolutions and achieving global contextual association of audio data. Simultaneously, the introduction of multi-head attention compensates for the potential decrease in feature extraction capability due to the reduction in the number of network layers. While maintaining and improving audio processing quality, it combines traditional deep convolutional networks with modern attention mechanisms, achieving high efficiency and lightweight audio codec networks through a "depth for attention" design approach.

[0048] Discriminator Its function is to distinguish between real, high-quality audio data and data generated by the generator. The generated adversarial data typically employs a one-dimensional convolutional neural network structure. Its function is to progressively reduce the dimensionality and extract features from the input audio data, ultimately outputting a scalar value to determine the authenticity of the input audio data. (Compared to the generator...) Unlike other structural innovations, the discriminator in this embodiment... A conventional discriminator network structure adapted to the benchmark Wave-U-Net can be used.

[0049] Step 15: Input the training samples from the training sample set into the generator. The generator generates adversarial samples that meet the realism and recognition conditions based on the training samples. The discriminator distinguishes between real speech and adversarial samples, thereby performing adversarial training on the generator and discriminator until convergence, and obtaining the trained generator.

[0050] In the specific implementation, after the generative adversarial network framework is constructed, adversarial training can begin. The server inputs training samples from the training sample set into the generator, which then generates adversarial samples that satisfy both realism and discriminability conditions based on the training samples. The discriminator distinguishes between real speech and adversarial samples, thus training the generator and discriminator adversarially until convergence, resulting in a trained generator. The realism condition is used to make adversarial samples difficult to distinguish from real speech, while the discriminability condition is used to cause the target automatic speech recognition system to incorrectly identify adversarial samples with a high probability.

[0051] In one example, the generative adversarial network framework and adversarial training process can be as follows: Figure 3 As shown.

[0052] The purpose of adversarial training is to optimize the generator. and discriminator The parameters will be the training samples in the training sample set. Input to generator In, the generator Capable of outputting adversarial examples that satisfy both realism and recognizability conditions. The authenticity condition means that it sounds natural and clear, and is difficult to distinguish from real speech. The recognition condition specifically means that the DeepSpssch automatic speech recognition system and the Lingvo automatic speech recognition system should mistakenly identify adversarial examples with a high probability. Identified as target text that differs from the actual content. By modifying the generator and discriminator Adversarial training enables the generator Its performance is constantly improving.

[0053] In the generator and discriminator When performing adversarial training, use a composite loss function. As a guide for adversarial training, the composite loss function Specifically, the generator counteracts the loss. DeepSpeech has a target to counteract loss Lingvo has a target to combat losses Regularization loss and additional penalty losses composition. and The design makes the generator Generated adversarial examples It can perform targeted attacks on DeepSpssch and Lingvo automatic speech recognition systems.

[0054] In one example, the composite loss function This can be expressed by the formula: ; ; ; ; ; in, , , , , They are respectively , , , , The weighting coefficients, Indicates the noise Expectations for Potential space, Indicates CTC loss. This refers to the DeepSpssch automatic speech recognition system. This represents sequence-to-sequence loss. This refers to the Lingvo automatic speech recognition system. This indicates taking the L2 norm.

[0055] Step 16: Input the original voice verification code and the adversarial target text into the trained generator to obtain the target voice adversarial verification code generated by the trained generator.

[0056] In practice, after training the generator, the server can input the original voice CAPTCHA and the adversarial target text into the trained generator to obtain the target voice adversarial CAPTCHA generated by the trained generator.

[0057] In one example, the core of this embodiment for proactive defense lies in using the characteristics of targeted adversarial voice CAPTCHAs to identify attackers. After receiving the adversarial CAPTCHA request from the requester, the server selects a clean, original voice data as the original voice CAPTCHA, and inputs the original voice CAPTCHA and the adversarial target text into the trained generator to obtain the targeted adversarial voice CAPTCHA generated by the trained generator.

[0058] Next, the targeted adversarial voice CAPTCHA is sent to the requester, the requester's recognition result of the targeted adversarial voice CAPTCHA is obtained, and the word error rate (WER) between the recognition result and the real label and adversarial target text corresponding to the original voice CAPTCHA is calculated respectively.

[0059] Word-level edit distance (WER) is a standard evaluation metric in the field of speech recognition technology. It measures the difference between two texts by calculating the minimum number of word substitutions, deletions, and insertions required to transform one text into another. The formula for WER is: ; in, , , These represent the number of word replacements, deletions, and insertions, respectively. Total word count The lower the value, the higher the similarity.

[0060] Specifically, if the word error rate between the recognition result and the adversarial target text is lower than a preset attack threshold, the requester is determined to be a machine attacker and added to the blacklist. If the word error rate between the recognition result and the real label corresponding to the original voice verification code is lower than a preset user threshold, the requester is determined to be a normal human user. The preset attack threshold and the preset user threshold are generally both set to 0.1.

[0061] It is important to note that once an attacker is identified as a machine attacker, the system will immediately trigger a defense response, including measures such as denial of service, recording the attacker's IP information, and adding the attacker to a blacklist.

[0062] In one example, if the word error rate between the recognition result and the adversarial target text is greater than or equal to the preset attack threshold, and the word error rate between the recognition result and the real label corresponding to the original voice verification code is greater than or equal to the preset user threshold, then a new targeted voice adversarial verification code needs to be sent to the requester for secondary judgment until it can be determined whether the requester is a machine attacker or a human user.

[0063] This embodiment proposes a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks (GANs), introducing and implementing an active defense mechanism for speech CAPTCHAs based on targeted adversarial attacks. A large amount of speech data containing only digit sequences and speech data containing natural language segments are collected. After standardization preprocessing, this data serves as the raw clean speech data. Perturbations guided by speech activity detection and acoustic reverberation are applied to the raw clean speech data, which is then input into a lightweight GAN architecture, significantly improving the stealth, audio quality, and generation efficiency of the adversarial speech. Through real-world simulation and multi-model ensemble training, the robustness of the speech adversarial CAPTCHA in complex acoustic environments and its transferability to unknown attack models are enhanced, providing a more advanced, efficient, and practical speech CAPTCHA security solution. The constructed GAN framework uses a Wave-U-Net network architecture for the generator and a network architecture adapted to Wave-U-Net for the discriminator, effectively reducing the number of parameters, lowering computational complexity, thus reducing training time, and adapting to resource-constrained deployment environments. Using a generator trained in this way to generate targeted adversarial voice CAPTCHAs can produce highly covert, high-quality voice CAPTCHAs with specific deceptive targets, effectively achieving proactive defense against attackers and significantly improving the security of the voice CAPTCHA system.

[0064] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0065] Another embodiment of this application proposes a targeted speech adversarial CAPTCHA generation system based on generative adversarial networks, which is applicable to automatic speech recognition system attack scenarios. The details of the targeted speech adversarial CAPTCHA generation system based on generative adversarial networks proposed in this embodiment are described below. The following content is only for the convenience of understanding and is not necessary for implementing this example. Figure 4 This is a schematic diagram of the structure of a target speech adversarial CAPTCHA generation system based on generative adversarial networks proposed in this embodiment, including: a data acquisition and preprocessing module 21, a preliminary perturbation addition module 22, an acoustic environment reverberation integration module 23, a network construction module 24, a network adversarial training module 25, and an execution module 26.

[0066] The data acquisition and preprocessing module 21 is used to acquire a large amount of speech data containing only digit sequences and speech data containing natural language segments. After standardization preprocessing, it is used as raw clean speech data and labeled with transcription information tags.

[0067] The preliminary perturbation addition module 22 is used to perform speech activity detection on the original clean speech data based on a preset analysis window and energy threshold, generate a binary mask corresponding to the original clean speech data, apply the binary mask to the original adversarial perturbation, obtain the perturbation guided by speech activity detection, and add it to the original clean speech data to obtain the initial speech data with targeted perturbation.

[0068] The acoustic environment reverberation integration module 23 is used to add room impulse response signals to the initial speech data with targeted perturbation to obtain adversarial speech data with reverberation effect, and to construct a training sample set based on all adversarial speech data with reverberation effect.

[0069] Network building module 24 is used to build a generative adversarial network framework based on an improved Wave-U-Net, consisting of a generator and a discriminator. The generator adopts the Wave-U-Net network architecture, and the discriminator adopts a network architecture adapted to Wave-U-Net.

[0070] The network adversarial training module 25 is used to input training samples from the training sample set into the generator. The generator generates adversarial samples that meet the realism and recognition conditions based on the training samples. The discriminator distinguishes between real speech and adversarial samples, thereby training the generator and discriminator adversarially until convergence, and obtaining the trained generator. The realism condition is used to make it difficult to distinguish between adversarial samples and real speech, and the recognition condition is used to make the target automatic speech recognition system misidentify adversarial samples with a high probability.

[0071] The execution module 26 is used to input the original voice verification code and the adversarial target text into the trained generator to obtain the target voice adversarial verification code generated by the trained generator.

[0072] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0073] It is worth mentioning that all modules and units involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.

[0074] Another embodiment of this application provides an electronic device, such as Figure 5 As shown, it includes: at least one processor 31; and a memory 32 communicatively connected to the at least one processor 31; wherein the memory 32 stores instructions executable by the at least one processor 31, the instructions being executed by the at least one processor 31 to enable the at least one processor 31 to perform a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks as described in the above method embodiment.

[0075] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0076] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0077] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a targeted speech adversarial CAPTCHA generation method based on generative adversarial networks as described in the above method embodiments.

[0078] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0079] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for generating targeted adversarial voice CAPTCHAs based on generative adversarial networks, applicable to attack scenarios of automatic speech recognition systems, characterized in that, The method includes: We acquired a large amount of speech data containing only digit sequences and speech data containing natural language segments. After standardization and preprocessing, the data was used as raw clean speech data and labeled with transcription information tags. Based on a preset analysis window and energy threshold, speech activity detection is performed on the original clean speech data to generate a binary mask corresponding to the original clean speech data. The binary mask is applied to the original adversarial perturbation to obtain a perturbation guided by speech activity detection, which is then added to the original clean speech data to obtain preliminary speech data with targeted perturbation. Room impulse response signals are added to the initial speech data with targeted perturbations to obtain adversarial speech data with reverberation effects, and a training sample set is constructed based on all adversarial speech data with reverberation effects. A generative adversarial network framework consisting of a generator and a discriminator is constructed based on an improved Wave-U-Net. The generator adopts the Wave-U-Net network architecture, and the discriminator adopts a network architecture adapted to Wave-U-Net. The training samples in the training sample set are input into the generator, which generates adversarial samples that meet the realism and recognition conditions based on the training samples. The discriminator distinguishes between real speech and adversarial samples, thereby training the generator and discriminator adversarially until convergence, resulting in a trained generator. The realism condition is used to make adversarial samples difficult to distinguish from real speech, and the recognition condition is used to make the target automatic speech recognition system incorrectly recognize adversarial samples with a high probability. The original voice CAPTCHA and the adversarial target text are input into the trained generator to obtain the target voice adversarial CAPTCHA generated by the trained generator.

2. The method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks according to claim 1, characterized in that, The process involves acquiring large amounts of speech data containing only digit sequences and speech data containing natural language segments. After standardization preprocessing, this data is used as raw, clean speech data, which is then labeled with transcription information tags, including: Acquire large amounts of speech data containing only digit sequences and speech data containing natural language segments, and convert them uniformly into standard WAV format as raw speech data; Noise segments are trimmed and preliminarily cleaned from both ends of all raw speech data. Then, zero-value padding is applied to both ends of the trimmed and preliminarily cleaned raw speech data, based on a preset maximum length, to obtain length-normalized raw clean speech data. ; Prepare real transcription information for each original clean speech data, and then use DeepSpssch automatic speech recognition system and Lingvo automatic speech recognition system to recognize each original clean speech data to obtain the first transcription information and the second transcription information. The transcription information set is composed of real transcription information, first transcription information, and second transcription information. The transcription information set is used as a transcription information label and assigned to the corresponding original clean speech data.

3. The method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks according to claim 2, characterized in that, The process involves performing speech activity detection on the original clean speech data based on a preset analysis window and energy threshold, generating a binary mask corresponding to the original clean speech data, applying the binary mask to the original adversarial perturbation to obtain a perturbation guided by speech activity detection, and adding it to the original clean speech data to obtain preliminary speech data with targeted perturbation, including: Based on a preset analysis window, the original clean speech data is sampled, and the original clean speech data is sampled as follows: , This represents a sampling point. For each sampling point, its energy value is checked against a preset energy threshold. If the energy value is greater than the preset energy threshold, a mask value of 1 is assigned; otherwise, a mask value of 0 is assigned. After assigning values ​​to all sampling points, the original clean speech data is obtained. Corresponding binary mask ; Original adversarial perturbation generated based on random noise ; The binary mask is obtained using the following formula. Applied to original counter-perturbation The perturbation was obtained through voice activity detection. ; ； in, This indicates element-wise multiplication; The perturbation guided by voice activity detection is obtained through the following formula. Added to raw clean speech data The above yields preliminary voice data with targeted perturbations. : ； in, This is the preset disturbance amplitude control factor.

4. The method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks according to claim 3, characterized in that, The process of adding a room impulse response signal to the initially targeted perturbated speech data to obtain adversarial speech data with reverberation includes: Acquire room impulse response signals through real room measurements or acoustic software simulation. ; The following formula is used to process the initial targeted perturbation of the speech data. and room impulse response signal Perform convolution operations to obtain adversarial speech data with reverberation effects. : ； in, This indicates that a convolution operation is being performed.

5. A method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks according to claim 4, characterized in that, generator The Wave-U-Net network architecture is adopted, with its U-shaped encoder-decoder structure and skip connections. Conformer modules are set in the key bottleneck layers of the encoder and decoder to embed the self-attention layer and feedforward network layer into the corresponding positions of the Wave-U-Net. In the Wave-U-Net network architecture, the encoder has 8 upsampling network layers and the decoder has 8 downsampling network layers. Multi-Head Self-Attention modules are deployed in the upsampling link, downsampling link, and upsampling reconstruction parts to adaptively identify and focus on key feature regions in audio data. This breaks through the local receptive field limitation of traditional convolution, realizes global contextual association of audio data, and introduces the compensating feature extraction capability of multi-head attention mechanism. Under the premise of ensuring audio processing quality, it achieves high efficiency and lightweight audio codec network. Discriminator A one-dimensional convolutional neural network structure is used to progressively reduce the dimensionality and extract features from the input audio data, and finally output a scalar value to judge the authenticity of the input audio data.

6. The method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks according to claim 5, characterized in that, Training samples in the training sample set Input to generator In, by generator Based on training samples Generate adversarial examples that satisfy both realism and recognizability conditions. The specific recognition condition is to enable the DeepSpssch and Lingvo automatic speech recognition systems to mistakenly identify adversarial examples with a high probability. Identified as target text that differs from the actual content. ; In the generator and discriminator When performing adversarial training, use a composite loss function. As a guide for adversarial training, the composite loss function Specifically, the generator counteracts the loss. DeepSpeech has a target to counteract loss. Lingvo has a target to combat losses Regularization loss and additional penalty losses composition; Composite loss function This can be expressed by the formula: ； ； ； ； ； in, , , , , They are respectively , , , , The weighting coefficients, Indicates noise Expectations for Potential space, Indicates CTC loss. This refers to the DeepSpssch automatic speech recognition system. This represents sequence-to-sequence loss. This refers to the Lingvo automatic speech recognition system. This indicates taking the L2 norm.

7. A method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks according to claim 6, characterized in that, The process of inputting the original voice CAPTCHA and the adversarial target text into a trained generator to obtain a target-based adversarial voice CAPTCHA generated by the trained generator includes: After receiving the adversarial CAPTCHA request from the requester, select a clean original speech data as the original speech CAPTCHA, input the original speech CAPTCHA and the adversarial target text into the trained generator, and obtain the target speech adversarial CAPTCHA generated by the trained generator. The target speech adversarial CAPTCHA is sent to the requester. The requester's recognition result of the target speech adversarial CAPTCHA is obtained, and the word error rate between the recognition result and the real label and adversarial target text corresponding to the original speech CAPTCHA is calculated respectively. If the word error rate between the recognition result and the adversarial target text is lower than the preset attack threshold, the requester is determined to be a machine attacker and added to the blacklist. If the word error rate between the recognition result and the real label corresponding to the original voice verification code is lower than the preset user threshold, the requester is determined to be a normal human user.

8. A targeted speech adversarial CAPTCHA generation system based on generative adversarial networks, suitable for attack scenarios on automatic speech recognition systems, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire a large amount of speech data containing only digit sequences and speech data containing natural language segments. After standardization preprocessing, it is used as raw clean speech data and labeled with transcription information tags. The initial perturbation addition module is used to perform speech activity detection on the original clean speech data based on a preset analysis window and energy threshold, generate a binary mask corresponding to the original clean speech data, apply the binary mask to the original adversarial perturbation to obtain a perturbation guided by speech activity detection, and add it to the original clean speech data to obtain speech data with initial targeted perturbation. The acoustic environment reverberation integration module is used to add room impulse response signals to the initial speech data with targeted perturbation to obtain adversarial speech data with reverberation effect, and to build a training sample set based on all adversarial speech data with reverberation effect. The network building module is used to build a generative adversarial network framework based on the improved Wave-U-Net, consisting of a generator and a discriminator. The generator adopts the Wave-U-Net network architecture, and the discriminator adopts a network architecture adapted to Wave-U-Net. The network adversarial training module is used to input training samples from the training sample set into the generator. The generator generates adversarial samples that meet the realism and recognizability conditions based on the training samples. The discriminator distinguishes between real speech and adversarial samples, thereby training the generator and discriminator adversarially until convergence, resulting in a trained generator. The realism condition is used to make adversarial samples difficult to distinguish from real speech, and the recognizability condition is used to make the target automatic speech recognition system misidentify adversarial samples with a high probability. The execution module is used to input the original voice CAPTCHA and the adversarial target text into the trained generator to obtain the target voice adversarial CAPTCHA generated by the trained generator.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a targeted speech adversarial CAPTCHA generation method based on any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a method for generating targeted speech adversarial CAPTCHAs based on generative adversarial networks as described in any one of claims 1 to 7.

Images