A black-box adversarial audio generation method and system

Abstract

This invention provides a black-box adversarial audio generation method and system, relating to the field of information security technology. The method first standardizes the original audio, generating speech attribute parameter vectors using a random initialization method to construct an initial population. Then, the generated speech attribute parameter vectors are input into an attribute conditional vocoder to synthesize initial adversarial audio, and a multi-objective fitness function is constructed to evaluate the initial adversarial audio. Next, a multi-objective evolutionary algorithm is used to iteratively optimize the speech attribute parameter vectors in the initial population based on the multi-objective fitness function value, outputting the optimized multi-objective population result. The optimal speech attribute parameter vector is determined after screening using the TOPSIS method. Finally, the optimal speech attribute parameter vector is fused with the original audio to generate the final adversarial audio sample. The adversarial audio samples generated by this invention can be directly used for adversarial training and defense strategy evaluation, thereby improving the robustness and security protection capabilities of automatic speech recognition models.

Description

Technical Field

[0001] This invention relates to the field of information security technology, specifically to a black-box adversarial audio generation method and system. Background Technology

[0002] With the rapid development of deep neural network (DNN) technology, automatic speech recognition (ASR) systems have been widely deployed in various voice interaction scenarios, including intelligent voice assistants (such as Alexa, Cortana, Siri, and Xiao Ai), in-vehicle voice control systems, real-time captioning and transcription services, voice search, and voice payment. Compared to traditional statistical modeling methods such as hidden Markov models (HMM) and Gaussian mixture models (GMM), DNN-based ASR systems possess stronger feature representation and context modeling capabilities in acoustic modeling, language modeling, and end-to-end recognition, significantly improving recognition accuracy, robustness, and adaptability, thus promoting the popularization and commercialization of voice interaction technology.

[0003] However, DNN-based ASR systems also face serious security vulnerabilities. Existing research shows that such systems are extremely vulnerable to adversarial examples. Attackers can superimpose carefully crafted micro-perturbations onto legitimate speech, making them almost indistinguishable to the human ear, yet the ASR system's transcription result is maliciously manipulated to represent the attacker's predetermined target text. For example, a voice command "turn on the air conditioner" might be interpreted as "open the door" after perturbation, potentially leading to property damage or even personal safety risks. Such attacks can be embedded in local audio files or played through speakers in physical environments, possessing strong concealment and posing a real threat, making it a crucial problem that urgently needs to be addressed in the field of voice security.

[0004] Adversarial attack techniques against ASR systems can be divided into two categories: white-box attacks and black-box attacks. White-box attacks allow attackers to directly obtain model architecture, parameters, and gradient information, enabling efficient perturbation generation through backpropagation. However, this is almost impossible to achieve in real-world commercial ASR services. Black-box attacks are closer to real-world scenarios, where attackers can only construct perturbations based on input-output interactions. However, existing black-box attack methods generally suffer from the following shortcomings: (1) Most methods perform perturbations at the waveform level or short-term spectrum level, resulting in a high search space dimension and high optimization difficulty. A large number of queries (often thousands) are required to achieve the target, leading to high attack costs and easy triggering of system-side risk control and flow restriction mechanisms; (2) Some prosodic statistical methods generate adversarial examples by adjusting global features such as pitch, duration, and energy. Although this improves the similarity to the original speech to some extent, the perturbation method does not completely match the natural attributes of the speech, which easily introduces unnatural prosodic changes or subtle timbre shifts, thus being perceived by the human ear at the perceptual level and more easily identified by detection mechanisms based on audio quality or spectrum anomalies; (3) There is a lack of effective multi-objective balancing mechanisms, which often favor one indicator of attack success rate or audio naturalness, sacrificing query efficiency and making it difficult to achieve organic unity among the three. The above problems largely restrict the generation efficiency and concealment of adversarial examples, and also limit their application value in adversarial training, defense evaluation, and robustness improvement. Summary of the Invention

[0005] Purpose of the invention: The first purpose of this invention is to provide a black-box adversarial audio generation method based on multi-objective optimization of speech attribute parameter vectors, and the second purpose is to provide a black-box adversarial audio generation system.

[0006] Technical solution: A black-box adversarial audio generation method, comprising the following steps:

[0007] S1. Standardize the original audio and use a random initialization method to generate speech attribute parameter vectors within the set range of speech rate coefficient, pitch contour coefficient, and loudness envelope coefficient to construct an initial population.

[0008] S2. Input the generated speech attribute parameter vector into the attribute conditional vocoder to synthesize the initial adversarial audio. Evaluate the initial adversarial audio using the attack success rate index, naturalness index, and query efficiency index to obtain the initial multi-target evaluation result.

[0009] S3. Using a multi-objective evolutionary algorithm, a multi-objective fitness function is constructed based on attack success rate, naturalness, and query efficiency indicators. Based on the initial multi-objective evaluation results, the speech attribute parameter vector in the initial population is iteratively optimized according to the multi-objective fitness function value, and the optimized multi-objective population result is output.

[0010] S4. The superior-inferior solution distance method is used to screen the optimized multi-target population results, and the speech attribute parameter vector with the largest proximity coefficient in the multi-target population results is determined as the optimal speech attribute parameter vector.

[0011] S5. Fuse the optimal speech attribute parameter vector with the original audio to generate the final adversarial audio sample.

[0012] Specifically, the standardization process for the original audio includes: standardizing the sampling rate of the original audio to eliminate spectral differences; normalizing the amplitude and scaling the maximum amplitude to a set range; and using a silence detection algorithm to remove silence segments without speech content.

[0013] Specifically, the speech rate coefficient is used to control the rate of syllable pronunciation, with a value range of 0.8-1.2 times the original speech rate; the intonation profile coefficient is used to control the fundamental frequency variation, with a value range of -15Hz to +15Hz; and the loudness envelope coefficient is used to control the variation of the overall volume envelope curve, with a value range of -3dB to +3dB.

[0014] Specifically, random initialization methods include uniform distribution random sampling, Gaussian distribution random sampling, and Latin hypercube sampling.

[0015] Specifically, the formula for calculating the attack success rate is as follows:

[0016]

[0017] In the formula: As an indicator of attack success rate, Represents the normalized Lewinstein distance; Indicates phoneme similarity based on the CMU dictionary; This represents the phonological distance based on the ALINE algorithm; Indicates the target attack text. This indicates the recognition output text. The phoneme sequence representing the target attack text. This represents the phoneme sequence of the output text. These are the weighting coefficients;

[0018] The naturalness score is calculated by the NISQA model and ranges from 1 to 5. The higher the naturalness score, the better the speech perception quality.

[0019] The formula for calculating the query efficiency index is:

[0020]

[0021] In the formula: For query efficiency metrics, This indicates the number of queries that have been executed so far. This indicates the maximum number of queries allowed.

[0022] Specifically, step S3 includes:

[0023] A dynamic weight adjustment mechanism based on query progress is introduced into the multi-objective evolutionary algorithm. Different weights are assigned to the three objectives of attack success rate, naturalness and query efficiency according to the query progress. At the beginning of the iteration optimization, the attack success rate is assigned the largest weight, and then the weights of naturalness and query efficiency are gradually increased to construct a multi-objective fitness function with dynamic weights.

[0024] The SGA-II algorithm is used to perform non-dominated sorting of the current population, dividing individuals in the current population into different levels according to Pareto superiority, and prioritizing the retention of superior individuals located at the Pareto front; at the same time, a crowding distance calculation method is introduced within the same level to measure the sparsity of individuals in the target space and select individuals with more dispersed distribution.

[0025] A simulated binary crossover operator is used to perform crossover operations on two parent individuals to generate new individuals with fused parental characteristics. During the mutation phase, a polynomial mutation operator is used to perturb some gene loci. Finally, the newly generated offspring individuals are merged with the parent population, and the fitness value is recalculated to provide a basis for the next round of non-dominated sorting.

[0026] Specifically, step S4 includes:

[0027] The attack success rate, naturalness, and query efficiency of each individual in the optimized multi-target population results are normalized. Then, virtual individuals that reach the theoretical optimal value in all three indicators are constructed as positive ideal solutions, and virtual individuals that reach the theoretical worst value in all three indicators are constructed as negative ideal solutions. For each individual in the population, the Euclidean distance between the individual and the positive and negative ideal solutions is calculated, and the proximity coefficient of the individual is calculated. The speech attribute parameter vector with the largest proximity coefficient is selected as the optimal speech attribute parameter vector.

[0028] Specifically, the formula for calculating the closeness coefficient is as follows:

[0029]

[0030] In the formula: For the first The proximity coefficient of each individual For the first The Euclidean distance between each individual and the positive ideal solution. For the first The Euclidean distance between each individual and the negative ideal solution. The closer the value is to 1, the better the overall performance of the individual.

[0031] Specifically, step S5 includes:

[0032] The optimal speech attribute parameter vector, along with the speaker embedding, semantic embedding, and target text embedding of the original audio, are input into the attribute conditional vocoder. Multimodal feature fusion is performed through an attention mechanism, and the decoder generates the final adversarial audio based on the fused features.

[0033] The present invention also provides a black-box adversarial audio generation system, comprising:

[0034] Speech attribute parameter vector generation module: This module is used to standardize the original audio. It uses a random initialization method to generate speech attribute parameter vectors within the set range of speech rate coefficient, pitch contour coefficient, and loudness envelope coefficient, and to construct the initial population.

[0035] Multi-target building module: This module is used to input the generated speech attribute parameter vector into the attribute conditional vocoder, synthesize the initial adversarial audio, and evaluate the initial adversarial audio using attack success rate, naturalness and query efficiency metrics to obtain the initial multi-target evaluation results.

[0036] Genetic optimization module: Employs a multi-objective evolutionary algorithm to construct a multi-objective fitness function based on attack success rate, naturalness, and query efficiency metrics. Based on the initial multi-objective evaluation results, iteratively optimizes the speech attribute parameter vector in the initial population according to the multi-objective fitness function value, and outputs the optimized multi-objective population result.

[0037] Speech attribute parameter vector selection module: The superior-inferior solution distance method is used to screen the optimized multi-target population results, and the speech attribute parameter vector with the largest proximity coefficient in the multi-target population results is determined as the optimal speech attribute parameter vector;

[0038] Adversarial audio generation module: used to fuse the optimal speech attribute parameter vector with the original audio to generate the final adversarial audio sample.

[0039] Beneficial Effects: Compared with existing technologies, the significant advantages of this invention are: It proposes an efficient adversarial sample generation framework for black-box ASR scenarios. By performing dynamic multi-objective optimization in an interpretable low-dimensional speech attribute parameter vector space, it significantly reduces query costs and improves the naturalness and attack success rate of generated samples. This invention uses an attribute-conditional vocoder to fuse multi-source embedded features for audio generation, effectively avoiding high-frequency artifacts and perceptible distortion caused by direct waveform perturbations. This invention combines dynamic weight allocation and the TOPSIS optimization mechanism to achieve a comprehensive balance between attack success rate, audio naturalness, and query efficiency. The adversarial audio samples obtained using the proposed black-box adversarial audio generation method can be directly used for adversarial training and defense strategy evaluation, thereby improving the robustness and security of automatic speech recognition models and filling the technical gap in existing technologies that balance multi-objective optimization under low query costs. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0041] Figure 2 This is a schematic diagram of the attribute conditional vocoder of Embodiment 1 of the present invention.

[0042] Figure 3 This is a schematic diagram of the final adversarial audio sample generation architecture in Embodiment 1 of the present invention. Detailed Implementation

[0043] A preferred embodiment of the present invention will be further described below with reference to the accompanying drawings.

[0044] Example 1

[0045] Please see Figure 1 As shown, this embodiment provides a black-box adversarial audio generation method, including the following steps:

[0046] S1. The original audio is standardized and a random initialization method is used to generate speech attribute parameter (SAP) vectors within the set range of speaking rate coefficient, intonation contour coefficient, and loudness envelope coefficient. This constructs an initial population for multi-objective evolutionary optimization, laying the foundation for subsequent parameter iteration and adversarial audio sample generation.

[0047] Specifically, the standardization process for the original audio includes: standardizing the sampling rate of the original audio (e.g., unifying it to a 16 kHz mono PCM format) to eliminate spectral differences caused by inconsistent sampling specifications from different data sources; normalizing the amplitude by scaling the maximum amplitude to a set range (e.g., -1 to 1); and using a silence detection algorithm (e.g., based on a short-time energy threshold) to remove silence segments without speech content, thereby reducing the impact of redundant data on the subsequent generation process.

[0048] In this invention, three core target parameters are defined: a speech rate coefficient to control the syllable pronunciation rate, ranging from 0.8 to 1.2 times the original speech rate; a pitch contour coefficient to control fundamental frequency variation, ranging from -15Hz to +15Hz to ensure pitch variation remains within the acceptable range of natural speech; and a loudness envelope coefficient to control the overall volume envelope curve variation, ranging from -3dB to +3dB to avoid significant distortion. These value ranges were determined by psychoacoustic experiments, providing sufficient search space for adversarial optimization while ensuring the naturalness of the generated samples. Within these parameter ranges, initial SAP vectors are generated using uniform or Gaussian random sampling. To improve population diversity, a Latin hypercube sampling (LHS) strategy can also be introduced to make the generated individuals more evenly distributed in the parameter space. The initial population size can be set according to computational resources, for example, 50 individuals.

[0049] S2. Input the generated speech attribute parameter vector into the attribute conditional vocoder to synthesize the initial adversarial audio. Evaluate the initial adversarial audio using the attack success rate index, naturalness index, and query efficiency index to obtain the initial multi-objective evaluation results, which will provide a reference for subsequent multi-objective evolutionary algorithms.

[0050] Attribute-conditional vocoders are speech generation models based on multi-source feature conditions. They separate and encode speech content and attributes, then adaptively couple them through a conditional fusion mechanism, and finally use a generator to reconstruct natural, high-fidelity speech signals. This architecture can generate semantically consistent but attribute-controllable audio outputs based on input speech attribute vectors (such as speech rate, prosody, loudness, etc.) and transcribed content information (such as text or MERGE features), providing foundational model support for tasks such as speech conversion, adversarial example generation, and robustness evaluation. Please refer to [reference needed]. Figure 2 As shown, the attribute conditional vocoder includes parallel attribute encoders and content encoders, a conditional fusion module that fuses the outputs of the attribute encoders and content encoders, and a generator that restores the fused speech features. These will be explained in detail below.

[0051] The attribute encoder is responsible for extracting high-level speech attribute features describing the dynamic characteristics of speech from the original audio and its speech attribute vectors. The attribute encoder encodes information such as prosody, fundamental frequency, loudness, and energy into low-dimensional latent representations, i.e., attribute latent vectors, which are used to control the style and expression of the generated speech. This module contains a combination of convolutional and recurrent layers to capture the smooth changes in the time series and distinguish between global attributes (such as speaking speed and timbre) and local attributes (such as energy curves and pitch variations). Through the attribute encoder, the model can learn a continuously adjustable speech style space, ensuring that the generated audio maintains naturalness and consistency under different speaking speeds and intonations.

[0052] The content encoder extracts the core semantic and structural information of the speech, ensuring that the generated result remains consistent with the input transcription at the linguistic content level. This module receives transcribed content or speech features (such as phoneme sequences, EL spectrum, or text vectors) extracted by the speech recognition module as input, and performs temporal modeling on these features using convolution or attention mechanisms to output a stable content feature representation. The design focus of the content encoder is to remove information irrelevant to speaker features and prosodic style, retaining only semantically relevant features to provide a "content baseline" for subsequent conditional fusion. This design ensures that the speech does not drift semantically under attribute changes, achieving the synthesis goal of "controllable attributes and invariant semantics."

[0053] The conditional fusion module is the core modulation unit of the entire architecture, used to fuse content and attribute information in a unified feature space. This module receives attribute latent vectors from the attribute encoder and content feature representations from the content encoder, and dynamically adjusts the distribution of content features using feature modulation mechanisms (such as FiLM, AdaIN, or conditional attention mechanisms), enabling the generator to adaptively synthesize speech under different attribute conditions. This module ensures the semantic integrity of the content features while achieving style transfer and expression control of speech through attribute modulation, thereby enabling precise control over dimensions such as speech rate, intonation, and energy.

[0054] The generator is the core decoding module of the vocoder, used to restore the high-level speech features fused by the conditional fusion module into a time-domain waveform signal. This module employs a convolution-based neural vocoder structure, such as HiFi-GAN, WaveRNN, or a diffusion vocoder, achieving high-fidelity audio reconstruction through multi-scale upsampling and residual learning. During the inference phase, the generator generates speech waveforms based on the feature distribution output by the conditional fusion module, making the synthesized audio perceptually natural and continuous, and accurately reflecting the set attribute features. This module also possesses good versatility and can be applied to various scenarios such as speech reconstruction, speech conversion, or adversarial example generation.

[0055] The generated initial adversarial audio is input into the target ASR system to obtain the recognized output text, which is then compared with the target attack text. The attack success rate is calculated using the following formula:

[0056]

[0057] In the formula: As an indicator of attack success rate, This represents the normalized Levenstein distance, used to measure character-level differences; This indicates phoneme similarity based on the CMU dictionary, used to reflect the degree of similarity in pronunciation; This represents the phonological distance based on the ALINE algorithm, used to measure the structural differences in phoneme pronunciation; Indicates the target attack text. This indicates the recognition output text. The phoneme sequence representing the target attack text. This represents the phoneme sequence of the output text. This is a weighting coefficient used to balance the contribution of the three distance indicators to the final score.

[0058] The naturalness index is calculated by the NISQA model and ranges from 1 to 5. The higher the naturalness index, the better the perceived quality of speech. The NISQA model quantifies the acceptability of subjective listening by evaluating features such as spectral smoothness, harmonic structure integrity, and transient components.

[0059] The query efficiency metric quantifies the utilization of the current number of queries relative to the allowed query limit, thereby minimizing query overhead while ensuring attack effectiveness. The query efficiency metric is calculated using the following formula:

[0060]

[0061] In the formula: For query efficiency metrics, This indicates the number of queries that have been executed so far. This indicates the maximum number of queries allowed.

[0062] S3. Employ a multi-objective evolutionary algorithm to construct a multi-objective fitness function based on attack success rate, naturalness, and query efficiency metrics. Based on the initial multi-objective evaluation results, iteratively optimize the speech attribute parameter vector in the initial population according to the multi-objective fitness function value, and output the optimized multi-objective population result.

[0063] In the multi-objective optimization process designed in this invention, the degree of attention paid to attack success rate, naturalness, and query efficiency varies at different stages. Specifically, this embodiment introduces a dynamic weight adjustment mechanism based on query progress into the multi-objective evolutionary algorithm. Different weights are assigned to the three objectives of attack success rate, naturalness, and query efficiency according to the query progress. In the early stage of iterative optimization, the attack success rate is assigned the largest weight, and then the weights of naturalness and query efficiency are gradually increased to construct a multi-objective fitness function with dynamic weights.

[0064] First, define the query progress ratio:

[0065]

[0066] In the formula: This represents the number of queries that have been executed so far. This represents the maximum number of queries allowed.

[0067] The weights of the three objectives are then dynamically adjusted using the following formula:

[0068]

[0069] In the formula: As the weight of the attack success rate metric, The weight of the naturalness index, As the weight of query efficiency metrics, To adjust the coefficient, This approach prioritizes increasing attack success rate in the early stages of optimization, then gradually increases the focus on audio naturalness and query efficiency, achieving a balanced optimization across multiple objectives. This dynamic weighting strategy can quickly approximate the feasible solution space in the early stages of an attack and refine perturbation features in the mid-to-late stages, thereby generating adversarial examples that combine high attack power with high naturalness, while avoiding excessive query consumption.

[0070] To balance attack success rate, naturalness, and query efficiency in multi-objective optimization, this invention employs the SGA-II algorithm to perform non-dominated sorting of the current population. Individuals within the current population are divided into different levels according to Pareto merit, prioritizing the retention of high-performing individuals at the Pareto front. Simultaneously, a crowding distance calculation method is introduced within the same level to measure the sparsity of individuals in the target space, selecting more dispersed individuals to maintain population diversity across the entire optimization space. This strategy effectively avoids the search getting trapped in local optima, thereby improving global search performance and convergence stability.

[0071] After individual selection is complete, this invention generates new candidate solutions through genetic operations. Specifically, a simulated binary crossover operator is used to crossover two parent individuals, generating new individuals with a fusion of parental characteristics; the crossover probability is typically set to 0.9 to ensure sufficient gene recombination. During the mutation phase, a polynomial mutation operator is used to perturb some gene loci; the mutation probability can be set between 0.1 and 0.2 to balance the exploratory and exploitative aspects of the search. By combining crossover and mutation, new feature combinations can be introduced while preserving superior genes, thereby improving the coverage and diversity of the optimization search. Finally, the newly generated offspring individuals are merged with the parent population, and the fitness value is recalculated to provide a basis for the next round of non-dominated ranking.

[0072] S4. The superior-inferior solution distance method is used to screen the optimized multi-target population results, and the speech attribute parameter vector with the largest proximity coefficient in the multi-target population results is determined as the optimal speech attribute parameter vector.

[0073] The attack success rate, naturalness, and query efficiency metrics of each individual in the optimized multi-target population results are normalized to ensure that the target values ​​are within a comparable range. Then, virtual individuals achieving theoretically optimal values ​​on all three metrics are constructed as positive ideal solutions, and virtual individuals achieving theoretically worst values ​​on all three metrics are constructed as negative ideal solutions. For each individual in the population, the Euclidean distance between that individual and both the positive and negative ideal solutions is calculated, and the proximity coefficient of that individual is obtained. The speech attribute parameter vector with the largest proximity coefficient is selected as the optimal speech attribute parameter vector.

[0074] The formula for calculating the proximity coefficient is:

[0075]

[0076] In the formula: For the first The proximity coefficient of each individual For the first The Euclidean distance between each individual and the positive ideal solution. For the first The Euclidean distance between each individual and the negative ideal solution. The closer the value is to 1, the better the overall performance of the individual.

[0077] S5. Fuse the optimal speech attribute parameter vector with the original audio to generate the final adversarial audio sample.

[0078] Please refer to Figure 3As shown, the optimal speech attribute parameter vector, along with the speaker embedding, semantic embedding, and target text embedding of the original audio, are input into an attribute conditional vocoder. Multimodal feature fusion is performed through an attention mechanism, and the decoder generates the final adversarial audio based on the fused features. This audio is highly similar to the original audio in subjective perception but can significantly mislead the ASR system into outputting incorrect recognition results. The final generated adversarial audio samples can not only be used for attack testing and system vulnerability verification but also serve as adversarial training data to improve the robustness of the model in black-box scenarios, contributing to the construction of more comprehensive detection and defense mechanisms.

[0079] To demonstrate the effectiveness and superiority of the present invention, an experiment was conducted in a specific application scenario.

[0080] The experimental platform was configured as follows: CPU: AMD Ryzen 5 5600X CPU; Memory size: 64GB; GPU: NVIDIA GTX3090 GPU.

[0081] In a Python environment, using the TensorFlow and PyTorch frameworks, adversarial audio samples obtained from the aforementioned black-box adversarial audio generation method were tested. This experiment selected several mainstream automatic speech recognition systems as attack targets, including DeepSpeech2, CMU Sphinx, and ytek API.

[0082] To verify the effectiveness of the present invention, a multi-dimensional performance evaluation was performed on the generated adversarial audio samples. Evaluation metrics included:

[0083] (1) Attack Success Rate (ASR): refers to the proportion of the target automatic speech recognition system that transcribes the adversarial audio into the preset target text. The higher the value, the better the attack effect.

[0084] (2) Naturalness: The generated adversarial audio is scored using a non-reference speech quality assessment model (such as NISQA) to reflect its similarity to natural human voice. The score ranges from 0 to 5, with higher values ​​indicating more natural sound quality.

[0085] (3) Query Efficiency: In black-box interaction mode, the average number of queries required to complete a successful attack by interacting with the target ASR system. The smaller the value, the higher the efficiency.

[0086] Please refer to Table 1 below, which uses the DeepSpeech2 platform as an example, to obtain a comparison of different black-box attack methods in terms of attack success rate, naturalness, and average number of queries.

[0087] Table 1. Comparison of Multi-Dimensional Indicators of DeepSpeech2 Black-Box Attack Methods

[0088] method Attack success rate Naturalness Average number of queries GA-GE 35% 3.03 9778 IMPGA 46% 3.15 7235 SMACK 82% 3.36 1257 This invention 82% 3.36 1085

[0089] As shown in Table 1, the adversarial audio samples generated by the black-box adversarial audio generation method proposed in this invention outperform the comparative methods in terms of attack success rate, naturalness, and query efficiency. This demonstrates that the proposed method can achieve a high attack success rate while ensuring audio naturalness and stealth, and significantly reduces the average number of queries, thus achieving a further improvement in overall performance compared to existing technologies.

[0090] Example 2

[0091] This embodiment provides a black-box adversarial audio generation system corresponding to the black-box adversarial audio generation system described in Embodiment 1, including:

[0092] Speech attribute parameter vector generation module: This module is used to standardize the original audio. It uses a random initialization method to generate speech attribute parameter vectors within the set range of speech rate coefficient, pitch contour coefficient, and loudness envelope coefficient, and to construct the initial population.

[0093] Multi-target building module: This module is used to input the generated speech attribute parameter vector into the attribute conditional vocoder, synthesize the initial adversarial audio, and evaluate the initial adversarial audio using attack success rate, naturalness and query efficiency metrics to obtain the initial multi-target evaluation results.

[0094] Genetic optimization module: Employs a multi-objective evolutionary algorithm to construct a multi-objective fitness function based on attack success rate, naturalness, and query efficiency metrics. Based on the initial multi-objective evaluation results, iteratively optimizes the speech attribute parameter vector in the initial population according to the multi-objective fitness function value, and outputs the optimized multi-objective population result.

[0095] Speech attribute parameter vector selection module: The superior-inferior solution distance method is used to screen the optimized multi-target population results, and the speech attribute parameter vector with the largest proximity coefficient in the multi-target population results is determined as the optimal speech attribute parameter vector;

[0096] Adversarial audio generation module: used to fuse the optimal speech attribute parameter vector with the original audio to generate the final adversarial audio sample.

Claims

1. A black-box adversarial audio generation method, characterized in that, Includes the following steps: S1. Standardize the original audio and use a random initialization method to generate speech attribute parameter vectors within the set range of speech rate coefficient, pitch contour coefficient, and loudness envelope coefficient to construct an initial population. S2. Input the generated speech attribute parameter vector into the attribute conditional vocoder, and evaluate the initial adversarial audio using the attack success rate index, naturalness index and query efficiency index to obtain the initial multi-target evaluation result. S3. Using a multi-objective evolutionary algorithm, a multi-objective fitness function is constructed based on attack success rate, naturalness, and query efficiency indicators. Based on the initial multi-objective evaluation results, the speech attribute parameter vector in the initial population is iteratively optimized according to the multi-objective fitness function value, and the optimized multi-objective population result is output. S4. The superior-inferior solution distance method is used to screen the optimized multi-target population results, and the speech attribute parameter vector with the largest proximity coefficient in the multi-target population results is determined as the optimal speech attribute parameter vector. S5. Fuse the optimal speech attribute parameter vector with the original audio to generate the final adversarial audio sample.

2. The black-box adversarial audio generation method according to claim 1, characterized in that, The standardization process for the original audio includes: standardizing the sampling rate of the original audio to eliminate spectral differences; normalizing the amplitude and scaling the maximum amplitude to a set amplitude range; and using a silence detection algorithm to remove silence segments without speech content.

3. The black-box adversarial audio generation method according to claim 1, characterized in that: The speech rate coefficient is used to control the syllable pronunciation rate, and its value ranges from 0.8 to 1.2 times the original speech rate; the intonation profile coefficient is used to control the fundamental frequency change, and its value ranges from -15Hz to +15Hz; the loudness envelope coefficient is used to control the change of the overall volume envelope curve, and its value ranges from -3dB to +3dB.

4. The black-box adversarial audio generation method according to claim 1, characterized in that: The random initialization methods include uniform distribution random sampling, Gaussian distribution random sampling, and Latin hypercube sampling.

5. The black-box adversarial audio generation method according to claim 1, characterized in that: The formula for calculating the attack success rate is as follows: In the formula: As an indicator of attack success rate, Represents the normalized Lewinstein distance; Indicates phoneme similarity based on the CMU dictionary; This represents the phonological distance based on the ALINE algorithm; Indicates the target attack text. This indicates the recognition output text. The phoneme sequence representing the target attack text. This represents the phoneme sequence of the output text. These are the weighting coefficients; The naturalness index is calculated by the NISQA model and ranges from 1 to 5. The higher the naturalness index, the better the speech perception quality. The formula for calculating the query efficiency index is as follows: In the formula: For query efficiency metrics, This indicates the number of queries that have been executed so far. This indicates the maximum number of queries allowed.

6. The black-box adversarial audio generation method according to claim 1, characterized in that, Step S3 includes: A dynamic weight adjustment mechanism based on query progress is introduced into the multi-objective evolutionary algorithm. Different weights are assigned to the three objectives of attack success rate, naturalness and query efficiency according to the query progress. At the beginning of the iteration optimization, the attack success rate is assigned the largest weight, and then the weights of naturalness and query efficiency are gradually increased to construct a multi-objective fitness function with dynamic weights. The SGA-II algorithm is used to perform non-dominated sorting of the current population, dividing individuals in the current population into different levels according to Pareto superiority, and prioritizing the retention of superior individuals located at the Pareto front; at the same time, a crowding distance calculation method is introduced within the same level to measure the sparsity of individuals in the target space and select individuals with more dispersed distribution. A simulated binary crossover operator is used to perform crossover operations on two parent individuals to generate new individuals with fused parental characteristics. During the mutation phase, a polynomial mutation operator is used to perturb some gene loci. Finally, the newly generated offspring individuals are merged with the parent population, and the fitness value is recalculated to provide a basis for the next round of non-dominated sorting.

7. The black-box adversarial audio generation method according to claim 1, characterized in that, Step S4 includes: The attack success rate, naturalness, and query efficiency of each individual in the optimized multi-target population results are normalized. Then, virtual individuals that reach the theoretical optimal value in all three indicators are constructed as positive ideal solutions, and virtual individuals that reach the theoretical worst value in all three indicators are constructed as negative ideal solutions. For each individual in the population, the Euclidean distance between the individual and the positive and negative ideal solutions is calculated, and the proximity coefficient of the individual is calculated. The speech attribute parameter vector with the largest proximity coefficient is selected as the optimal speech attribute parameter vector.

8. The black-box adversarial audio generation method according to claim 1, characterized in that, The formula for calculating the proximity coefficient is as follows: In the formula: For the first The proximity coefficient of each individual For the first The Euclidean distance between each individual and the positive ideal solution. For the first The Euclidean distance between each individual and the negative ideal solution. The closer the value is to 1, the better the overall performance of the individual.

9. The black-box adversarial audio generation method according to claim 1, characterized in that, Step S5 includes: The optimal speech attribute parameter vector, along with the speaker embedding, semantic embedding, and target text embedding of the original audio, are input into the attribute conditional vocoder. Multimodal feature fusion is performed through an attention mechanism, and the decoder generates the final adversarial audio based on the fused features.

10. A black-box adversarial audio generation system, characterized in that, include: Speech attribute parameter vector generation module: This module is used to standardize the original audio. It uses a random initialization method to generate speech attribute parameter vectors within the set range of speech rate coefficient, pitch contour coefficient, and loudness envelope coefficient, and to construct the initial population. Multi-target building module: This module is used to input the generated speech attribute parameter vector into the attribute conditional vocoder, synthesize the initial adversarial audio, and evaluate the initial adversarial audio using attack success rate, naturalness and query efficiency metrics to obtain the initial multi-target evaluation results. Genetic optimization module: Employs a multi-objective evolutionary algorithm to construct a multi-objective fitness function based on attack success rate, naturalness, and query efficiency metrics. Based on the initial multi-objective evaluation results, iteratively optimizes the speech attribute parameter vector in the initial population according to the multi-objective fitness function value, and outputs the optimized multi-objective population result. Speech attribute parameter vector selection module: The superior-inferior solution distance method is used to screen the optimized multi-target population results, and the speech attribute parameter vector with the largest proximity coefficient in the multi-target population results is determined as the optimal speech attribute parameter vector; Adversarial audio generation module: used to fuse the optimal speech attribute parameter vector with the original audio to generate the final adversarial audio sample.

Images