Voice conversion confrontation audio generation method and device based on conditional diffusion model

Abstract

The invention discloses a voice conversion confrontation audio generation method and device based on a conditional diffusion model, and the method comprises the steps: preparing a reference audio sample, generating a confrontation audio sample in a voice conversion task based on the reference audio sample in a white-box attack mode, and enabling the reference audio sample and the confrontation audio sample to form a sample pair; taking an adversarial audio sample in the sample pair as input, and performing training based on an adversarial audio generation task on the conditional diffusion model under the condition that a reference audio sample in the sample pair is taken as a guide condition; and generating an adversarial audio which is related to the reference audio sample and is used for black box voice conversion under the guidance of the reference audio sample by utilizing a reverse generation process of the trained conditional diffusion model, so as to solve the technical problem that the efficiency and the effect of generating the adversarial audio for voice conversion in the existing black box scene are insufficient.

Description

Technical Field

[0001] The present invention belongs to the technical field of adversarial sample generation in speech conversion, and specifically relates to a speech conversion adversarial audio generation method and device based on a conditional diffusion model. Background Art

[0002] Voice Conversion (VC) is a sound control technology that aims to change specific acoustic properties, such as speaker identity, emotion, or accent, while preserving the speaker's content information. When using machine learning for voice conversion, it mainly relies on a corpus of high-quality recording conditions. In real-world scenarios, due to the diversity and instability of recording conditions, the actual collected speech data often differs significantly from the ideal corpus. Recent studies have found that even the current best-performing voice conversion models will significantly degrade in the face of noisy recording environments and voice data loss during channel transmission. How to solve the problem of mismatch between training audio quality and test audio quality is a key issue in robust voice conversion work. In order to comprehensively evaluate the robustness of voice conversion models in the face of degraded input data, it is necessary to simulate various noise interferences that may be encountered in the real world by generating adversarial audio.

[0003] The adversarial audio generation of speaker timbre is a process of optimizing speaker representation. The basic idea is to ensure that the representation distance between the adversarial audio and the manually specified audio is as close as possible, while making the representation distance as far away as possible from the target reference audio.

[0004] The black box scenario is a real-world speech conversion test environment. In this black box scenario, the specific information of the black box speech conversion model used for speech conversion (such as model structure, weights, gradients, etc.) is unknown. In order to explore the robustness of the black box speech conversion model, reveal its vulnerabilities and improve its robustness, researchers have proposed many black box attack query methods in recent years. They repeatedly query the victim model by adding perturbed adversarial audio and adjust the perturbation based on the returned prediction probability and hard labels. However, these query methods have three major technical defects when generating adversarial audio: (1) The query cost is high. Tens of thousands of queries will consume a lot of computing resources, time, and money, which is not applicable in practical applications. (2) Easy to be filtered. Artificially generated adversarial audio is easily filtered by the upsampling process of the built-in vocoder module of the victim model, resulting in the attack test effect being less than expected; (3) The black box attack relies heavily on the white box voice conversion model whose structural parameters are known and used for voice conversion. The success of the black box attack is closely related to the similarity between the white box voice conversion model used as a proxy and the black box voice conversion model actually attacked. If there is a large difference between the white box voice conversion model used as a proxy and the black box voice conversion model actually attacked, the attack test effect will be significantly reduced. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for generating voice conversion adversarial audio based on a conditional diffusion model, so as to solve the technical problem of insufficient efficiency and effect of voice conversion adversarial audio generation in existing black box scenarios.

[0006] To achieve the above-mentioned purpose of the invention, an embodiment provides a method for generating voice conversion adversarial audio based on a conditional diffusion model, comprising the following steps: Sample pair construction: prepare reference audio samples, generate adversarial audio samples in the voice conversion task based on the reference audio samples and adopt white-box attack method, and form sample pairs with reference audio samples and adversarial audio samples; Model training: Using the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as the guide condition, the conditional diffusion model is trained based on the adversarial audio generation task; Adversarial audio generation: Using the inverse generation process of the trained conditional diffusion model, under the guidance of the reference audio samples, adversarial audio related to the reference audio samples and used for black-box speech conversion is generated.

[0007] The present invention aims to transform the generation problem of speech conversion adversarial samples into the problem of audio sample generation across data distribution domains, and regard the adversarial audio samples as specific variants of reference audio samples as references, thereby avoiding complex optimization processes, simplifying the generation process, and realizing efficient generation of adversarial audio samples under the guidance of reference audio samples. Specifically, a conditional diffusion model is used as a converter of audio data, and a processing flow including three stages of sample pair construction, model training, and adversarial audio generation is constructed. The processing flow makes the generation of adversarial audio no longer overly dependent on the inherent properties of the speech conversion model, but uses the output results of the adversarial audio to the attacked black box speech conversion model as hard labels to verify whether the attack is successful. In the model training stage, the training of the reverse diffusion model further enhances the generalization ability of the attack and improves the transferability of the attack. In addition, through this processing flow, a large number of adversarial audios can be generated in batches, and the generation efficiency and quality of the adversarial audios are high, thereby significantly improving the attack test efficiency and scalability of the black box speech conversion model.

[0008] Preferably, generating adversarial audio samples in a speech conversion task based on reference audio samples and using a white-box attack method includes: By using adversarial audio samples and manually specified audio samples The representation distance is as close as possible to the reference audio sample As far as possible from the optimization goal, find a noise disturbance factor , so that the adversarial audio sample In the white-box scenario, the converted audio attack generated by the speech conversion model is effective and sounds closer to the specified audio sample. timbre, based on which adversarial audio samples are generated.

[0009] Preferably, the specific process of generating adversarial audio samples is: Given parameter initial values , disturbance coefficient , characterizing the adjustment coefficient And the disturbance factor is required ,in, Represents the hyperbolic tangent function, generating adversarial audio samples based on the perturbation factor ; Preparing source audio samples , Reference Audio Samples , specify audio sample , using content encoder Source audio samples Encoding to obtain semantic representation , using the speaker encoder For clean audio samples , specify audio sample , and adversarial audio samples Encoding to get speaker representation , ,as well as ; Use decoder D to represent the pair ( , )、( , ),as well as( , ) to decode and get the converted audio , ,as well as ; Constructing a loss function based on converted audio , and the loss function Minimize the target parameters And find the optimal perturbation factor , and then generate adversarial audio samples , ; in, Calculates the difference between the two.

[0010] Preferably, taking the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as a guide condition, training the conditional diffusion model based on the adversarial audio generation task includes: In the forward diffusion process, the adversarial audio sample is used as the sample at time 0. , and at each time step t, Collecting real noise And add it to the generated sample obtained at the previous time step t-1 , get the sample for each time step t ,sample It obeys the prior Gaussian distribution and is expressed as: ; in, Indicates based on Generated by adding noise The prior Gaussian distribution of N represents the Gaussian distribution. It represents the real variance used when collecting real noise at time step t, and its value is 0-1. represents the mean of the Gaussian distribution, express Obey Gaussian distribution; In the forward diffusion process, the sample It is also input to the neural network contained in the conditional diffusion model M and use the reference audio sample as the guide condition c The same input to the neural network M Predicting noise , and based on real noise With prediction noise Calculated between norm, through the following loss function Update the Neural Network M Parameters , realize the training of conditional diffusion model; ; in, for norm, , represents the infinite norm, The larger the value, the more diverse the adversarial audio samples generated.

[0011] Preferably, the neural network M Adopt U-Net structure with residual connection; The reference audio samples are encoded and input into the neural network M .

[0012] Preferably, the inverse generation process of the trained conditional diffusion model is used to generate adversarial audio related to the reference audio sample and used for black box speech conversion under the guidance of the reference audio sample, including: In the reverse generation process, the reference audio sample is also used as a guide condition c , from the normal distribution Random noise collected in As a sample at time T , and at each time step t Based on the bootstrap condition c , generate samples Through neural network M Prediction Noise , time step t The generated sample Subtract prediction noise That is, as the time step t -1 generated sample , generate samples It obeys the posterior Gaussian probability distribution, expressed as: ; in, Indicates based on Generated by denoising The posterior Gaussian distribution of Represents the neural network parameters The mean of the fit, Represents the neural network parameters The variance of the fit, express Obey Gaussian distribution; After 0-T steps of reverse generation, the generated sample at time step 0 Adversarial audio correlated with the reference audio sample for black-box speech conversion.

[0013] Preferably, the method further comprises: performing a cropping operation on the generated adversarial audio to control the difference between the adversarial audio and the reference audio sample to be within a set range.

[0014] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a speech conversion adversarial audio generation device based on a conditional diffusion model, comprising: A sample pair construction module is used to prepare reference audio samples, generate adversarial audio samples in the voice conversion task based on the reference audio samples and adopt a white-box attack method, and form a sample pair with the reference audio samples and the adversarial audio samples; A model training module, which is used to train the conditional diffusion model based on the adversarial audio generation task using the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as a guide condition; The adversarial audio generation module is used to utilize the inverse generation process of the trained conditional diffusion model to generate adversarial audio related to the reference audio sample and used for black-box speech conversion under the guidance of the reference audio sample.

[0015] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned voice conversion adversarial audio generation method based on the conditional diffusion model.

[0016] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned method for generating speech conversion adversarial audio based on the conditional diffusion model is implemented.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention generates adversarial audio samples in a speech conversion task based on reference audio samples and using a white-box attack method as samples for training a conditional diffusion model. Since the samples are generated in the speech conversion task using a white-box attack method, they are mainly used for black-box attack tests of the speech conversion task. Therefore, the trained conditional diffusion model serves as a data generator of adversarial audio samples guided by reference audio samples. It effectively supports adversarial audio generation in a black-box scenario through sampling, and eliminates the need for complex query and optimization iteration processes in speech conversion adversarial audio generation in a black-box scenario, thereby improving generation efficiency and saving costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a flowchart of a method for generating voice conversion adversarial audio based on a conditional diffusion model provided by an embodiment; Figure 2 It is a flowchart of generating adversarial audio samples in a speech conversion task using a white box attack method provided by an embodiment; Figure 3It is a structural schematic diagram of a speech conversion adversarial audio generation device based on a conditional diffusion model provided in an embodiment. DETAILED DESCRIPTION

[0020] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0021] The inventive concept of the present invention is: in order to cope with the impact of low-quality input audio on the black-box speech conversion effect, especially the challenges in emotional expression and natural dialogue, adversarial audio provides an effective testing method, which can locate the root cause of the black-box speech conversion model vulnerability and help improve the robustness and security of the speech conversion model. Existing adversarial audio generation is highly dependent on speaker encoders, reference audio and specified audio, which is not conducive to simulating various degraded audios in complex real-life scenarios. In black-box scenarios, researchers try to use multiple query iterations or integrate multiple speaker encoders to adjust the perturbation to improve adaptability and transferability, but still face technical problems such as high query cost, noise filtering and over-reliance on the white-box speech conversion model as an agent. To this end, an embodiment of the present invention provides a speech conversion adversarial audio generation scheme based on a conditional diffusion model, which simulates real-world noise scenarios to test the vulnerability and potential vulnerabilities of the black-box speech conversion model, that is, the degree of degradation robustness. While ensuring that the test cost remains at a low level, it balances the attack effect and speech perceptibility on the black-box speech conversion model, and also helps to develop a speech conversion model with higher robustness to degraded audio.

[0022] like Figure 1 As shown, the voice conversion adversarial audio generation method based on the conditional diffusion model provided in the embodiment includes the following steps: S1, sample pair construction: prepare reference audio samples, generate adversarial audio samples in the voice conversion task based on the reference audio samples and adopt a white-box attack method, and form sample pairs with the reference audio samples and the adversarial audio samples.

[0023] In the embodiment, audio samples are prepared. Specifically, a total of 50 hours of audio can be randomly selected from the English audiobook VCTK, Chinese corpus AISHELL, English data set LibriTTS, English dialect data set TIMIT, Japanese speaker database JVS, multilingual database CommonVoice and other speech conversion field benchmark corpora to construct an audio data set, and the audio data is intercepted as an audio sample. Each sentence of the intercepted audio must be greater than 3s, without long pauses, clear pronunciation, pure timbre, and a sampling rate of 44.1kHz or 48kHz to ensure voice quality. After completing the selection of voice quality audio samples, an audio sample set is obtained.A ={ O , X}, further, according to the content of all audio texts in the dataset, the audio sample set is divided into A Divide into source audio sample sets O ={ o 1, o 2,…, o n} and the target reference sample set X ={ x 1, x 2,…, x n},in o i and x i Not the same speaker, source audio samples o i The corresponding text content in the dataset and the reference audio samples x i different, i is the sample index, n is the sample size.

[0024] You also need to prepare the specified audio sample , the specified audio sample y Refers to the audio sample specified by the attacker. y The timbre is different from the reference audio sample x and source audio samples o timbre.

[0025] After all samples are prepared, adversarial audio samples are generated in the voice conversion task based on the reference audio samples and using a white-box attack method. , where the adversarial audio sample The goal is to compare the reference audio samples The difference should be as large as possible. If it is a targeted attack, it is required to be consistent with the specified audio sample. The difference is as small as possible.

[0026] In the embodiment, the main idea of ​​using the white box attack method to generate adversarial audio samples in the voice conversion task is: and manually specified audio samples The representation distance is as close as possible to the reference audio sample As far as possible from the optimization goal, find a noise disturbance factor , so that the adversarial audio sample In the white-box scenario, the converted audio attack generated by the speech conversion model is effective and sounds closer to the specified audio sample. timbre, based on which adversarial audio samples are generated.

[0027] Based on the above main ideas, Figure 2 As shown in Figure 1, the specific process of generating adversarial audio samples in the voice conversion task using a white-box attack is as follows: (1) Given the initial value of the parameter , disturbance coefficient , characterizing the adjustment coefficient And the disturbance factor is required ,in, Represents the hyperbolic tangent function, generating adversarial audio samples based on the perturbation factor .

[0028] (2) Prepare source audio samples , Reference Audio Samples , specify audio sample , using the content encoder ( , ) for the source audio samples Encoding to obtain semantic representation , using the speaker encoder ( ) for the clean audio samples , specify audio sample , and adversarial audio samples Encoding to get speaker representation , ,as well as ; ; ; ; ; After obtaining the vector representation of each audio, the abstract speech expression can be converted into mathematical form to facilitate subsequent processing.

[0029] (3) Use decoder D to represent the pair ( , )、( , ),as well as( , ) to decode and get the converted audio , ,as well as ; (4) Constructing a loss function based on converted audio , and the loss function Minimize the target parameters And find the optimal perturbation factor , and then generate adversarial audio samples ,

[0030] in, To calculate the difference between the two, the mean square error RMSE can be used.

[0031] Adversarial audio samples constructed through the above process With reference audio samples The sample pairs that make up the training conditional diffusion model .

[0032] S2, model training: Taking the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as the guiding condition, the conditional diffusion model is trained based on the adversarial audio generation task.

[0033] In the embodiment, a conditional diffusion model is first constructed, which is used to implement the adversarial audio generation task, and includes a neural network for predicting noise. M , the neural network can adopt a U-Net structure with residual connections.

[0034] First, calculate the denoising result generated by the forward diffusion process at time T , specifically in the forward diffusion process, given a time step that conforms to the uniform distribution , to counter the audio sample As a sample at time 0 , and obeys the distribution , and at each time step t from Collecting real noise And add the generated sample of the previous time step t-1 Get the sample for each time step t ,sample It obeys the prior Gaussian distribution and is expressed as: ; in, Indicates based on Generated by adding noise The prior Gaussian distribution of N represents the Gaussian distribution. It represents the real variance used when collecting real noise at time step t, and its value is 0-1. represents the mean of the Gaussian distribution, express Obeying the Gaussian distribution, after time T, it is obtained that obeys the Gaussian distribution The diffuse noise ,Right now .

[0035] Since the Markov process is only related to the state of the previous time step, the forward diffusion process is based on the prior Gaussian distribution. Generate samples for each time step, that is, t Sample of time steps , Represents the index of time step t.

[0036] The core of training the conditional diffusion model in the forward diffusion process is to let the model predict a reliable noise , in the forward diffusion process, the sample It is also input to the neural network contained in the conditional diffusion model M and reference audio samples As a guide condition c The same input to the neural network M Predicting noise , and based on real noise With prediction noise Calculated between norm, through the following loss function Update the Neural Network M Parameters , realize the training of conditional diffusion model; ; in, for norm, , represents the infinite norm, The larger the value, the more diverse the adversarial audio samples generated.

[0037] In the embodiment, the reference audio sample As a guide condition c , the boot condition c The vector is encoded before being input into the neural network for guidance. Based on this guidance, the noise generated by the neural network is consistent with the reference audio sample. Related.

[0038] S3, Adversarial Audio Generation: Using the inverse generation process of the trained conditional diffusion model, under the guidance of the reference audio samples, adversarial audio related to the reference audio samples and used for black-box speech conversion is generated.

[0039] In an embodiment, after the training of the conditional diffusion model is completed, the neural network it contains has the ability to generate relevant accurate noise based on the guidance conditions. Therefore, the inverse generation process of the trained conditional diffusion model is utilized to generate adversarial audio related to the reference audio sample and used for black box speech conversion under the guidance of the reference audio sample.

[0040] Specifically, in the reverse generation process, the reference audio sample is also used as a guiding condition c , from the normal distribution Random noise collected in As a sample at time T , and at each time step t Based on the bootstrap condition c , generate samples Through neural network M Prediction Noise , time step t The generated sample Subtract prediction noise That is, as the time step t -1 generated sample , generate samples It obeys the posterior Gaussian probability distribution, expressed as: ; in, Indicates based on Generated by denoising The posterior Gaussian distribution of Represents the neural network parameters The mean of the fit, Represents the neural network parameters The variance of the fit, express Obey Gaussian distribution; After 0-T steps of reverse generation, the generated sample at time step 0 Adversarial audio used for black-box voice conversion as a correlation with reference audio samples .

[0041] In the embodiment, in order to ensure the generation of adversarial audio , which sounds similar to the reference audio sample used as the guide condition x The difference is not big, so it is necessary to fight against audio Further trimming steps to control adversarial audio With reference audio samples x The gap between Operation will be against audio With reference audio samples xThe gap between them is constrained to be between 0 and 1.

[0042] In the embodiment, the adversarial audio generated by the conditional diffusion model is also used As a sample to test its aggressiveness to the black box voice conversion model, when the attack is successful, it is used to evaluate the robustness of the black box voice conversion model in the face of degraded input data. When the attack is unsuccessful, repeat step S3 to repeatedly generate adversarial audio , until the attack is successful. Due to the existence of the conditional diffusion model as a sample generator, the number of iterations It will be greatly reduced, thus saving time cost to a certain extent, and it can be made independent of the white-box speech conversion model as an agent.

[0043] The above-mentioned speech conversion adversarial audio generation method generates adversarial audio that does not depend on specific speech conversion test data or models, and supports multiple datasets, such as the English audiobook dataset LibriTTS, the English dataset VCTK, the English dialect dataset TIMIT, and the Chinese dataset AISHELL. The supported speech conversion models include AdaptVC, AdaINVC, S2VC, etc.

[0044] The adversarial audio generated by the above-mentioned voice conversion adversarial audio generation method is suitable for both non-targeted black box attack and targeted black box attack scenarios. In these scenarios, without knowing the voice conversion model information, the audio can be converted to be close to the timbre of a third-party speaker specified by the attacker in the targeted black box attack scenario, or close to the timbre of a random speaker of any non-reference audio in the non-targeted black box attack scenario.

[0045] Based on the same inventive concept, Figure 3 As shown, the embodiment also provides a speech conversion adversarial audio generation device 30 based on the conditional diffusion model, including: a sample pair construction module 31, a model training module 32, and an adversarial audio generation module 33, wherein the sample pair construction module 31 is used to prepare reference audio samples, generate adversarial audio samples in the speech conversion task based on the reference audio samples and using a white-box attack method, and form a sample pair with the reference audio samples and the adversarial audio samples; the model training module 32 is used to use the adversarial audio samples in the sample pair as input, and train the conditional diffusion model based on the adversarial audio generation task under the reference audio samples in the sample pair as the guiding condition; the adversarial audio generation module 33 is used to utilize the inverse generation process of the trained conditional diffusion model, and generate adversarial audio related to the reference audio samples and used for black-box speech conversion under the guidance of the reference audio samples.

[0046] It should be noted that when the voice conversion adversarial audio generation device based on the conditional diffusion model provided in the above embodiment performs voice conversion adversarial audio generation, the division of the above functional modules should be used as an example to illustrate, and the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the voice conversion adversarial audio generation device based on the conditional diffusion model provided in the above embodiment and the voice conversion adversarial audio generation method embodiment based on the conditional diffusion model belong to the same concept, and the specific implementation process and effects are detailed in the voice conversion adversarial audio generation method embodiment based on the conditional diffusion model, which will not be repeated here.

[0047] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein an executable code is stored in the memory, and when the one or more processors execute the executable code, the method for generating voice conversion adversarial audio based on the conditional diffusion model is implemented, specifically including the following steps: S1, sample pair construction: prepare reference audio samples, generate adversarial audio samples in the voice conversion task based on the reference audio samples and adopt white-box attack method, and form sample pairs with reference audio samples and adversarial audio samples; S2, model training: taking the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as the guiding condition, the conditional diffusion model is trained based on the adversarial audio generation task; S3, Adversarial Audio Generation: Using the inverse generation process of the trained conditional diffusion model, under the guidance of the reference audio samples, adversarial audio related to the reference audio samples and used for black-box speech conversion is generated.

[0048] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the voice conversion adversarial audio generation method based on the conditional diffusion model described in S1-S3 above. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0049] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned method for generating voice conversion adversarial audio based on a conditional diffusion model is implemented, which specifically includes the following steps: S1, sample pair construction: prepare reference audio samples, generate adversarial audio samples in the voice conversion task based on the reference audio samples and adopt white-box attack method, and form sample pairs with reference audio samples and adversarial audio samples; S2, model training: taking the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as the guiding condition, the conditional diffusion model is trained based on the adversarial audio generation task; S3, Adversarial Audio Generation: Using the inverse generation process of the trained conditional diffusion model, under the guidance of the reference audio samples, adversarial audio related to the reference audio samples and used for black-box speech conversion is generated.

[0050] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0051] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating voice conversion adversarial audio based on a conditional diffusion model, characterized in that: The following steps are involved: Sample pair construction: prepare reference audio samples, generate adversarial audio samples in the voice conversion task based on the reference audio samples and adopt white-box attack method, and form sample pairs with reference audio samples and adversarial audio samples; Model training: Using the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as the guide condition, the conditional diffusion model is trained based on the adversarial audio generation task; Adversarial audio generation: Using the inverse generation process of the trained conditional diffusion model, under the guidance of the reference audio samples, adversarial audio related to the reference audio samples and used for black-box speech conversion is generated.

2. The method for generating voice conversion adversarial audio based on a conditional diffusion model according to claim 1, characterized in that: Generate adversarial audio samples in the voice conversion task based on reference audio samples and using white-box attack methods, including: By using adversarial audio samples and manually specified audio samples The representation distance is as close as possible to the reference audio sample As far away from the optimization goal as possible, find a noise disturbance factor , so that the adversarial audio sample In the white-box scenario, the converted audio attack generated by the speech conversion model is effective and sounds closer to the specified audio sample. timbre, based on which adversarial audio samples are generated.

3. The method for generating voice conversion adversarial audio based on conditional diffusion model according to claim 2, characterized in that: The specific process of generating adversarial audio samples is: Given parameter initial values , disturbance coefficient , characterizing the adjustment coefficient And the disturbance factor is required ,in, Represents the hyperbolic tangent function, generating adversarial audio samples based on the perturbation factor ; Preparing source audio samples , Reference Audio Samples , specify audio sample , using content encoder Source audio samples Encoding to obtain semantic representation , using the speaker encoder For clean audio samples , specify audio sample , and adversarial audio samples Encoding to get speaker representation , ,as well as ; Use decoder D to represent the pair ( , )、( , ),as well as( , ) to decode and get the converted audio , ,as well as ; Constructing a loss function based on converted audio , and the loss function Minimize the target parameters And find the optimal perturbation factor , and then generate adversarial audio samples , ； in, Calculates the difference between the two.

4. The method for generating voice conversion adversarial audio based on a conditional diffusion model according to claim 1, characterized in that: Taking the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as the guide condition, the conditional diffusion model is trained based on the adversarial audio generation task, including: In the forward diffusion process, the adversarial audio sample is used as the sample at time 0. , and at each time step t, Collecting real noise And add it to the generated sample obtained at the previous time step t-1 , get the sample for each time step t ,sample It obeys the prior Gaussian distribution and is expressed as: ； in, Indicates based on Generated by adding noise The prior Gaussian distribution of N represents the Gaussian distribution. It represents the real variance used when collecting real noise at time step t, and its value is 0-1. represents the mean of the Gaussian distribution, express Obey Gaussian distribution; In the forward diffusion process, the sample It is also input to the neural network contained in the conditional diffusion model M and use the reference audio sample as the guide condition c The same input to the neural network M Predicting noise , and based on real noise With prediction noise Calculated between norm, through the following loss function Update the Neural Network M Parameters , realize the training of conditional diffusion model; ； in, for norm, , represents the infinite norm, The larger the value, the more diverse the adversarial audio samples generated.

5. The method for generating voice conversion adversarial audio based on conditional diffusion model according to claim 4, characterized in that: The neural network M Adopt U-Net structure with residual connection; The reference audio samples are encoded and input into the neural network M .

6. The method for generating voice conversion adversarial audio based on conditional diffusion model according to claim 4, characterized in that: Using the inverse generation process of the trained conditional diffusion model, under the guidance of the reference audio sample, adversarial audio related to the reference audio sample and used for black-box speech conversion is generated, including: In the reverse generation process, the reference audio sample is also used as a guide condition c , from the normal distribution Random noise collected in As a sample at time T , and at each time step t Based on the bootstrap condition c , generate samples Through neural network M Prediction Noise , time step t The generated sample Subtract prediction noise That is, as the time step t -1 generated sample , generate samples It obeys the posterior Gaussian probability distribution, expressed as: ； in, Indicates based on Generated by denoising The posterior Gaussian distribution of Represents the neural network parameters The mean of the fit, Represents the neural network parameters The variance of the fit, express Obey Gaussian distribution; After 0-T steps of reverse generation, the generated sample at time step 0 Adversarial audio correlated with the reference audio sample for black-box speech conversion.

7. The method for generating voice conversion adversarial audio based on conditional diffusion model according to claim 1, characterized in that: Also includes: The generated adversarial audio is cropped to control the gap between the adversarial audio and the reference audio sample to be within a set range.

8. A speech conversion adversarial audio generation device based on a conditional diffusion model, characterized in that: include: A sample pair construction module is used to prepare reference audio samples, generate adversarial audio samples in the voice conversion task based on the reference audio samples and adopt a white-box attack method, and form a sample pair with the reference audio samples and the adversarial audio samples; A model training module, which is used to train the conditional diffusion model based on the adversarial audio generation task using the adversarial audio sample in the sample pair as input and the reference audio sample in the sample pair as a guide condition; The adversarial audio generation module is used to utilize the inverse generation process of the trained conditional diffusion model to generate adversarial audio related to the reference audio sample and used for black-box speech conversion under the guidance of the reference audio sample.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the voice conversion adversarial audio generation method based on the conditional diffusion model described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for generating speech conversion adversarial audio based on a conditional diffusion model as described in any one of claims 1 to 7 is implemented.

Images