Privacy protection noise generation method based on black box adversarial sample

Abstract

The invention provides a privacy protection noise generation method based on black box adversarial samples, and relates to the technical field of noise generation. Comprising the steps of constructing a voice data set; randomly generating initial white noise data; superposing the initial white noise data and target voice in the voice data set to obtain a superposed voice data set; performing data amplification on the superposed voice data set to obtain an amplified data set; inputting the amplified data set into a local white-box speech recognition model to obtain final white noise data; performing noise amplification on the final white noise data, and superposing the final white noise data with the voice data set to obtain a multi-intensity noise-added voice data set; and performing fine tuning training on the local white box model by using the multi-intensity noise-added voice data set to obtain a final noise generation model so as to generate final privacy protection noise. The technical problem that interference noise cannot be optimized based on an adversarial sample technology when the black box model parameters and architecture cannot be obtained and only the final recognition result can be obtained in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of noise generation technology, and in particular to a privacy-preserving noise generation method based on black-box adversarial samples. Background Art

[0002] With breakthroughs in artificial intelligence technologies like speech recognition and speech synthesis, smart IoT devices with voice interaction capabilities (including mobile devices, wearable devices, and smart home systems) have become deeply integrated into modern life, and their market penetration continues to rise. It's worth noting that while the exponential growth of these devices has increased convenience, it has also spawned new security risks. A large number of smart devices equipped with audio recording modules have permeated every aspect of daily life. Due to their closed architecture and long-term operation, users find it difficult to monitor device status in real time, posing a significant challenge to privacy protection. Criminals can remotely access user voice data and, using deep learning-driven speech recognition systems, extract the semantic content of voice signals to conduct information theft.

[0003] Therefore, there is an urgent need for privacy protection technology to return privacy rights to users. This protection technology should not affect the user's normal use of voice-activated smart devices, such as calls and voice commands. When users believe their voice privacy needs to be protected, they can actively adopt protection technology to resist malicious privacy theft, rather than relying solely on the manufacturer's privacy protection promises.

[0004] In response to this, some researchers have proposed ultrasonic jamming schemes. Their basic principle is to inject noise based on the nonlinearity of the device's microphone, thereby jamming eavesdropping devices without disturbing users in the environment. In "Wearable Microphone Jamming," Yuxin Chen et al. designed a wearable wristband with multiple ultrasonic transmitters that continuously emits ultrasound waves to jam recording devices in the environment. In "Patronus: Preventing Unauthorized Speech Recordings with Support for Selective Unscrambling," Lingkun Li et al. designed an ultrasonic transmitter that transmits variable-frequency noise based on a pre-generated key, jamming unauthorized recording devices while allowing authorized ones to record. While these various jamming methods can effectively inject noise into eavesdropping devices, they can also interfere with the normal operation of the microphone, impacting normal functions such as phone calls. Furthermore, these methods require specialized hardware, which users' smart voice devices often lack, making them unsuitable for everyday scenarios.

[0005] Another speech privacy protection technology exploits the vulnerabilities of neural networks and uses adversarial examples to generate subtle interference noise. This noise interferes with speech recognition models without disrupting normal user experience, thereby protecting user privacy. In "UniAP: Protecting Speech Privacy With Non-Targeted Universal Adversarial Perturbations," Peng Cheng et al. designed a privacy protection method based on white-box speech adversarial examples. This method generates universal, non-targeted adversarial examples to interfere with white-box speech recognition models, causing them to misinterpret any speech as incorrect. While this method can effectively interfere with white-box speech recognition models with known model parameters and architecture, it is less effective against unknown black-box models, such as commercial ones. Given that the majority of speech recognition models used in the backends of various smart devices are commercial black-box models, there is an urgent need to develop adversarial example generation methods that can generate interference noise for black-box models. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a privacy-preserving noise generation method based on black-box adversarial samples. The present invention solves the technical problem in the existing technology that when the black-box model parameters and architecture cannot be known and only the final recognition results can be obtained, it is impossible to optimize the interference noise based on the adversarial sample technology.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A privacy-preserving noise generation method based on black-box adversarial examples, comprising:

[0009] Build a speech dataset;

[0010] Randomly generate initial white noise data;

[0011] Superimposing the initial white noise data with the target speech in the speech data set to obtain a superimposed speech data set;

[0012] Performing data amplification on the superimposed speech data set to obtain an amplified data set;

[0013] Inputting the amplified data set into a local white-box speech recognition model, performing a gradient update on the perturbation of the local white-box speech recognition model based on a negative CTC loss and a two-norm regularization term until the average word error rate reaches a first preset threshold, thereby obtaining final white noise data;

[0014] Amplifying the final white noise data and superimposing the final white noise data with the speech data set to obtain a multi-intensity noisy speech data set;

[0015] The local white-box model is fine-tuned and trained using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate the final privacy-preserving noise.

[0016] Preferably, fine-tuning and training a local white-box model using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate final privacy-preserving noise includes:

[0017] Fine-tuning the local white-box model using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate final privacy-preserving noise includes:

[0018] Calculate the first word error rate between the black box recognition result and the correct text under the baseline strength;

[0019] Calculate the second word error rate between the black box recognition result and the local white box recognition result at each amplification strength;

[0020] Screening noisy speech with a second word error rate greater than a second preset threshold and its black box recognition result to form a training pair;

[0021] Preferably, superimposing the initial white noise data with the target speech in the speech data set to obtain a superimposed speech data set comprises:

[0022] Determining the length of the initial white noise data and the length of the target speech;

[0023] Determine whether the length of the initial white noise data is greater than the length of the target speech. If so, directly perform superposition. If not, perform length splicing on the initial white noise data until the length of the initial white noise data is greater than the length of the target speech and then perform superposition.

[0024] Preferably, performing data amplification on the superimposed speech data set to obtain an amplified data set includes:

[0025] adding noise to the superimposed speech data set to obtain intermediate data;

[0026] adding reverberation to the intermediate data to obtain an augmented data set;

[0027] The expression of the amplified data set is:

[0028] ;

[0029] in, To expand the dataset, is the target voice, is a superimposed speech dataset, where To expand the dataset, is the target voice, To superimpose speech datasets, repeat represents a repetitive algorithm, which cyclically splices the interference noise v. Shift represents an offset algorithm, which randomly cuts off and deletes the beginning noise fragments while keeping the total length of the noise consistent with the audio length, simulating a scenario where the audio and noise are not aligned. Augment represents a data augmentation method that adds noise and reverberation to the overall audio.

[0030] The present invention discloses the following technical effects:

[0031] The present invention provides a privacy-preserving noise generation method based on black-box adversarial samples, comprising: constructing a speech dataset; randomly generating initial white noise data; superimposing the initial white noise data with a target speech in the speech dataset to obtain a superimposed speech dataset; performing data amplification on the superimposed speech dataset to obtain an amplified dataset; inputting the amplified dataset into a local white-box speech recognition model, performing gradient update on the perturbation of the local white-box speech recognition model based on a negative CTC loss and a two-norm regularization term until an average word error rate reaches a first preset threshold, thereby obtaining final white noise data; performing noise amplification on the final white noise data and superimposing the data with the speech dataset to obtain a multi-intensity noisy speech dataset; and fine-tuning the local white-box model using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate final privacy-preserving noise. The interference noise generated in the present invention can achieve a stronger interference effect on the speech recognition model at the same energy, and can interfere with the speech recognition model without interfering with the normal use of the user; compared with the existing speech interference noise based on white-box adversarial samples, the interference noise generated in the present invention can achieve a stronger interference effect on the black-box speech recognition model at the same energy, greatly improving the word error rate (CER) of the black-box speech recognition model; users do not need to introduce a third party in the process of playing adversarial sample perturbations for privacy protection. Users can choose to use the privacy protection function when speaking, and the voice privacy right is returned to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 This is a flowchart of a privacy-preserving noise generation method based on black-box adversarial samples provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] like Figure 1 As shown, the present invention provides a privacy-preserving noise generation method based on black-box adversarial samples, comprising:

[0037] Build a speech dataset;

[0038] Specifically, a large amount of speech data containing different speakers and different speech content is obtained to build a speech dataset.

[0039] Randomly generate initial white noise data;

[0040] Superimposing the initial white noise data with the target speech in the speech data set to obtain a superimposed speech data set;

[0041] Performing data amplification on the superimposed speech data set to obtain an amplified data set;

[0042] Specifically, we generate interference noise on a locally deployed white-box speech recognition model:

[0043] 1) Initialize the interference noise length L and generate random white noise of corresponding length as the initial interference noise ;

[0044] Among them, whiteNois represents an algorithm for generating white noise of a specified length.

[0045] 2) Randomly select N voices x from the voice dataset i , , the interference noise is superimposed on the speech, where the interference noise length is short (for example, L = 0.5 seconds). The noise length is made longer than the speech length by cyclic replication, and a random offset is applied to simulate the situation in which the noise starting point cannot be aligned with the speech starting point in actual use. For example, the noise length L = 0.5 and the speech length len( )=3.1, then 8 noise segments are spliced ​​together cyclically to obtain , and randomly intercept 3.1 seconds from 0-0.5 seconds as the starting point, that is ;

[0046] 3) Speech with superimposed interference noise Perform data augmentation, including adding noise and reverberation processing, to obtain:

[0047]

[0048] Input the local white box speech recognition model for recognition and obtain the recognition result , using optimization algorithms (such as Adam or SGD), with the goal of keeping the model recognition results away from the correct semantic content of the speech, to optimize the interference noise:

[0049]

[0050]

[0051] in, For Voice Corresponding to the correct semantic text, Measures the distance between the noisy speech recognition result and the correct semantics. Adding a negative sign reverses the optimization direction, making the recognition result as far away from the correct result as possible, which interferes with the model recognition effect. It is a disturbance The second norm penalty term is used to avoid excessive disturbance amplitude; based on gradient descent, the disturbance Update, including is the learning rate; To expand the dataset, is the target voice, To superimpose the speech dataset, repeat represents the repetition algorithm, i.e., cyclically splicing the interference noise v; shift represents the offset algorithm, randomly cutting and deleting the beginning noise fragments, and keeping the total length of the noise consistent with the audio length, simulating the scenario where the audio and noise are not aligned; augment represents the method of adding noise and reverberation data to the overall audio. Refers to calculating the gradient of the loss function Loss(v) with respect to v.

[0052] 4) Continue optimizing until the average character error rate (CER) between the model recognition results of these noise-superimposed speech and the correct semantic text reaches a preset threshold;

[0053] In this step, by cyclically superimposing short disturbances, adding random offsets when optimizing disturbances, and collaboratively optimizing multiple different voices, the generated interference noise can effectively interfere with any content and any playback time.

[0054] Inputting the amplified data set into a local white-box speech recognition model, performing a gradient update on the perturbation of the local white-box speech recognition model based on a negative CTC loss and a two-norm regularization term until the average word error rate reaches a first preset threshold, thereby obtaining final white noise data;

[0055] Amplifying the final white noise data and superimposing the final white noise data with the speech data set to obtain a multi-intensity noisy speech data set;

[0056] Specifically, the optimized interference noise is amplified by different times and superimposed with the speech in the speech dataset:

[0057] ;

[0058] A batch of speech data with noise of different intensities is formed, where The noise amplification factor is set to 1, 2, 5, and 10 times. This factor is input into the black-box speech recognition model for recognition, and the model recognition results are recorded. The design of amplifying the interference noise in this step can amplify the noise energy when the existing interference noise has a poor interference effect on the black-box model, and find the difference in the recognition boundary between the white-box model and the black-box model.

[0059] The local white-box model is fine-tuned and trained using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate the final privacy-preserving noise.

[0060] Furthermore, fine-tuning the local white-box model using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate final privacy-preserving noise includes:

[0061] Fine-tuning the local white-box model using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate the final privacy-preserving noise, including:

[0062] Calculate the first word error rate between the black box recognition result and the correct text under the baseline strength;

[0063] Calculate the second word error rate between the black box recognition result and the local white box recognition result at each amplification strength;

[0064] Screening noisy speech with a second word error rate greater than a second preset threshold and its black box recognition result to form a training pair;

[0065] Specifically, the word error rate (CER) between the black-box model recognition results of the noisy speech at the baseline noise intensity and the correct semantic content of the corresponding speech is calculated; the word error rate (CER) between the black-box model recognition results of the noisy speech at all noise intensities and the local white-box model recognition results is calculated, and the noisy speech with a word error rate greater than 0.5 and the corresponding black-box model recognition results are retained;

[0066] The retained noisy speech and the corresponding black-box recognition results are used as training sets to fine-tune the local white-box model. This design compares the black-box model's recognition results with the white-box model's to locate the difference in the recognition boundaries between the two models. The white-box model is then fine-tuned and trained on a noisy speech dataset using the black-box model's recognition results as speech labels, allowing the white-box model to learn to approximate the black-box model's recognition boundaries. This allows interference noise generated for the white-box model to effectively interfere with the black-box model's recognition results.

[0067] Repeat the above steps until the word error rate (CER) between the black box model recognition result of the noisy speech with the baseline noise intensity and the correct semantic content of the corresponding speech reaches a third preset threshold. At this time, the interference noise is the interference noise generated by this method.

[0068] Specifically, the initial white noise data is superimposed with the target speech in the speech data set to obtain a superimposed speech data set, including:

[0069] Determining the length of the initial white noise data and the length of the target speech;

[0070] Determine whether the length of the initial white noise data is greater than the length of the target speech. If so, directly perform superposition. If not, perform length splicing on the initial white noise data until the length of the initial white noise data is greater than the length of the target speech and then perform superposition.

[0071] The present invention also proposes another embodiment:

[0072] Deploy a white-box speech recognition model (such as ESPnet) on a local server and build a speech dataset.

[0073] Generate interference noise in the local white-box model;

[0074] The data is then combined with the speech data and sent to a commercial black box speech recognition model for recognition (such as Tencent Cloud and Alibaba Cloud).

[0075] Fine-tune the local white-box model based on the black-box model recognition results;

[0076] Iterate until you get interference noise that has a good interference effect on the black box model;

[0077] Using smart devices that may steal voice privacy (such as smartphones and smart speakers) to play interference noise in a loop can protect user voice privacy.

[0078] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0079] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A privacy-preserving noise generation method based on black-box adversarial samples, characterized in that: include: Build a speech dataset; Randomly generate initial white noise data; Superimposing the initial white noise data with the target speech in the speech data set to obtain a superimposed speech data set; Performing data amplification on the superimposed speech data set to obtain an amplified data set; Inputting the amplified data set into a local white-box speech recognition model, performing a gradient update on the perturbation of the local white-box speech recognition model based on a negative CTC loss and a two-norm regularization term until the average word error rate reaches a first preset threshold, thereby obtaining final white noise data; Amplifying the final white noise data and superimposing the final white noise data with the speech data set to obtain a multi-intensity noisy speech data set; The local white-box model is fine-tuned and trained using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate the final privacy-preserving noise.

2. The privacy-preserving noise generation method based on black-box adversarial samples according to claim 1, characterized in that: Fine-tuning the local white-box model using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate final privacy-preserving noise includes: Fine-tuning the local white-box model using the multi-intensity noisy speech dataset to obtain a final noise generation model to generate final privacy-preserving noise includes: Calculate the first word error rate between the black box recognition result and the correct text under the baseline strength; Calculate the second word error rate between the black box recognition result and the local white box recognition result at each amplification strength; Screening noisy speech with a second word error rate greater than a second preset threshold and its black box recognition result to form a training pair; The local white-box model is trained using the training pairs until the first word error rate reaches a third preset threshold.

3. The privacy-preserving noise generation method based on black-box adversarial samples according to claim 1, characterized in that: Superimposing the initial white noise data with the target speech in the speech data set to obtain a superimposed speech data set includes: Determining the length of the initial white noise data and the length of the target speech; Determine whether the length of the initial white noise data is greater than the length of the target speech. If so, directly perform superposition. If not, perform length splicing on the initial white noise data until the length of the initial white noise data is greater than the length of the target speech and then perform superposition.

4. The privacy-preserving noise generation method based on black-box adversarial samples according to claim 1, characterized in that: The step of amplifying the superimposed speech data set to obtain an amplified data set includes: adding noise to the superimposed speech data set to obtain intermediate data; adding reverberation to the intermediate data to obtain an augmented data set; The expression of the amplified data set is: ； in, To expand the dataset, is the target voice, To superimpose speech datasets, repeat represents a repetitive algorithm, which cyclically splices the interference noise v. Shift represents an offset algorithm, which randomly cuts off and deletes the beginning noise fragments while keeping the total length of the noise consistent with the audio length, simulating a scenario where the audio and noise are not aligned. Augment represents a data augmentation method that adds noise and reverberation to the overall audio.

Images