Privacy protection method and device based on white-box speech adversarial samples
By generating and looping non-target white boxes that meet the actual application scenarios to combat sample perturbation, the content irrelevance and shift irrelevance problems in voice privacy protection are solved, and the effective interference of the voice recognition model is achieved to ensure that user privacy is not leaked. And reduces repetition, and users can use privacy protection functions independently.
Patent Information
- Application Number
- CN202210996591.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-21
- Filing Date
- 2022-08-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-08-19
AI Technical Summary
The prior art is difficult to achieve effective protection of voice privacy without affecting the normal use of voice smart devices by users, especially in white box counter-sample sample generation under the constraints of content irrelevance and shift irrelevance, and real-time interference methods. There is a third-party trust issue.
By obtaining a large number of voice data of different speakers and contents, building a voice data set, setting block perturbation and performing random white noise initialization, simulating the impact of the real physical world, and updating the perturbation using optimization algorithms to generate non-target whites that meet the actual application scenarios The box fights against sample perturbation, forms a set of perturbations and plays them loop, ensuring the effectiveness of perturbations under different initial conditions and user-autonomous control.
It realizes effective content irrelevance and shift irrelevance perturbation in the voice recognition model, reducing repetition, and users can use privacy protection functions independently to avoid voice privacy leakage, while not affecting voice calls and listening experience.
Smart Images

Figure CN115208507B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of (voice privacy protection), and specifically, is a privacy protection method and device based on white-box voice adversarial samples. Background Art
[0002] Mobile phones, voice-controlled locks, smart speakers, and voice-controlled smart devices are increasingly becoming an important part of people's daily lives. As the main audio sensor supporting the functions of voice-controlled smart devices, the deployment of microphones continues to increase, and concerns about microphones exposing user privacy are gradually emerging. This risk can be summarized into two main scenarios: when users use voice-controlled smart devices (such as using software with voice functions on mobile phones), the voice recorded by the microphone may be analyzed without permission; and without the user's knowledge, malware can illegally obtain the user's voice through the microphone and obtain the privacy information contained therein. Due to the need for user portrait analysis, the monitor will obtain a large amount of voice data; and due to the large scale of voice data, the monitor will tend to use voice recognition software to identify the voice data, thereby reducing labor costs. This risk is real. In 2019, Google and Amazon admitted that they would analyze users' voice data to improve service quality.
[0003] Now, a privacy protection technology is needed to return privacy rights to users. This protection technology should not affect users' normal use of voice intelligent devices, such as calls, voice commands, etc. When users believe that their voice privacy needs to be protected, users can actively adopt protection technology to resist malicious privacy theft, rather than relying solely on the manufacturer's privacy protection commitments.
[0004] In this regard, some researchers have proposed using ultrasound to block the normal recording of the microphone, but this will interfere with the normal operation of the microphone and affect normal functions such as calls; at the same time, this method requires special hardware equipment, which is often not available on the user's smart voice device, so this method is not suitable for daily life scenarios. In addition, some privacy protection methods introduce a third party to intercept and modify the recorded audio, but this involves the question of whether the third party is trustworthy.
[0005] With the development of deep learning technology, in the field of speech recognition, end-to-end methods based on neural networks have gradually replaced traditional methods based on hidden Markov models and have become a hot topic of research and application. The speech recognition system based on traditional methods consists of multiple modules including acoustic models, language models and decoders. Recognizing speech requires the coordination of multiple modules and the process is complicated. The speech recognition process based on neural networks is relatively simple. The speech audio is input into the neural network after feature processing, and the network can directly output the corresponding text of the audio. The characteristics of the neural network can be directly used for privacy protection.
[0006] Szegedy et al. first revealed in "Intriguing properties of neural networks" that adding disturbances to the input of a neural network can mislead the correct classification of the neural network; the added disturbances are very subtle, and it is difficult for humans to distinguish between the inputs before and after the disturbances are added. This type of input that has the ability to mislead after superimposing disturbances is called an adversarial sample. According to the type of misleading classification, adversarial samples can be divided into target classes and non-target classes. The target class refers to misleading the input to a specific classification, and the non-target class refers to misleading the input to a non-real classification; depending on whether the structural parameters of the model need to be obtained when generating adversarial samples, adversarial samples can be divided into white box and black box. Adversarial samples that need to be optimized based on the structure and parameters of the target model when generating adversarial samples are called white box adversarial samples, while those that do not need to be optimized are called black box adversarial samples. In the field of speech adversarial samples, a classic example is the article Audio Adversarial Examples: Targeted Attacks on Speech-to-Text by Nicholas Carlini and David Wagner. The article superimposes perturbations on the audio, uses the perturbations as optimization variables, optimizes the model loss function and regularization terms, and generates perturbations with low energy that enable the model to recognize speech as target text.
[0007] Based on this, a privacy protection technology solution based on voice adversarial samples can be proposed: when the user is speaking and believes that the current content needs to be protected, the audio player can be opened to play the adversarial sample perturbation; assuming that at the same time, malware on a smart voice device is listening with the help of a microphone, the software will record the audio superimposed with the adversarial sample perturbation, and after the audio is input into the speech recognition model, it cannot be correctly recognized, and the privacy information cannot be obtained. This technical solution has several advantages: 1. The energy of the adversarial sample perturbation is low, the hearing is good, and the impact on the user is small; at the same time, it will not affect the normal operation of the voice call; 2. No special hardware equipment is required, and common smart devices with ordinary speakers can play adversarial sample perturbations; 3. The playback of adversarial sample perturbations is controlled by the user, giving the user the initiative to protect privacy.
[0008] For the task of protecting speech privacy, it is not necessary to perturb the speech into a specific text, but it is necessary to perturb the model to mislead the model and obtain the wrong recognition text. Therefore, unlike the target adversarial samples in the research of Nicholas Carlini and David Wagner, the present invention focuses on non-target speech adversarial samples.
[0009] In the field of speech recognition, the open source community is booming; open source speech recognition software based on neural networks, such as Espnet, DeepSpeech and WeNet, provides trained model structures and parameters, is easy to deploy, and has been widely used; the convenience of open source software makes some companies tend to use it directly, or conduct secondary development based on open source software, rather than developing a complete speech recognition system on their own. Since it is difficult to generate black-box adversarial samples directly for commercial speech recognition systems and there is a lack of mature generation algorithms; while there are mature technical routes that can be used to generate white-box adversarial samples for open source speech recognition systems, and the generated adversarial samples will be more easily generalized to commercial speech recognition software based on open source recognition systems, thereby preventing eavesdroppers from using such speech recognition to obtain user privacy information. Therefore, the present invention generates white-box adversarial samples for some common open source speech recognition systems.
[0010] Common adversarial sample research directly superimposes disturbances with audio in the data domain and inputs them into the recognition model, and only verifies them in the digital domain; in the real physical world, affected by the hardware characteristics of the audio player, the propagation space and environmental noise, the misleading ability of adversarial samples recorded by microphones is greatly reduced. In this regard, Hiromu Yakura and Jun Sakuma proposed a method for generating adversarial samples that can maintain effective interference in the real physical world in "Robust Audio Adversarial Example for a Physical Attack". By introducing bandpass filtering, room impulse response, and white noise in the training process, the effects of audio players (frequency band selectivity), propagation space (reverberation) and environmental noise on speech signals when used in real scenarios are simulated respectively. The present invention will draw on this technical solution to ensure that the generated adversarial sample perturbations are effective in the real physical world.
[0011] At the same time, in practical applications, it is impossible to predict what the user will say when using a voice device, as well as the exact time of speaking, which poses two constraints: 1. No matter what the speaker says, the adversarial sample perturbation can prevent the speech recognition model from correctly recognizing the recorded perturbed speech; this constraint can be referred to as content independence; 2. When the perturbation starts playing, the time when the speaker starts speaking is not fixed; the perturbation needs to work at any time when the speaker starts speaking; this constraint can be referred to as shift independence. Most existing studies are based on the same setting: the perturbation is only generated for a specific voice segment; the perturbation and audio are of the same length, and start and end at the same time. This means that when the voice segments are different, or when the perturbation and audio start at different times, the misleading ability of the adversarial sample will be affected. In this regard, Zhuohang Li et al. proposed in AdvPulse: Universal, Synchronization-free, and Targeted Audio Adversarial Attacks via Subsecond Perturbations to collect a large amount of speech instead of training on a single speech segment to improve the content independence of the disturbance. They designed a segment disturbance with a length of less than 1 second, which is different from the traditional disturbance of the same length as the speech. When generating the disturbance, the disturbance is randomly shifted to simulate the situation in the real scene where the user's starting time of speaking cannot be accurately predicted, so as to improve the displacement independence of the disturbance. However, the speech recognition model interfered by this method is a relatively simple voice command classification model, not a complete speech recognition model. Generating content- and displacement-independent disturbances on a complete speech recognition model remains to be solved. Mia Chiquier et al. proposed a method for real-time interference in Real-Time Neural Voice Camouflage. The neural network obtains the speech at the current time, predicts and generates effective disturbances in the future, continuously obtains speech and continuously generates disturbances, so that the effective disturbance can completely cover the entire speech. However, this method requires the use of an additional microphone to collect the speaker's voice in real time, and it also has its own trust issues in the user's third party.
[0012] In summary, existing solutions have multiple problems:
[0013] 1. Most white-box adversarial sample generation strategies cannot meet the content independence and shift independence constraints in practical applications;
[0014] 2. The adversarial sample perturbation generation strategy that can meet the constraints of practical applications has only been verified to be effective on the voice command recognition model, and the perturbation effect on the voice recognition model has not been verified;
[0015] 3. The real-time interference method requires the additional use of a microphone, which raises trust issues with third parties. Summary of the invention
[0016] The present invention improves upon the deficiencies of the prior art and proposes a privacy protection method and device based on white-box speech adversarial samples. The interference method for open source speech recognition models based on non-target white-box adversarial samples is effective in the real physical world and complies with the constraints of actual application scenarios, so as to achieve speech privacy protection without affecting the normal use of speech intelligent devices by users. Users can autonomously play specially generated disturbances while speaking, so that the recorded disturbed speech cannot be correctly recognized by the speech recognition software, thereby avoiding the leakage of privacy information.
[0017] The objective of the present invention is achieved through the following technical solutions:
[0018] The present invention discloses a privacy protection method based on white-box speech adversarial samples, comprising:
[0019] 1) Obtain a large amount of speech data containing different speakers and different speech contents to build a speech dataset;
[0020] 2) Set the block perturbation length and initialize the block perturbation with random white noise to obtain the initial block perturbation;
[0021] 3) Obtain N randomly sampled voices from the voice data set;
[0022] 4) The initial block disturbance is superimposed on the speech obtained in the previous step and processed to simulate the impact of the real physical world on the disturbance, and N disturbed speech sounds are obtained after multiple processing;
[0023] 5) Optimize and update the N disturbed speech after multiple processing based on a specific optimization algorithm, so as to increase the difference between the model recognition text and the real text and keep the disturbance energy low as the optimization goal, and obtain the block disturbance after optimization and update;
[0024] 6) Obtain M optimized and updated block disturbances under different initial disturbance conditions, and arbitrarily select at least one of them to form a disturbance set;
[0025] 7) The block perturbations in the perturbation set are played in a random order in a loop. After a block perturbation is finished playing, the next randomly selected block perturbation is played immediately. This process is repeated to achieve continuous interference.
[0026] As a further improvement, the initialization in step 2) of the present invention is to initialize the length and initial content of the disturbance, the length is half a second, and the initial content is white noise generated by different random number seeds.
[0027] As a further improvement, in step 3) of the present invention, the acquired speech is generated by random sampling of the speech data set.
[0028] As a further improvement, in step 4) of the present invention, the processing of the N disturbed speech is performed by randomly delaying the speech and / or low-pass filtering and / or room response impact and / or superimposing the ambient noise.
[0029] As a further improvement, in step 5) of the present invention, the specific optimization algorithm is SGD or ADAM, etc.; the optimization update process includes N loops; in each loop, all the disturbed speech that has undergone multiple processing is input into the speech recognition model to increase the difference between the model recognized text and the real text and keep the disturbance energy low as the optimization goal, and output block disturbance.
[0030] As a further improvement, in step 5) of the present invention, the block perturbation obtained after optimization and updating is a non-target white-box adversarial sample perturbation that is effective for different speakers and different speech contents.
[0031] As a further improvement, in step 6) of the present invention, M block disturbances after optimization and updating under different initial disturbance conditions are obtained, and at least two of them are arbitrarily selected to form a disturbance set.
[0032] The present invention also discloses a privacy protection device based on white-box speech adversarial samples, comprising:
[0033] Acquisition unit: used to acquire a large amount of speech data containing different speakers and different speech contents, and build a speech data set;
[0034] Initialization unit: used to set the block disturbance length and initialize the block disturbance with random white noise to obtain the initial block disturbance;
[0035] Sampled speech unit: used to obtain N randomly sampled speech in the speech data set;
[0036] Obtaining a disturbed speech unit: used to superimpose the initial block disturbance on the speech obtained in the previous step and process it, simulating the influence of the real physical world on the disturbance, and obtaining N disturbed speech after multiple processing;
[0037] Update unit: used to optimize and update N disturbed speech after multiple processing based on a specific optimization algorithm, so as to increase the difference between the model recognition text and the real text and keep the disturbance energy low as the optimization goal, and obtain the block disturbance after optimization and update;
[0038] Constructing a disturbance collection unit: used to obtain M block disturbances after optimization and update under different initial disturbance conditions, and arbitrarily select at least one of them to form a disturbance set; specifically, the three units of sampling speech unit, obtaining perturbed speech unit and updating unit are cyclically sampled multiple times, and the cycle is stopped when the optimization upper limit is reached, thereby obtaining a block disturbance; repeating the above process M times to obtain M block disturbances after optimization and update under different initial disturbance conditions, and arbitrarily select at least two of them to form a disturbance set;
[0039] Interference unit: It is used to play the block disturbances in the disturbance set in a random order. After one block disturbance is finished playing, the next randomly selected block disturbance is played immediately. This process is repeated to achieve continuous interference.
[0040] At present, the issue of voice privacy has received widespread attention. Potential eavesdroppers can use the microphones of ubiquitous smart audio devices to obtain a large amount of user voice data and quickly analyze it based on voice recognition software to achieve goals such as user portrait analysis and accurate advertising push. Based on this, the present invention discloses a privacy protection method and device for large-scale semantic extraction based on a non-target adversarial sample interference white-box voice recognition model. The present invention designs a sub-second block disturbance to achieve effective interference with the target voice recognition model in actual application scenarios. In the block disturbance generation stage, the defender establishes a voice database, and generates non-target white-box adversarial sample disturbances that meet the constraints of actual application scenarios for the open source voice recognition model; based on different initial states, the defender can generate a large number of different block disturbances to form a disturbance set. In the privacy protection stage, the defender randomly selects disturbances from the disturbance set for playback, so that the eavesdropper cannot obtain the block disturbance waveform and cannot remove the disturbance; the voice superimposed with the disturbance cannot be correctly recognized by the model after being recorded by the microphone, thereby protecting the user's privacy from being leaked. The system includes a disturbance generation unit and a privacy protection unit. The present invention can generate block disturbances with less interference to humans, resist privacy leakage from large-scale speech recognition, and thus return speech privacy rights to users.
[0041] Specifically, the beneficial effects of the present invention are:
[0042] (1) The perturbations are generated based on speech data containing different speakers and different speech contents, achieving the content independence of the perturbations; the consideration of the non-fixed speaking time of the speakers is taken into account during the perturbation generation process, achieving the shift independence of the perturbations;
[0043] (2) Adversarial sample perturbations are effective for speech recognition models, surpassing solutions that are only effective for speech command recognition models;
[0044] (3) Users do not need to introduce a third party when playing adversarial sample perturbations to protect privacy. Users can choose to use the privacy protection function when speaking, returning the voice privacy rights to the users.
[0045] (4) In step (6) of the present invention, at least two block disturbances are selected to form a disturbance set to reduce repeatability. The higher the number, the lower the repeatability. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A flow chart of a voice privacy protection method based on white-box adversarial samples provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0047] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0048] S1: Construct a speech dataset. The content of the speech dataset should be as rich as possible, covering the speakers’ different ages, genders, accents, and the diversity of their speech content.
[0049] S2: Set the block disturbance length to L and use random white noise of length L Block disturbance Perform initialization to obtain initial block disturbance;
[0050]
[0051] S3: Get N voices in the voice data set. Each voice is randomly sampled from the voice data set and is represented as ;
[0052] S4: Block disturbance Perform a cycle ( ), which is superimposed with each speech and randomly delayed relative to the cyclic block perturbation when superimposed ( ), to simulate the uncertain time when the user starts speaking, the random delay can be set to a minimum value, such as half a second, to characterize the speaker's reaction time between using the privacy protection function and speaking;
[0053]
[0054] N disturbed speech is processed to simulate the influence of disturbance in the real physical world, and N disturbed speech is obtained after multiple processing, among which bandpass filtering Frequency selectivity and unit impulse response of analog playback equipment Convolution simulates the reverberation of sound propagating in the air, and superimposes white noise to simulate environmental noise;
[0055] ;
[0056] S5: Optimize and update the N disturbed speech after various processing based on a specific optimization algorithm to increase the difference between the model recognition text and the real text and keep the disturbance energy low as the optimization goal , obtain the block disturbance after optimization and update; is the speech recognition model, It is the disturbed speech after the disturbance effect of the real physical world is simulated in step S5. is the real text corresponding to the speech. It is used to measure Model output text and real text , It is a disturbance The second norm penalty term of ; based on the gradient descent perturbation Update, including is the learning rate;
[0057]
[0058] S6: Obtain M block disturbances after optimization and updating under different initial disturbance conditions, and arbitrarily select at least one of them to form a disturbance set; here it can also be at least two, and at least two block disturbances are selected to form a disturbance collection to reduce repeatability. The higher the number, the lower the repeatability.
[0059] S7: The block disturbances in the disturbance set are played in a random order in a loop. After a block disturbance is finished playing, the next randomly selected block disturbance is played immediately. This process is repeated to achieve continuous disturbance.
[0060] The present invention also discloses a privacy protection device based on white-box speech adversarial samples, comprising:
[0061] Acquisition unit: used to acquire a large amount of speech data containing different speakers and different speech contents, and build a speech data set;
[0062] Initialization unit: used to set the block disturbance length and initialize the block disturbance with random white noise to obtain the initial block disturbance;
[0063] Sampled speech unit: used to obtain N randomly sampled speech in the speech data set;
[0064] Obtaining a disturbed speech unit: used to superimpose the initial block disturbance on the speech obtained in the previous step and process it, simulating the influence of the real physical world on the disturbance, and obtaining N disturbed speech after multiple processing;
[0065] Update unit: used to optimize and update N disturbed speech after multiple processing based on a specific optimization algorithm, so as to increase the difference between the model recognition text and the real text and keep the disturbance energy low as the optimization goal, and obtain the block disturbance after optimization and update;
[0066] Constructing a disturbance collection unit: used to obtain M block disturbances after optimization and update under different initial disturbance conditions, and arbitrarily select at least one of them to form a disturbance set; specifically, the three units of sampling speech unit, obtaining perturbed speech unit and updating unit are cyclically sampled multiple times in sequence, and the cycle is stopped when the optimization upper limit is reached, thereby obtaining a block disturbance; repeating the above process M times to obtain M block disturbances after optimization and update under different initial disturbance conditions, and arbitrarily select at least two of them to form a disturbance set.
[0067] Interference unit: It is used to play the block disturbances in the disturbance set in a random order. After one block disturbance is finished playing, the next randomly selected block disturbance is played immediately. This process is repeated to achieve continuous interference.
[0068] There are two prerequisites for the implementation of the technical solution of the present invention: first, white-box speech adversarial samples need to be generated for speech recognition models with known parameters and structures, and cannot be generated for models with unknown parameters and structures; in addition, the adversarial sample perturbations generated for a specific speech recognition model can only play an interference effect on this model, and the interference effect on other speech recognition models is often weak. However, these restrictions do not affect the practicability of the present invention. Since building a speech recognition system by oneself requires a certain technical threshold, a large number of companies that need to perform large-scale semantic extraction will achieve this function by directly deploying or partially transforming open source speech recognition models. This creates conditions for obtaining parameters through open source speech recognition models. By generating adversarial sample perturbations for all mainstream open source speech recognition models, the defect of generating adversarial sample perturbations only for a single speech recognition model can be overcome.
[0069] The present invention experiments with the above-mentioned privacy protection method based on white-box adversarial samples.
[0070] Experiment 1 is mainly to verify the effectiveness of this protection method and the low impact of disturbance on people. The experimental scene sets up a disturbance sound source, a speech sound source and a recording device. The disturbance sound source randomly selects block disturbances from the disturbance set to play, the speech sound source plays English speech, and the recording device records the disturbed speech. The disturbed speech is input into the speech recognition model (in this experiment, DeepSpeech, a mainstream open source speech recognition model, is used), and the character error rate (CER, a criterion for measuring the difference between the recognition result and the real text, the larger the value, the greater the difference) between the recognition result and the real text is calculated, and the interference success rate JSR is calculated (for a certain speech, if the recognition CER is greater than 0.5, it is considered that the interference is successful, and JSR is the ratio of the number of successfully interfered speech to the number of all speech). In addition, the experiment also measured the noise-to-signal ratio NSR (the ratio of disturbance energy to speech energy. The smaller the value, the lighter the disturbance is and the smaller the impact on people). The experimental results show that the interference success rate JSR is 90.2%, indicating that most speech can be successfully interfered with, and the noise-to-signal ratio NSR is -39.7 dB, indicating that the disturbance energy is much smaller than the speech energy and the disturbance has little impact on people.
[0071] Experiment 2 is mainly to verify the advantages of using adversarial sample perturbations to interfere with speech recognition models compared to other traditional noises. The experiment compared the signal-to-noise ratio (SNR) of adversarial sample perturbations and white noise when they reached a specific JSR (in contrast to the signal-to-noise ratio, the ratio of speech energy to perturbation energy is measured. The larger the value, the lighter the perturbation is and the smaller the impact on people). In this experiment, the SNR when the adversarial sample perturbation reached a JSR of more than 90% was about 24dB, while the SNR when the white noise reached more than 60% was about -2dB. Adversarial sample perturbations have huge advantages over white noise.
[0072] Application Scenario
[0073] The present invention is applicable to protecting the voice privacy of users in the Internet of Everything scenario. The types and number of smart devices around people are gradually increasing. Smart devices include but are not limited to smart phones, wearable devices, and smart speakers represented by Tmall Genie, Baidu Xiaodu, etc., as well as various smart home devices. In order to provide users with a variety of rich functions and improve the user experience, these devices generally use voice commands as the main human-computer interaction method. Therefore, most of today's smart devices have an acoustic sensor element, a microphone, and have the hardware ability to record user voices. Considering the voice function of these devices, users usually agree to the permission request of microphone data of smart applications running on these devices. In today's big data context, user data is a core resource, and voice data is one of the most important user data. This resource can be used for algorithm improvement, precision advertising, data education, etc. The news that Amazon and Google have been exposed in recent years to use voice data recorded by smart speakers to improve their own services also proves the economic value of voice data. Driven by this interest, and the current situation that there are multiple recording devices in the user and the environment in which they are located, the user's voice privacy may be illegally infringed and used for profit. The purpose of the present invention is based on this background to protect the user's voice privacy.
[0074] Assuming that there is an organization that attempts to profit from user voice, they usually use two methods to secretly record voice. The first method is to use computer hacking technology to control smart devices with microphone recording capabilities (such as smart TVs or smart speakers) in the user's environment, and then turn on the recording, so that a large number of recordings can be obtained without the user's knowledge; the second method is to release some smart apps (such as smartphone applications), disguised as normal software with voice input and other functions, obtain recording permissions, and start recording without the user's knowledge, thereby obtaining a large amount of data. After obtaining a large amount of voice data through the above two methods, the privacy stealer usually does not manually listen to the recording and manually translate the voice content, but uses voice recognition technology to automatically convert voice to text, thereby obtaining a large amount of text containing private information such as user interests, hobbies, and living habits.
[0075] Based on the specific privacy theft scenarios described above, the technology proposed in this invention can protect the user's voice from being secretly recorded by malicious apps or smart devices hijacked by hackers in the environment. Under reasonable restrictions, the recording cannot be correctly recognized by the automatic speech recognition model. In addition, during the operation of the privacy protection technology, the user's normal voice call function will not be disturbed, and the hearing experience will not be greatly affected.
[0076] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.
Claims
1. A privacy protection method based on white-box speech adversarial samples, characterized in that: include: 1) Obtain a large amount of speech data containing different speakers and different speech content to build a speech dataset; 2) Set the block disturbance length L, and initialize the block disturbance v with random white noise whiteNoise(L) of length L to obtain the initial block disturbance; v←whiteNoise(L); 3) Obtain N voices in the voice data set, each of which is randomly sampled from the voice data set, denoted as x i , i=1,2,...N; 4) Repeat the block disturbance v in a loop and set a random delay shift to simulate the uncertain time when the user starts speaking. The random delay is set to a minimum value to characterize the speaker's reaction time between using the privacy protection function and speaking. Then, the cyclic block disturbance repeat(v) with random delay is subjected to bandpass filtering to simulate the frequency selectivity of the playback device; and then it is superimposed with the speech signal to obtain N disturbed speech; N disturbed speech sounds are processed to simulate the influence of disturbance in the real physical world, and the processed N disturbed speech sounds are obtained, wherein the convolution with the unit impulse response h simulates the reverberation of sound propagating in the air, and the white noise w is superimposed to simulate the environmental noise; 5) Optimizing and updating the above-processed N disturbed speech based on the optimization algorithm, so as to increase the difference between the model recognition text and the real text and keep the disturbance energy low as the optimization target J(v), and obtain the block disturbance after optimization and update; Where f is the speech recognition model, is the disturbed speech after simulating the disturbance effect of the real physical world in step 4), y i is the real text corresponding to the speech, and ctcloss is used to measure Model output text and real text y i The difference function, ||v|| is the two-norm penalty term of the block perturbation v; the block perturbation v is updated based on gradient descent: Where α is the learning rate; 6) Obtain M optimized and updated block disturbances under different initial disturbance conditions, and arbitrarily select at least 2 of them to form a disturbance set; 7) The block disturbances in the disturbance set are played cyclically in a random order. After one block disturbance is finished playing, the next randomly selected block disturbance is played immediately, and this process is repeated to achieve continuous interference; the continuous interference is aimed at illegal eavesdroppers of the voice.
2. The privacy protection method based on white-box speech adversarial samples according to claim 1 is characterized in that: The initialization in step 2) is to initialize the length and initial content of the disturbance, the length is half a second, and the initial content is white noise generated by different random number seeds.
3. The privacy protection method based on white-box speech adversarial samples according to claim 1 is characterized in that: In the step 5), the optimization algorithm is SGD or ADAM; the optimization update process includes N cycles; in each cycle, all processed disturbed speech is input into the speech recognition model to increase the difference between the model recognized text and the real text and keep the disturbance energy low as the optimization goal, and output block disturbance.
4. The privacy protection method based on white-box speech adversarial samples according to claim 1, 2 or 3, characterized in that: In the step 5), the block perturbation obtained after optimization and updating is a non-target white-box adversarial sample perturbation that is effective for different speakers and different speech contents.
5. A privacy protection device based on white-box speech adversarial samples, characterized in that: include: Acquisition unit: used to acquire a large amount of speech data containing different speakers and different speech contents, and build a speech data set; Initialization unit: used to set the block disturbance length L, and initialize the block disturbance v with random white noise whiteNoise(L) of length L to obtain the initial block disturbance; v←whiteNoise(L); Sampled speech unit: used to obtain N speech in the speech data set. Each speech is randomly sampled from the speech data set and is represented by x i , i=1,2,...N; Obtaining a disturbed speech unit: for repeating the block disturbance v in a loop, and setting a random delay shift to simulate the uncertain time when the user starts speaking, wherein the random delay is set to a minimum value to characterize the speaker's reaction time between using the privacy protection function and speaking; then performing a bandpass filtering process on the cyclic block disturbance repeat (v) with a random delay to simulate the frequency selectivity of the playback device, and then superimposing it with the speech signal to obtain N disturbed speech; processing the N disturbed speech to simulate the influence of the real physical world on the disturbance, and obtaining the processed N disturbed speech, wherein the convolution with the unit impulse response h simulates the reverberation existing in the propagation of sound in the air, and superimposing the white noise w to simulate the environmental noise; Update unit: used to optimize and update the above processed N disturbed speech based on the optimization algorithm, so as to increase the difference between the model recognition text and the real text and keep the disturbance energy low as the optimization target J(v), and obtain the block disturbance after optimization and update: Where f is the speech recognition model, is the disturbed speech after simulating the disturbance effect of the real physical world in step 4), y i is the real text corresponding to the speech, and ctcloss is used to measure Model output text and real text y i The difference function of ||v|| is the second norm penalty term of the block perturbation v; Update the block perturbation v based on gradient descent: Where α is the learning rate; Constructing a disturbance collection unit: used to obtain M optimized and updated block disturbances under different initial disturbance conditions, and arbitrarily selecting at least 2 of them to form a disturbance set; Interference unit: used to play the block disturbances in the disturbance set in a random order. After a block disturbance is finished playing, the next randomly selected block disturbance is played immediately. This process is repeated to achieve continuous interference. The continuous interference is directed against illegal eavesdroppers of the speech.
