A method and system for protecting user voice privacy

Abstract

This invention discloses a method and system for protecting user voice privacy. It automatically starts / stops playing generated adversarial perturbation audio to deceive the voice assistant without affecting the user's ability to wake up the voice assistant and interact with it. The adversarial perturbation audio is generated offline by a black-box adversarial attack model based on reinforcement learning proposed in this invention. This invention uses a Raspberry Pi to build the system prototype and conducts relevant experiments on commercial voice assistants. It is low-cost, easy to deploy, and can protect user voice privacy well with minimal impact on user experience.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to a method and system for protecting user voice privacy. Background Technology

[0002] Voice has become a widely used human-computer interaction method, especially in smart terminals in smart homes and smart cars. Voice recognition technology is the foundation of voice interaction; it converts human speech into computer-readable text, thus enabling human-computer interaction. While the development of voice recognition technology has made human-computer interaction more convenient, it has also brought about some security and privacy issues. For example, when a user uses a voice assistant, their voice information is sent to the cloud. The cloud-based voice recognition system converts the voice information into text and performs semantic understanding before responding to the user. However, the user's private voice conversations can also be accessed by the cloud-based voice recognition system, leading to serious privacy and security problems. Therefore, protecting users' voice privacy has become an important issue.

[0003] Existing voice privacy protection technologies are mainly based on two approaches: one is to directly attack the physical device, and the other is to attack the voice recognition model.

[0004] Attacks on physical devices primarily target the microphone, for example, by sending noise (such as ultra-high-energy ultrasound) to the microphone, causing it to reach its acoustic overload point (AOP), thus severely distorting the recorded sound and supposedly protecting user privacy. However, this method has a drawback: while the ultrasound waves are inaudible to humans, they are audible to animals with sensitive hearing (such as cats and dogs). Furthermore, this method relies on an additional microphone to determine if the user is speaking, thereby controlling the transmission and cessation of noise. Therefore, this approach does not fundamentally solve the problem of voice privacy protection because users cannot fully trust the newly added microphone.

[0005] Attacks on speech recognition models utilize adversarial example techniques. Adversarial examples are generated from original speech, making subtle alterations that are indistinguishable to the user but cause the speech recognition model to identify them as other speech, thus protecting user privacy. However, existing general adversarial example generation methods are white-box based, requiring knowledge of the target model's internal structure to generate adversarial examples. Existing black-box based adversarial example generation methods can only generate non-general adversarial examples, requiring a separate adversarial example for each speech instance, which is clearly unacceptable in reality. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for protecting user voice privacy, which addresses the shortcomings of the prior art and solves the technical problem of privacy insecurity caused by the acquisition of user voice information by the cloud.

[0007] The present invention adopts the following technical solution:

[0008] A method for protecting user voice privacy includes the following steps:

[0009] S1. Collect wake word speech and process it into a wake word speech database in the corresponding format, and at the same time prepare a general speech database;

[0010] S2. Generate an initial adversarial perturbation audio, preprocess it and input it into the reinforcement learning model. Use the wake word speech database and general speech database obtained in step S1 to train the reinforcement learning model. Perform inverse short-time Fourier transform on the spectrum of the trained adversarial perturbation audio to obtain a black-box, general and asynchronous adversarial perturbation audio.

[0011] S3. Initialize the voice assistant device, run the Raspberry Pi deployed on the voice assistant device, play the anti-disturbance audio obtained in step S2 for the corresponding voice assistant, and detect the activation status of the voice assistant device. Control the playback and pause of the anti-disturbance audio according to the activation status to achieve user voice privacy protection.

[0012] Specifically, step S2 is as follows:

[0013] S201. Randomly generate an initial perturbation audio δ of length 1s, and perform a short-time Fourier transform on the initial perturbation audio δ. Prune the spectrum of the obtained initial perturbation audio δ, retaining the portion of the spectrum less than 4000Hz as the initial state S of the environment env in the reinforcement learning model. TF ;

[0014] S202, The initial state S obtained in step S201 is... TF Divide the data into multiple small square blocks, then input them into the agent, allowing the agent to process the initial state S. TF After modification, the spectrum S′ is obtained. TF ;

[0015] S203. Use the O-value completion step S202 to obtain the spectrum S′ TFThe high-frequency information is obtained and inverse short-time Fourier transform is performed to obtain the perturbation audio δ′. The perturbation audio δ′ and the initial perturbation audio δ are respectively input into the reward calculation module. The reward calculation module obtains reward values ​​R1 and R2 based on the performance of δ and δ′ in the wake word speech database and the general speech database obtained in step S1. The reward difference is obtained by subtracting R1 and R2, which is the reward r of the environment env to the agent's behavior. pseudo ;

[0016] S204. When the reward value R2 is greater than the set threshold, δ′ will be used as the generated counter-perturbation output.

[0017] S205. When the reward value R2 is less than the set threshold, use the interaction data S between the agent and the environment env. TF S′ TF r pseudo Train the agent and update the agent's algorithm and parameters;

[0018] S206. Randomly generate another perturbation audio δ with a length of 1 second. new According to 1-r pseudo With r pseudo The probability from δ new Choose one of δ' as the perturbation audio δ for the next round and return to step S201.

[0019] Furthermore, in step S202, the agent includes a performer network and an evaluator network. The performer network is used to generate adversarial perturbations, and the evaluator network is used to evaluate the quality of the adversarial perturbations. Both the performer network and the evaluator network are composed of Transformer networks. The performer network and the evaluator network share the position encoder and the encoder part of the Transformer network. The decoder part of the performer network is used to generate adversarial perturbations, and the decoder part of the evaluator network is used to evaluate the quality of the adversarial perturbations.

[0020] Furthermore, the process of constructing an intelligent agent is as follows:

[0021] S2021. The network input is position-encoded according to its original position before segmentation. The position-encoded data is then flattened and used as the input for the next step.

[0022] S2022. Randomly select a portion of the position-encoded input data as the data to be modified by the performer network;

[0023] S2023. The position-encoded data obtained in step S2021 and the data to be modified by the performer network obtained in step S2022 are used as the inputs to the encoder and decoder of the performer network, respectively, to obtain the output of the performer network, that is, the modification result of the data to be modified by the performer network in step S2022. The output is used to replace the data at the corresponding position in the original data to obtain new data.

[0024] S2024. The output obtained in S2023 is subjected to the same position encoding as in S2021. The position-encoded data is then flattened again and used as the input for the next step.

[0025] S2025. The position-encoded data obtained in S2021 and the position-encoded data obtained in S2024 are used as the inputs to the encoder and decoder of the evaluator network, respectively. The output is then passed through a fully connected layer to obtain the output of the evaluator network, which is the evaluator network's evaluation of the performer network's output.

[0026] S2026. Compare the output of the evaluator network obtained in S2025 with the evaluation results actually obtained through the speech recognition model or interface to obtain the loss function of the evaluator network and the loss function of the performer network.

[0027] Furthermore, the loss function l of the evaluator network critic for:

[0028]

[0029] Where score is the output of the evaluator network, r pseudo The evaluation result is obtained through a speech recognition model or interface, and smooth_loss is the smoothing loss function.

[0030] Furthermore, the loss function l of the performer network actor They are respectively:

[0031] l actor = smooth_loss(score, 1)

[0032] Here, score is the output of the evaluator network, and smooth_loss is the smoothing loss function.

[0033] Furthermore, step S203 specifically includes:

[0034] S2031. Select N data points from both the general speech database A and the wake-word speech database W, where N is a positive integer, and generate N sets of data a. i w i ;

[0035] S2032. Repeat the generated adversarial perturbation L times. Randomly select a moment from the previous second of the adversarial perturbation after repeating L times and start adding a. i until a i End, new data a′ obtained i ;

[0036] S2033, Regarding wake word speech data w i Perform step S2031 to obtain new wake word speech data w′ i ;

[0037] S2034, Take the data a′ obtained in step S2032 i Compared with the original data a i The data are input into the speech recognition model respectively, and the speech recognition model's response to the speech data a′ is obtained. i With a i The recognition result s′ i With s i Calculate s′ i Compared to s i Word error rate

[0038] S2035, Take the data w′ obtained in step S2033 i Inputting the wake word recognition model yields the wake word recognition model's response to the speech data w′. i Recognition results

[0039] S2036. Convert the N sets of outputs obtained in steps S2034 and S2035 into a single output. Sum the results separately and then divide by N to get the average word error rate r. S2T Compared with the average wake word recognition result F WW Multiplying them together yields the final reward r. pesudo .

[0040] Furthermore, in step S2032, the number of repetitions L is:

[0041] L = ceil(length(a i ))+1

[0042] The ceil function is a rounding function, and the length function is used to calculate the length of the speech data.

[0043] Specifically, step S3 is as follows:

[0044] S301, Device initialization;

[0045] S302: Read the anti-disturbance audio into memory, decode the anti-disturbance audio format into a unified standard format to be sent, and start sending the anti-disturbance.

[0046] S303, sleep for 0.1ms, read the resistance value of the photoresistor connected to the bus;

[0047] S304. If the resistance value of the photoresistor obtained in step S303 is less than the set threshold and is sending counter-interference, stop sending counter-interference and jump to step S2.

[0048] S305. If the resistance value of the photoresistor obtained in step S303 is greater than or equal to the set threshold, and no counter-perturbation is sent, start sending counter-perturbation and jump to step S2.

[0049] Secondly, embodiments of the present invention provide a user voice privacy protection system, characterized in that it includes:

[0050] The data module collects wake word speech and processes it into a wake word speech database in the appropriate format, while also preparing a general speech database;

[0051] The generation module generates an initial adversarial perturbation audio, which is preprocessed and then input into the reinforcement learning model. The reinforcement learning model is trained using the wake word speech database and general speech database obtained from the data module. The spectrum of the trained adversarial perturbation audio is subjected to inverse short-time Fourier transform to obtain a black-box, general and asynchronous adversarial perturbation audio.

[0052] The protection module initializes the voice assistant device, runs the Raspberry Pi deployed on the voice assistant device, plays the anti-disturbance audio generated by the corresponding voice assistant playback module, detects the activation status of the voice assistant device, and controls the playback and pause of the anti-disturbance audio according to the activation status, thereby realizing the protection of user voice privacy.

[0053] Compared with the prior art, the present invention has at least the following beneficial effects:

[0054] A method for protecting user voice privacy generates universal, synchronization-free adversarial examples based on a black-box model, eliminating the need to generate adversarial examples for each piece of voice data. This universal adversarial example is offensive against any voice data and is asynchronous, meaning it does not require the voice data to be aligned or synchronized with the adversarial perturbation. Instead, it allows users to speak at any time, and even if the voice data is out of sync with the adversarial perturbation, it can still produce an attack effect. The generated adversarial example is based on a black-box model, meaning it does not need to know the internal structure and parameter settings of the voice recognition system, only its input and output. For commercial voice assistants, their internal structure and parameter settings are often unknown; therefore, the adversarial examples generated by the approach or method provided in this invention can be used for attacks. The generated adversarial example can simultaneously achieve two goals: it allows the voice assistant's wake word recognition module to function normally, without affecting the user experience, while simultaneously disabling the voice assistant's voice recognition module, thereby protecting user privacy.

[0055] Furthermore, the generated adversarial examples can effectively avoid the problem of loss of high-frequency speech information and subsequent performance degradation caused by different choices of speech compression algorithms by the voice assistant recording device.

[0056] Furthermore, to generate the required adversarial examples more efficiently, this invention employs a widely validated performer-evaluator reinforcement learning structure and a Transformer network.

[0057] Furthermore, this invention efficiently improves the performance of the generated adversarial examples. It performs block segmentation and positional encoding on the network input, enabling the network to comprehensively consider both the overall distribution of the input data and the local distribution of the data to be improved, thereby enhancing the overall performance of the network output.

[0058] Furthermore, the loss function of the evaluator network can accurately evaluate the degree of utility improvement of the improved adversarial examples on actual speech recognition models.

[0059] Furthermore, the loss function of the performer network can continuously improve the performer network's ability to improve adversarial examples, ultimately resulting in high-performance adversarial examples.

[0060] Furthermore, testing on real-world speech databases can improve the generalization ability of the adversarial examples generated by the method of this invention in practical use.

[0061] Furthermore, the generated adversarial perturbations are made to start at random time points and completely cover the target speech, so that they can perform better in real-world use environments.

[0062] Furthermore, the hardware designed using this invention can precisely and promptly control playback and stop to counteract disturbances, thus providing more comprehensive protection for user privacy.

[0063] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0064] In summary, this invention is low-cost, easy to deploy, and can protect user privacy without affecting the use of the voice assistant.

[0065] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the workflow of the present invention;

[0067] Figure 2 This invention generates a schematic diagram of resistance to disturbances;

[0068] Figure 3 This is a schematic diagram of the neural network structure designed in this invention;

[0069] Figure 4 This is a schematic diagram of the return calculation method designed in this invention;

[0070] Figure 5 This is a schematic diagram of the hardware implementation of the present invention.

[0071] Figure 6 This is a schematic diagram illustrating the specific effects of the present invention. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0074] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0075] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" relationship.

[0076] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0077] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0078] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0079] This invention provides a method for protecting user voice privacy. It uses reinforcement learning to generate a dual-objective, black-box, universal, and synchronously independent adversarial perturbation audio offline. The generated adversarial perturbation audio is automatically started / stopped to deceive the voice assistant (intelligent speech recognition model) without affecting its normal functionality (it does not affect the user's ability to wake up and interact with the voice assistant). The adversarial perturbation audio is generated offline by the proposed reinforcement learning-based black-box adversarial attack model. This invention uses a Raspberry Pi for system prototyping and has conducted related experiments on commercial voice assistants. It is low-cost, easy to deploy, and effectively protects user voice privacy with minimal impact on user experience.

[0080] This invention discloses a method for protecting user voice privacy, comprising the following steps:

[0081] S1. Collect wake word voices for specific voice assistants and process them into a wake word voice database in the corresponding format. At the same time, prepare a general voice database and save the wake word voice database and the general voice database on a general computer for use in step S2.

[0082] S2. Generate an initial adversarial perturbation audio, preprocess it, and input it into the designed reinforcement learning model. Use the wake word speech database and general speech database obtained in step S1 to train the reinforcement learning model. During the training process, this reinforcement learning model will continuously improve the spectrum of the initial adversarial perturbation audio. After a certain number of rounds, perform inverse short-time Fourier transform on the spectrum of the obtained adversarial perturbation audio to obtain a black-box, general and asynchronous adversarial perturbation audio.

[0083] Please see Figure 2 The anti-perturbation audio is generated as follows:

[0084] S201. Randomly generate an initial perturbation audio δ of length 1s, and perform a short-time Fourier transform on it to obtain its spectrum. Then, prune the spectrum, retaining only the portion less than 4000Hz as the initial state S of the environment env in the reinforcement learning model. TF ;

[0085] Considering that, on the one hand, the frequency of speech emitted by a natural person generally does not exceed 4000Hz, and on the other hand, if a voice assistant uses audio compression technology, it is highly likely that high-frequency information in the audio will be discarded, in order to maximize the effectiveness of this invention, the portion of the spectrum below 4000Hz is used as the input S. TF .

[0086] S202, The initial state S obtained in step S201 is... TFThe data is divided into multiple small square blocks, which are then input into the agent, allowing the agent to process the initial state S. TF After modification, the spectrum S is obtained. ′ TF ;

[0087] Among them, the environment (env) and the agent are unique concepts in the reinforcement learning method used in this invention. In reinforcement learning, the agent continuously interacts with the environment and receives feedback from the environment. Through these interactions, the agent continuously evolves, aiming to enable the agent to automatically take corresponding interactive behaviors to maximize the feedback from the environment.

[0088] Please see Figure 3 The agent consists of two parts: an actor network and a critic network. The actor generates adversarial perturbations, and the critic evaluates the quality of these perturbations. The actor's input and output are both sliced ​​spectrograms. The critic's input and output are the actor's input and output, with the output being its evaluation of the actor's output. Both networks are primarily composed of Transformer networks, sharing the position encoder and encoder portion of the Transformer network. However, their decoder portions differ: the actor's decoder generates the adversarial perturbations, while the critic's decoder evaluates their quality. The specific steps are as follows:

[0089] S2021. The network input is position-encoded according to its original position before segmentation. The position-encoded data is then flattened and used as the input for the next step.

[0090] S2022. Randomly select a portion of the position-encoded input data as the data to be modified by the Actor;

[0091] S2023. The position-encoded data obtained in step S2021 and the data to be modified by the Actor obtained in step S2022 are used as the inputs of the Actor's encoder and decoder, respectively, to obtain the Actor's output, which is the modification result of the data to be modified by the Actor in step S2022. The Actor's output is used to replace the data at the corresponding position in the original data to obtain new data.

[0092] S2024. The output obtained in S2023 is subjected to the same position encoding as in S2021. The position-encoded data is then flattened again and used as the input for the next step.

[0093] S2025. The position-encoded data obtained in S2021 and the position-encoded data obtained in S2024 are used as the inputs to the encoder and decoder of Critic, respectively. The output is then passed through a fully connected layer to obtain the output of Critic, that is, Critic's evaluation of Actor's output.

[0094] S2026. Compare the output of Critic obtained in S2025 with the actual evaluation results obtained through the speech recognition model or interface to obtain the loss function of Critic.

[0095] The loss function of Critic is as follows:

[0096]

[0097] Where score is the output of Critic, r pseudo The evaluation result is obtained through a speech recognition model or interface. `smooth_loss` is the smoothing loss function, i.e.:

[0098]

[0099] The loss function for the Actor is:

[0100] l actor = smooth_loss(score, 1)

[0101] Here, score is the output of Critic, and smooth_loss is the smoothing loss function, which is the same as the smoothing loss function of Critic.

[0102] S203. The spectrum obtained in step S202, S′ TF The high-frequency information is padded with zero values ​​and subjected to inverse short-time Fourier transform to obtain the perturbation audio δ'. The perturbation audio δ' and the initial perturbation audio δ are respectively input into the reward calculation module. The reward calculation module obtains two reward values ​​R1 and R2 based on the performance of δ and δ' in the wake word speech database and the general speech database obtained in step S1. The reward difference is obtained by subtracting R1 from R2, which is the reward r of the environment env for the action taken by the agent. pseudo ;

[0103] Please see Figure 4To ensure that the generated adversarial example audio does not significantly impact users' normal use of the voice assistant in practice, the reward calculation module is designed with two parts. One part measures whether the adversarial example audio affects the user's normal use of the voice assistant, i.e., the effectiveness of the adversarial perturbation when applied to the corresponding wake word. The other part measures whether the adversarial perturbation audio can protect user privacy, i.e., the effectiveness of the adversarial perturbation audio when applied to normal speech. Details are as follows:

[0104] S2031. Select N data points (where N is a positive integer) from both the general speech database A and the wake-up word speech database W, generating N sets of data. For each set of data a i w i All operations from step S2032 to step S2035 are performed:

[0105] S2032. Repeat the generated adversarial perturbation L times, where L = ceil(length(a i ))+1, the ceil function is the floor function, and the length function calculates the length of the speech data in seconds. Speech a is added starting from a random moment one second before the adversarial perturbation after L repetitions. i Until voice a i End, new voice data a′ obtained i ;

[0106] S2033, Regarding wake word speech data w i Perform the operation described in step S2031 to obtain new wake word speech data w′. i ;

[0107] S2034. Transfer the voice data a′ obtained in step S2032. i Compared with the original speech data a i The data are input into the speech recognition model respectively, and the speech recognition model's response to the speech data a′ is obtained. i With a i The recognition result s′ i With s i Calculate s′ i Compared to s i The word error rate, specifically, is calculated by s′. i With s i Edit distance, then divide by s i The length of the word is used to obtain the word error rate, and the obtained word error rate is recorded as follows:

[0108] S2035, Transfer the voice data w′ obtained in step S2033 to... iThe data is input into the wake word recognition model to obtain the wake word recognition model's response to the speech data w′. i The recognition result is recorded as follows: if the wake-word recognition model can recognize the wake-word at this time, the recognition result is recorded as 1; otherwise, the recognition result is recorded as 0.

[0109] S2036. Obtain the N sets of outputs. Sum the results separately and then divide by N to get the average word error rate r. S2T Compared with the average wake word recognition result r WW Then multiply the two to get the final reward r. pesudo That is, r pesudo =r S2T ×r ww .

[0110] S204. If the reward value R2 is greater than the set threshold, then δ′ is used as the generated adversarial perturbation output, and all steps are terminated.

[0111] S205. If the reward value R2 is less than the set threshold, use the interaction data S between the agent and the environment env. TF S′ TF r pseudo Train the agent and update the agent's algorithm and parameters;

[0112] S206. Randomly generate another perturbation audio δ with a length of 1 second. new According to 1-r pseudo With r pseudo The probability from δ new Choose one of δ′ as the perturbation audio δ for the next round and return to step S201;

[0113] S3. Initialize the voice assistant device, run the Raspberry Pi, play anti-disturbance audio for the corresponding voice assistant and detect the voice assistant's activation status. Control the playback and pause of the anti-disturbance audio according to its activation status to protect the user's voice privacy.

[0114] The necessary components are deployed on the voice assistant device and connected to the Raspberry Pi. These components include photoresistors, speakers, indicator lights, and a switching system.

[0115] S201. Initialize the control device, including initializing and checking the photoresistor connected to the device bus and the device working status indicator light, and initializing the audio transmission device.

[0116] S202. Check the storage location of the counter-disturbance audio, read the counter-disturbance audio into memory, decode the format of the counter-disturbance audio into a unified standard format to be sent, and start sending the counter-disturbance.

[0117] S203, sleep for 0.1ms, read the resistance value of the photoresistor connected to the bus;

[0118] S204. If the resistance value is less than a certain threshold and anti-interference is being transmitted, stop transmitting anti-interference and proceed to step S3.

[0119] S205. If the resistance value is not less than a certain threshold and no countermeasures are sent, then start sending countermeasures and jump to step S3.

[0120] S206, Jump to step S203.

[0121] Please see Figure 1 The workflow of a voice assistant device is as follows:

[0122] The voice assistant device begins playing anti-disturbance audio to protect user privacy. When the user begins interacting with the voice assistant, they first utter a wake-up word. The voice assistant is activated upon detecting the wake-up word, and its indicator light illuminates. The photoresistor on the voice assistant device detects changes in ambient light, indicating that the user has begun interacting with the device. At this point, the voice assistant stops playing the anti-disturbance audio. The user then interacts normally with the voice assistant, which performs corresponding operations based on the user's commands. When the interaction ends, the indicator light on the voice assistant turns off. The photoresistor on the voice assistant device then detects changes in ambient light, confirming that the user has finished interacting with the device. At this point, the voice assistant resumes playing the anti-disturbance audio to protect user privacy.

[0123] In another embodiment of the present invention, a user voice privacy protection system is provided. This system can be used to implement the above-mentioned user voice privacy protection method. Specifically, the user voice privacy protection system includes a data module, a generation module, and a protection module.

[0124] The data module collects wake word speech and processes it into a wake word speech database in the corresponding format, while also preparing a general speech database.

[0125] The generation module generates an initial adversarial perturbation audio, which is preprocessed and then input into the reinforcement learning model. The reinforcement learning model is trained using the wake word speech database and general speech database obtained from the data module. The spectrum of the trained adversarial perturbation audio is subjected to inverse short-time Fourier transform to obtain a black-box, general and asynchronous adversarial perturbation audio.

[0126] The protection module initializes the voice assistant device, runs the Raspberry Pi deployed on the voice assistant device, plays the anti-disturbance audio generated by the corresponding voice assistant playback module, detects the activation status of the voice assistant device, and controls the playback and pause of the anti-disturbance audio according to the activation status, thereby realizing the protection of user voice privacy.

[0127] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, the computer program including program instructions, and the processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a user voice privacy protection method, including:

[0128] The system collects wake word speech and processes it into a wake word speech database in the appropriate format, while also preparing a general speech database. An initial adversarial perturbation audio is generated, preprocessed, and input into a reinforcement learning model. The model is trained using the wake word speech database and the general speech database. An inverse short-time Fourier transform is performed on the spectrum of the trained adversarial perturbation audio to obtain a black-box, general, and asynchronous adversarial perturbation audio. The system initializes a voice assistant device, runs the Raspberry Pi deployed on the device, plays the adversarial perturbation audio for the corresponding voice assistant, and detects the activation state of the voice assistant device. Based on the activation state, the system controls the playback and pause of the adversarial perturbation audio to protect user voice privacy.

[0129] In another embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0130] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the user voice privacy protection method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0131] The system collects wake word speech and processes it into a wake word speech database in the appropriate format, while also preparing a general speech database. An initial adversarial perturbation audio is generated, preprocessed, and input into a reinforcement learning model. The model is trained using the wake word speech database and the general speech database. An inverse short-time Fourier transform is performed on the spectrum of the trained adversarial perturbation audio to obtain a black-box, general, and asynchronous adversarial perturbation audio. The system initializes a voice assistant device, runs the Raspberry Pi deployed on the device, plays the adversarial perturbation audio for the corresponding voice assistant, and detects the activation state of the voice assistant device. Based on the activation state, the system controls the playback and pause of the adversarial perturbation audio to protect user voice privacy.

[0132] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0133] Please see Figure 5 The hardware used in deploying this invention is very simple and can be implemented using low-cost logic circuits.

[0134] Figure 5 A prototype of the invention is shown, comprising a Raspberry Pi, a photoresistor (PT0603), and several earpieces (depending on the number of microphones in the target voice assistant). The Raspberry Pi is connected to the photoresistor to detect the state of the target voice assistant and to control the earpieces to transmit noise-resistant audio.

[0135] To maximize the effectiveness of adversarial audio, this invention uses a 3D printer to print a kit for securing headphones. The headphones are fixed to the surface of the target voice assistant by this kit and positioned as directly as possible towards the microphone of the target voice assistant.

[0136] In addition to the necessary hardware mentioned above, the prototype of this invention is also equipped with an LED bulb to indicate the system's operating status and a switch to control the entire system's on and off states.

[0137] Please see Figure 6 A prototype system based on this invention was implemented and deployed and tested on a commercial voice assistant. The figure shows the test results. To more comprehensively evaluate the actual effect of this invention, the success rate was tested on three different voice assistant devices (Amazon Echo, Google Nest, and Apple Siri) under three different voice conditions: the wake word corresponding to the voice assistant, sentences with more words, and short voice commands.

[0138] It should be noted that during the testing process of this invention, for wake words, the success rate refers to the wake-up rate when the user successfully wakes up the voice assistant by uttering the wake word while playing the generated adversarial audio using the prototype system deployed by this invention; for sentences with many words or short voice commands, the success rate refers to the voice assistant's error rate in recognizing the user's spoken sentences or commands while playing the adversarial audio using the prototype system deployed by this invention. To demonstrate the effectiveness of this invention in protecting user privacy, it is compared with related work NI-Occam.

[0139] The test results are shown in the figure. The average success rate of the wake-up word of this invention exceeds 90%; the sentence recognition error rate exceeds 96%; and for short voice commands, this invention can achieve success rates of 71.5%, 93.3%, and 96.7%, respectively. In contrast, the success rates of the NI-Occam system are 40.0%, 40.0%, and 60.0%, respectively. Compared to the NI-Occam system, this invention achieves a higher success rate under several test conditions. This indicates that this invention can better protect user privacy in practical use without significantly impacting the user experience.

[0140] In summary, this invention provides a method and system for protecting user voice privacy. It employs deep neural network technology to generate adversarial audio offline and uses a Raspberry Pi to control the playback of the perturbation audio. This invention does not rely on special, expensive equipment. Results show that this invention can be applied to various scenarios and effectively protects user privacy without affecting the user's normal use of the voice assistant.

[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0142] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0144] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0147] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0148] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A user voice privacy protection method, characterized by, Comprise the following steps: S1, collect the wake-up word voice and process it into a wake-up word voice database in a corresponding format, while preparing a general voice database; S2, generate an initial adversarial noise audio, input it into a pre-processing reinforcement learning model, use the wake-up word voice database and general voice database obtained in step S1 to train the reinforcement learning model, and perform inverse short-time Fourier transform on the frequency spectrum of the adversarial noise audio obtained by training to obtain a black-box-based, general, and non-synchronous adversarial noise audio, specifically: S201、Randomly generate an initial perturbation audio with a length of 1s , and the initial perturbation audio is subjected to short-time Fourier transform, and the obtained initial perturbation audio spectrum is subjected to clipping processing, and the part less than 4000Hz in the spectrum is retained as the initial state of the environment in the reinforcement learning model ; S202, The initial state obtained in step S201 Divide into multiple small square blocks, then input into the intelligent agent. In the middle, let the intelligent agent For the initial state The spectrum was obtained after modification. ; S203, using 0 value to complete the spectrum obtained in step S202 high frequency information, and performs inverse short-time Fourier transform to obtain disturbance audio , the disturbance audio is input into the reward calculation module respectively, and the reward calculation module obtains the reward value according to and the performance of the wake-up word voice database and the general voice database obtained in step S1 and , and , the reward difference value is obtained after the reward difference value is obtained after and are subtracted, that is, the reward of the environment to the action taken by the agent ;​ S204、when the return value greater than the set threshold, the as the generated adversarial perturbation output; S205, When the return value When the value is less than the set threshold, use an intelligent agent. With the environment Interactive data , , For intelligent agents Train and update the intelligent agent Algorithm and parameters; S206, randomly generate a disturbance audio of 1s length again , according to and probability, select one from and as the disturbance audio of next round , and return to step S201; S3, initialize the voice assistant device, run the Raspberry Pi deployed on the voice assistant device, play the adversarial noise audio obtained in step S2 to the corresponding voice assistant, and detect the activation state of the voice assistant device, control the playing and pausing of the adversarial noise audio according to the activation state, and realize user voice privacy protection. 2.The user voice privacy protection method of claim 1, wherein, In step S202, the agent The method comprises a performer network and an evaluator network, the performer network is used to generate an adversarial perturbation, and the evaluator network is used to evaluate the quality of the adversarial perturbation; the performer network and the evaluator network are both composed of a Transformer network, the performer network and the evaluator network share a position encoder and an encoder part of the Transformer network, a decoder part of the performer network is used to generate the adversarial perturbation, and a decoder part of the evaluator network is used to evaluate the quality of the adversarial perturbation. 3.The user voice privacy protection method of claim 2, wherein, Intelligent agent The construction process is as follows: S2021, encode the input of the network according to the position before segmentation to obtain position-encoded data, and reflatten the position-encoded data as the input of the next step; S2022, randomly select a part of the position-encoded data as the data to be modified by the performer network; S2023, use the position-encoded data obtained in step S2021 and the data to be modified by the performer network obtained in step S2022 as the input of the encoder and decoder of the performer network respectively, obtain the output of the performer network, that is, the modification result of the data to be modified by the performer network in step S2022, replace the data at the corresponding position in the original data with the modification result, and obtain new data; S2024, encode the output obtained in step S2023 in the same position as in step S2021 to obtain position-encoded data, and reflatten the position-encoded data as the input of the next step; S2025, use the position-encoded data obtained in step S2021 and the position-encoded data obtained in step S2024 as the input of the encoder and decoder of the evaluator network respectively, pass the output through a fully connected layer to obtain the output of the evaluator network, that is, the evaluation of the output of the performer network by the evaluator network; S2026, compare the output of the evaluator network obtained in step S2025 with the actual evaluation result obtained through a speech recognition model or interface to obtain the loss function of the evaluator network and the loss function of the performer network. 4.The user voice privacy protection method of claim 3, wherein, Loss function of the evaluator network is: wherein, the output of the evaluator network, r pseudo the evaluation result obtained by the speech recognition model or interface, is a smoothing loss function. 5.The user voice privacy protection method of claim 3, wherein, Loss function for performer network Respectively: wherein, is the output of the evaluator network, is the smoothing loss function. 6.The user voice privacy protection method of claim 1, wherein, Step S203 is specifically: S2031、selecting N data from the general voice database with the wake-up word voice database respectively, N being a positive integer, to generate N groups of data ; S2032, repeat the generated adversarial perturbation second, from the repetition second before the adversarial perturbation, randomly select a time to start joining , until end, get new data ; S2033, for the wake-up word voice data , the operation of step S2031 is performed to obtain new wake-up word voice data ; S2034, inputting the data obtained in step S2032 into the speech recognition model with the original data respectively, obtaining the recognition result of the speech recognition model on the speech data with with , calculating the word error rate of the recognition result relative to ;​​​ S2035、obtaining the data from step S2033 In the inputting of the wake-up word recognition model, the recognition result of the voice data by the wake-up word recognition model is obtained ;​ S2036, accumulating the N groups of outputs obtained in steps S2034 and S2035 Respectively accumulated, and divided by N to obtain the average word error rate Multiply the average wake-up word recognition result , to obtain the final return .

7. The user voice privacy protection method of claim 6, wherein, In step S2032, the number of repetitions is: wherein, The function is a ceiling function, The function calculates the length of the voice data. 8.The user voice privacy protection method of claim 1, wherein, Step S3 is specifically: S301, device initialization; S302, read the adversarial noise audio into the memory, decode the adversarial noise audio into a unified standard format to be sent, and start sending the adversarial noise; S303, sleep for 0.1ms, and read the resistance value of the photosensitive resistor connected to the bus; S304, if the resistance value of the photosensitive resistor obtained in step S303 is less than the set threshold value, and the adversarial noise is being sent, stop sending the adversarial noise, and jump to step S2; S305, if the resistance value of the photosensitive resistor obtained in step S303 is greater than or equal to the set threshold value, and the adversarial noise is not being sent, start sending the adversarial noise, and jump to step S2.

9. A user voice privacy protection system, characterized by, Comprise: The data module collects the wake-up word voice and processes the wake-up word voice database in a corresponding format, and prepares a general voice database; The generation module generates an initial adversarial noise audio, which is input into a reinforcement learning model after preprocessing, and uses the wake-up word voice database and the general voice database obtained by the data module to train the reinforcement learning model. The frequency spectrum of the adversarial noise audio obtained by training is subjected to inverse short-time Fourier transform to obtain a black-box-based, general and non-synchronous adversarial noise audio, specifically: An initial perturbation audio with a length of 1s is randomly generated , and the initial perturbation audio is subjected to short-time Fourier transform, and the spectrum of the obtained initial perturbation audio is subjected to clipping processing, and the part less than 4000 Hz in the spectrum is retained as the initial state of the environment in the reinforcement learning model ; The initial state obtained is cut into multiple small square blocks, and then input the agent , let the agent modify the initial state After processing, the spectrum diagram is obtained ; The spectrum obtained by padding with 0 values The high-frequency information is obtained and inverse short-time Fourier transform is performed to obtain the perturbed audio. Disturbing audio Compared with the initial perturbation audio Input the information into the reward calculation module, and the reward calculation module will calculate the reward based on the input. and The reward value is obtained based on the performance in the obtained wake word speech database and general speech database. and ,Will and The difference between the two is the difference in returns, i.e., the environment. For intelligent agents Rewards for the actions taken ; When the return value is greater than the set threshold, the generated adversarial perturbation output; When the return value When the value is less than the set threshold, use an intelligent agent. With the environment Interactive data , , For intelligent agents Train and update the intelligent agent Algorithm and parameters; Again, randomly generate a perturbation audio of length 1s , according to and select one from and as the next round of perturbation audio and return; The protection module initializes the voice assistant device, runs the Raspberry Pi deployed on the voice assistant device, plays the adversarial noise audio generated by the generation module for the corresponding voice assistant, and detects the activation state of the voice assistant device. According to the activation state, the playing and pausing of the adversarial noise audio are controlled to realize user voice privacy protection.

Images