Voice privacy protection method and device based on high-frequency sound signal mixing

Abstract

The invention discloses a voice privacy protection method and device based on high-frequency sound signal mixing, and the method comprises the steps: firstly collecting user voice data, and constructing a personalized user corpus; then a user customizes a scene demand, selects a text theme synthesized by speech, and carries out user corpus extension; then, the user selects a noise theme related to the interference from the selected text theme according to the specific living situation where the user is located, and voice segments are randomly extracted from a user corpus corresponding to the noise theme to generate interference noise; modulating the interference noise to a preset frequency band interval; and finally, when it is detected that the user is talking, the controller sends a modulated interference signal to an interference unit, and the interference unit injects the interference signal into a nearby microphone. The method has practicability and security, can effectively protect user privacy, and can resist attacks of various denoising technologies such as speech enhancement and blind source separation.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and relates to a voice privacy protection method and device, and specifically to a voice privacy protection method and device based on high-frequency sound signal mixing. Background Art

[0002] Voice assistants are widely used in smart speakers today. These assistants enable users to control smart speakers through voice commands without manual operation. With their convenience, voice assistants have gradually become an indispensable part of many people's daily lives. However, in order to ensure real-time response, voice assistants usually capture and record all input audio signals and stream them to the cloud for processing and response. Once the user's voice data is recorded by the voice assistant, the control and flow of the data are difficult to grasp and trace, which has caused serious privacy concerns among users. Many research reports point out that voice assistants have certain defects and face the risk of being abused for illegal wiretapping.

[0003] In existing voice privacy protection technologies, researchers usually use the inherent nonlinear characteristics of commercial microphones to achieve privacy protection by playing carefully designed ultrasonic interference signals near smart speakers. When the interference signal is received by the microphone of the smart speaker, it will be demodulated and produce audible interference noise, thereby effectively confusing unauthorized recordings and protecting user privacy. For example, Chinese patent document number CN202310653928, published on July 4, 2023, records a voice privacy protection method based on ultrasonic interference. The invention discloses a method, system and device for designing ultrasonic interference samples to prevent eavesdropping, and the method includes: preprocessing a normal human voice data set; obtaining an audio transfer function of a digital domain audio demodulated by an eavesdropping microphone after ultrasonic modulation; obtaining the playback length of the ultrasonic interference sample; establishing an objective function to optimize the interference sample training, and after the training is completed, obtaining the ultrasonic interference sample release time, ultrasonic interference sample optimized amplitude and normal human voice optimized amplitude; calculating a mixed audio based on the audio transfer function, ultrasonic interference sample release time, ultrasonic interference sample optimized amplitude and normal human voice optimized amplitude, and inputting the mixed audio into a speech recognition system after limiting processing; inputting the training set data into the speech recognition system after processing, calculating the average loss function, and performing gradient descent optimization; evaluating the data in the prediction set to complete the design of the ultrasonic interference sample.

[0004] Although the ultrasonic method combines effectiveness and user-friendliness, its practical application is constrained by the need for specific ultrasonic transmitters or arrays. Existing ultrasonic jamming schemes use ultrasonic signals for noise injection, which often rely on dedicated ultrasonic transmitters or arrays, and are relatively expensive. In addition, existing noise schemes have certain limitations. Gaussian white noise can be easily filtered out by attackers using existing noise reduction algorithms, cosine wave noise can hardly cover the entire speaking frequency band, and the actual interference effect of phoneme-level noise is relatively insufficient. In addition, the continuous playback of interference signals will not only lead to unnecessary energy consumption, but also affect the voice interaction between users and smart speakers, seriously reducing the user experience. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a voice privacy protection method and device based on high-frequency sound signal mixing based on the insensitivity of the human ear to near-ultrasonic signals and the inherent nonlinear defects of commercial microphones.

[0006] The technical solution adopted by the method of the present invention is: a voice privacy protection method based on high-frequency sound signal mixing, comprising the following steps:

[0007] Step 1: Collect user voice data and build a personalized user corpus;

[0008] Step 2: Users customize scenario requirements, select text topics for speech synthesis, and expand the user corpus;

[0009] Step 3: The user selects a noise topic related to the interference from the text topic selected in step 2 according to the specific life situation in which he / she lives, and randomly extracts speech segments from the user corpus corresponding to the noise topic to generate interference noise;

[0010] Step 4: Modulate the interference noise to a preset frequency band;

[0011] Step 5: When it is detected that the user is talking, the controller sends a modulated jammer signal to the jammer, which injects the jammer signal into a nearby microphone.

[0012] Preferably, the specific implementation process of step 2 is to use speech synthesis technology to expand the user's personal corpus, first collect audio samples from different speakers, and then add emotion category labels to the collected data.

[0013] Preferably, the text encoder and duration predictor of the VITS model are optimized to accept speaker ID and emotion category embeddings;

[0014] The optimization of the text encoder is to add two embedding layers for speaker ID and emotion category to the input layer of the text encoder, and the two embedding layers convert the discrete speaker ID and emotion category into continuous vector representation respectively, and the word embedding of the text will be spliced ​​with the embedding vector of the speaker ID and emotion category;

[0015] The duration predictor is optimized by increasing the number of layers and the number of nodes in each layer of the duration predictor according to the new input dimension, increasing the original number of layers from 2 to 4, and increasing the number of nodes in each layer from 256 to 512, to ensure that the model can effectively process the additional embedded information and enhance the expressiveness of the model.

[0016] Preferably, the specific implementation of step 3 includes the following steps:

[0017] Step 3.1: Use a random acceleration factor α to accelerate the randomly selected speech segment;

[0018] Step 3.2: Perform gap removal processing on the accelerated speech segment to remove speech gaps whose energy is continuously lower than a specific threshold;

[0019] Step 3.3: Superimpose the processed speech segments in the time domain to generate mixed speech interference noise.

[0020] Preferably, in step 3.1, the randomly selected speech segment is converted into the frequency domain using Fourier transform; in the frequency domain, each frequency component has its own phase evolution over time, and the duration of the signal is shortened by adjusting the phase information and the step size of the phase evolution; finally, an inverse Fourier transform is performed to convert the audio signal into the time domain, thereby obtaining accelerated audio that maintains the pitch.

[0021] Preferably, in step 3.2, the speech gaps whose energy is continuously below a specific threshold are removed by analyzing the sampling data in the accelerated audio stream frame by frame, calculating the short-time energy of the audio signal, and if a certain audio segment lasts longer than threshold A and the short-time energy of all the audio frames contained therein are lower than threshold B, then the audio segment will be marked as silent; all audio portions marked as silent are removed from the audio stream.

[0022] Preferably, in step 4, the interference noise is modulated to a high frequency band of 17 kHz to 21 kHz using an upper sideband modulation technique.

[0023] Preferably, in step 5, voice activity detection technology is used to detect voice activity in real time; the controller collects ambient audio signals in real time, divides the audio signals into frame signals of preset length, analyzes each frame independently, extracts energy distribution, spectrum tilt and peak features of each frame to distinguish between voice and non-voice features; if at least N consecutive frame signals are detected to have voice features, it is determined that user voice activity exists at this time; wherein N is a preset value. Preferably, in step 5, an automatic interference control strategy is used to perform interference control; the specific implementation includes the following steps:

[0024] Step 5.1: Perform voice activity detection in real time; when voice activity is detected, execute the following step 5.2; otherwise, execute the following step 5.3;

[0025] Step 5.2: Turn on jamming & wait for voice command. Through the phoneme-level wake-up word detection technology, when the controller recognizes the first phoneme in the predefined wake-up word phoneme sequence, the controller instructs the jammer to switch to the stop jamming state while continuing to monitor subsequent phonemes until the entire phoneme sequence is recognized; if the wake-up word phoneme sequence is not detected, the jammer returns to the jamming state;

[0026] The trained BiLSTM-CRF model is used for single phoneme recognition. During training, the features of the 10 frames before and after the current frame are combined with the features of the current frame as the features of the frame for training. After training, the wake-up word dataset is combined with a part of the original speech dataset to fine-tune the model.

[0027] Step 5.3: Determine whether interference is in progress; if so, stop interference; if not, do nothing; determine whether the voice assistant is in follow-up mode or non-follow-up mode; when the voice assistant is in non-follow-up mode, use voice activity detection and a preset silent period strategy to recover from interference; when the voice assistant is in follow-up mode, once the voice assistant responds to user instructions, the controller resumes monitoring voice activity and resumes interference when voice activity is detected again.

[0028] The technical solution adopted by the device of the present invention is: a voice privacy protection device based on high-frequency sound signal mixing, comprising:

[0029] one or more processors;

[0030] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the voice privacy protection method based on high-frequency sound signal mixing.

[0031] Compared with the prior art, the beneficial effects of the present invention include:

[0032] (1) The present invention uses speech synthesis technology to expand the user's personal corpus. The user does not need to access the Internet during the operation, thus eliminating the risk of user voice privacy leakage. In addition, the user can select the text topic of speech synthesis according to their specific situational needs, such as economics, culture, and daily life, to better disrupt eavesdroppers.

[0033] (2) The present invention performs gap removal processing on the speech segments used to constitute the noise, which not only effectively reduces the long silent gaps and reduces the possibility of eavesdroppers capturing clear speech, but also increases the effective components of the interference noise, thereby improving the overall interference effect.

[0034] (3) The present invention superimposes user voice segments in the time domain to generate mixed voice interference noise. Increasing the number of simultaneously superimposed voice segments can produce more complex noise signals, but it will also dilute the energy allocated to each segment while maintaining the same signal-to-noise ratio, thereby reducing the overall interference effect. At the same time, using too few superimposed signals may achieve a stronger interference effect, but it will also make the noise easier to understand and susceptible to the attacker's voice separation model, thereby possibly exposing the original voice.

[0035] (4) The present invention accelerates and removes gaps from randomly extracted speech segments before performing time domain superposition, which significantly improves the interference effectiveness of mixed speech noise.

[0036] (5) The present invention generates interference noise based on the user corpus, which can more effectively confuse unauthorized recordings and has stronger resistance to various denoising models such as speech enhancement and blind source separation. In addition, the present invention expands the user corpus and increases the diversity of noise through speech synthesis.

[0037] (6) The present invention eliminates the reliance on dedicated ultrasonic equipment: The present invention uses near-ultrasonic injection jamming noise of 17kHz to 21kHz, allowing any commercial speaker with wireless communication capabilities to be used as a jammer without the need for dedicated ultrasonic transmitters or arrays.

[0038] (7) The present invention controls the activation, suspension and resumption of interference by designing an automatic interference control strategy. While effectively preventing unauthorized recording, the present invention retains the normal voice interaction between the user and the smart speaker, thereby achieving automatic interference control. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0041] Figure 2 This is a flow chart of the automatic interference control strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] This invention is mainly based on the privacy security of mobile devices, considering the nonlinear effects of commercial microphones, and proposes a voice privacy protection method based on high-frequency sound signal mixing. This invention makes full use of the nonlinear effects of commercial microphones, introduces a mixed noise interference scheme with both strong interference and high robustness, and combines it with an automatic interference control strategy to effectively prevent unauthorized recording while retaining the normal voice interaction between users and smart devices.

[0044] Please see Figure 1 , this embodiment provides a voice privacy protection method based on high-frequency sound signal mixing, comprising the following steps:

[0045] Step 1: Collect user voice data and build a personalized user corpus;

[0046] In one embodiment, during the registration process, the user needs to record a small portion of the voice sample according to the prompts to build a personalized user corpus. In addition, the user can choose to provide additional voice recordings when time permits to further enrich the personal corpus.

[0047] Step 2: Users customize scenario requirements, select text topics for speech synthesis, and expand the user corpus;

[0048] In one implementation, the present embodiment generates interference noise based on the user's corpus, and the diversity of the generated noise is limited by the amount of voice data available to the user. The less the amount of voice data in the user's corpus, the more likely the noise is to be repeated, which may lead to the potential risk that the adversary can separate the original voice from the interfered recording. Therefore, in order to reduce this risk, the present embodiment uses speech synthesis technology to expand the user's personal corpus. After obtaining IRB approval, the present embodiment collected audio samples from different speakers, totaling about 40,000 sentences, with an average sentence length of 8 seconds. In the collected data, the present embodiment added emotional classification labels such as rhythm, pauses, and intonation. These labels help to generate more natural and expressive synthesized speech. The present embodiment modifies the text encoder, duration predictor, and flow layer of the VITS model so that it can accept speaker ID and emotion category embedding. The training process follows the standard procedures provided by VITS. In this way, the trained model is able to read the input text according to different base speakers and generate speech under different emotions. The user only needs to provide a small portion of the speech sample, and this embodiment can extract the unique timbre features of the speaker and apply these features to the synthesized speech when generating the speech.

[0049] The text encoder and duration predictor of the VITS model are optimized to accept speaker ID and emotion category embedding; this embodiment adds two embedding layers for speaker ID and emotion category to the input layer of the text encoder. These embedding layers convert discrete speaker IDs and emotion categories into continuous vector representations. The word embedding of the text will be spliced ​​with the embedding vectors of the speaker ID and emotion category. In this way, the input of the text encoder contains not only text information, but also information about the speaker's identity and emotional state. According to the new input dimension, this embodiment increases the number of layers and the number of nodes per layer of the duration predictor, increasing the original number of layers from 2 to 4, and increasing the number of nodes per layer from 256 to 512, to ensure that the model can effectively process this additional embedding information and enhance the model's expressiveness. During the training process, a diverse data set containing multiple speakers and emotions is used to ensure that the model can learn the impact of different speakers and emotions on the generated audio features.

[0050] It is worth noting that users do not need to access the Internet during the operation, thus eliminating the risk of user voice privacy leakage. In addition, users can choose the text topic of speech synthesis according to their specific situational needs, such as economy, culture and daily life, to better disrupt eavesdroppers.

[0051] Step 3: The user selects a noise topic related to the interference from the text topic selected in step 2 according to the specific life situation in which he / she lives, and randomly extracts speech segments from the user corpus corresponding to the noise topic to generate interference noise;

[0052] In one implementation, the specific implementation process is:

[0053] Step 3.1: This embodiment performs acceleration processing on randomly selected voice segments. This embodiment uses Fourier transform to convert the audio signal into the frequency domain. In the frequency domain, each frequency component has its own phase evolution over time. The phase change between adjacent frames reflects the speed of the signal frequency component. In order to accelerate the audio, this embodiment adjusts these phase information so that the transition between frames is faster. After reasonably adjusting the phase information, this embodiment adjusts the step size of the phase evolution to shorten the duration of the signal and not affect the pitch. Finally, this embodiment performs an inverse Fourier transform on the modified signal and converts the audio signal into the time domain, thereby obtaining accelerated audio that maintains the pitch. Through acceleration, more semantic information that is meaningless to the attacker can be generated in the same time period, thereby enhancing the noisy feeling of the interference noise and improving the interference effect. After experiments, this embodiment determines that the optimal range of the random acceleration factor α is [1.4, 1.8] and applies it to each voice segment.

[0054] Step 3.2: This embodiment generates interference noise based on the corpus registered by the user. However, humans usually pause naturally when speaking, which may leak part of the original speech. To this end, this embodiment performs gap removal processing on the speech segments used to constitute the noise, and removes speech gaps whose energy is continuously below a specific threshold. This embodiment analyzes the sampled data in the audio stream after acceleration processing frame by frame. For each audio frame, this embodiment calculates the short-time energy of the audio signal. If a certain audio segment lasts longer than 0.1s and the short-time energy of all the audio frames contained is lower than '-40dB', then the audio segment will be marked as silent. This embodiment removes all audio parts marked as silent from the audio stream. This process not only effectively prevents long silence intervals and reduces the possibility of eavesdroppers capturing clear speech, but also increases the effective components of interference noise, thereby improving the overall interference effect.

[0055] Step 3.3: This embodiment superimposes the above-processed user voice segments in the time domain to generate mixed voice interference noise. Increasing the number of voice segments superimposed simultaneously can produce more complex noise signals, but it will also dilute the energy allocated to each segment while maintaining the same signal-to-noise ratio, thereby reducing the overall interference effect. At the same time, using too few superimposed signals may achieve a stronger interference effect, but it will also make the noise easier to understand and susceptible to the attacker's voice separation model, which may expose the original voice. In order to balance the effectiveness of interference and the resistance to the voice separation model, this embodiment sets the number of voice segments used for superposition to 2.

[0056] Step 4: Modulate the interference noise to a preset frequency band;

[0057] Theoretically, the maximum output frequency of most commercial loudspeakers can reach 24kHz. However, in practical applications, the frequency response of these loudspeakers is significantly reduced above 21kHz. At the same time, the human ear has almost no perception of sounds above 17kHz. In order to maximize the effective speech component of the demodulated noise, it is necessary to ensure that the part between 50Hz and 4kHz of the noise before modulation is modulated into the frequency range where the microphone is relatively sensitive.

[0058] Therefore, in one implementation, this embodiment uses upper sideband modulation technology to modulate the interference noise to a near-ultrasonic frequency band of 17kHz to 21kHz. This embodiment uses the user's mobile device as a controller and a commercial speaker as a jammer, and deploys the solution on the controller side to control and manage the jammer. The user places the jammer near the smart device, and the controller transmits the interference signal generated by the modulation to the jammer via Bluetooth. The jammer injects interference signals into the surrounding microphones to prevent unauthorized eavesdropping and protect user privacy.

[0059] Step 5: When it is detected that the user is talking, the controller sends a modulated near-ultrasonic jamming signal to the jammer, and the jammer injects the jamming signal into the nearby hidden microphone.

[0060] Continuous jamming places a high power demand on both the controller and the jammer. In addition, injecting jammers when the user is not speaking may inadvertently trigger responses from smart devices, degrading the user experience.

[0061] In one implementation, this embodiment detects the user's voice activity in real time. The controller collects the ambient audio signal in real time. This embodiment divides the audio signal into frame signals of 20ms length and analyzes each frame independently. This embodiment extracts energy distribution, spectrum tilt and peak features for each frame to distinguish between speech and non-speech features. If at least 10 consecutive frame signals are detected to have speech features, it means that there is user voice activity at this time. When it is detected that the user is speaking, the controller sends a modulated near-ultrasonic interference signal to the jammer via Bluetooth. Then, the jammer injects the interference signal into a nearby hidden microphone.

[0062] Please see Figure 2 In one implementation, an automatic interference control strategy is used to perform interference control; the specific implementation includes the following steps:

[0063] Step 5.1: Perform voice activity detection in real time; when voice activity is detected, execute the following step 5.2; otherwise, execute the following step 5.3;

[0064] Step 5.2: Turn on interference & wait for voice commands. At the same time, in order to preserve the normal interaction between the user and the smart device, this embodiment continues to provide a strategy for pausing interference. The controller is usually located outside the effective interference radius of the jammer and close to the user, in an undisturbed state. Therefore, the controller continuously captures voice signals to monitor the user's intention. When it detects that the user intends to issue a wake-up command, it instructs the jammer to temporarily stop interfering. This embodiment implements a phoneme-level wake-up word detection technology for user voice. When the controller recognizes the first phoneme in the predefined wake-up word phoneme sequence (for example, " / h / " in "Hey Alexa"), the controller instructs the jammer to switch to the stop interference state while continuing to monitor subsequent phonemes until the entire phoneme sequence is recognized. If the phoneme sequence of the keyword is not detected, the jammer returns to the interference state. In order to identify a single phoneme, this embodiment uses a trained BiLSTM-CRF model for single phoneme recognition. During training, this embodiment does not directly use the mfcc features of each frame for training, but combines the features of the 10 frames before and after the current frame with the features of the current frame as the features of the frame for training. Moreover, after training, this embodiment will combine the wake-up word dataset with a part of the original speech dataset to fine-tune the model. This not only makes the model prefer specific phonemes of the wake-up word, but also prevents the model from classifying all phonemes as wake-up word phonemes, thus ensuring the effectiveness of interference with normal speech.

[0065] Step 5.3: No user voice activity is detected at this time, so no interference is required. Determine whether interference is in progress at this time; if so, stop interference; if not, do nothing. When the voice assistant returns to the inactive state, the controller will continue to monitor voice activity to decide whether to resume interference. The voice assistant has two interactive states: follow-up mode and non-follow-up mode. In follow-up mode, the voice assistant remains active after the initial interaction, waiting for further commands without the need for another wake-up word command. If no further interaction occurs within a predefined time period, the voice assistant will automatically switch to an inactive state. For this mode, this embodiment simulates the workflow of the voice assistant to determine when it ends its activity. Specifically, this embodiment uses voice activity detection and a preset silent period, and if there is no voice activity during the silent period, the voice assistant is considered to be inactive. In non-follow-up mode, the voice assistant responds to a single voice command and then returns to an inactive state immediately after the command execution is completed. In this mode, once the voice assistant begins to respond, the controller resumes monitoring voice activity and resumes interference when voice activity is detected again. In particular, this embodiment distinguishes the captured signal frames by comparing the Mel-Frequency Cepstral Coefficient (MFCC) features with the voice features of the user and the voice assistant, thereby determining the exact moment when the voice assistant starts to respond.

[0066] This embodiment also provides a voice privacy protection device based on high-frequency sound signal mixing, including:

[0067] one or more processors;

[0068] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the voice privacy protection method based on high-frequency sound signal mixing.

[0069] The present invention is further described below through specific experiments.

[0070] The present invention designs and implements experiments in the digital domain and the physical domain, respectively. In the digital domain experiment, the present invention tests a total of about 1,400 hours of audio data covering 11 different signal-to-noise ratios (-5 to 5); in the physical domain experiment, the present invention collects more than 100 hours of audio data for testing. The present invention uses three commercial automatic speech recognition models (Tencent, Google, iFlytek) and two open source ASR models (Wenet, Whisper) for evaluation. The results show that, whether in the digital domain or the physical domain environment, when the signal-to-noise ratio is 0, the speech recognition error rate caused by the mixed speech noise of the present invention exceeds 60%, which is 22% higher than Gaussian white noise; and when the signal-to-noise ratio is reduced to -4, the error rate further increases to more than 90%, which is 18% higher than Gaussian white noise. In addition, speech enhancement and blind source separation techniques have limited effects on reducing the speech recognition error rate caused by mixed speech noise, and the error rates only drop by less than 3% and 2%, respectively.

[0071] Through a large number of experimental verifications in the digital domain and the physical domain, the present invention significantly improves the security and user-friendliness of voice privacy protection. Compared with the prior art, the present invention has better voice privacy protection effect at the same signal-to-noise ratio, does not require special ultrasonic equipment, and retains the normal voice interaction between the user and the smart speaker while effectively interfering.

[0072] The present invention adopts a novel and robust mixed voice noise solution, and through an automatic interference control strategy, it achieves effective interference while maintaining normal voice interaction between the user and the smart speaker. In view of the exemption of the near-ultrasonic signal used in the present invention from the dependence on special equipment and the acoustic characteristics of the mixed voice noise that effectively confuses the original voice, the present invention has both practicality and security, and can not only effectively protect user privacy, but also resist attacks from various denoising technologies such as voice enhancement and blind source separation.

[0073] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0074] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A voice privacy protection method based on high-frequency sound signal mixing, characterized in that: The following steps are involved: Step 1: Collect user voice data and build a personalized user corpus; Step 2: Users customize scenario requirements, select text topics for speech synthesis, and expand the user corpus; Step 3: The user selects a noise topic related to the interference from the text topic selected in step 2 according to the specific life situation in which he / she lives, and randomly extracts speech segments from the user corpus corresponding to the noise topic to generate interference noise; Step 4: Modulate the interference noise to a preset frequency band; Step 5: When it is detected that the user is talking, the controller sends a modulated jammer signal to the jammer, which injects the jammer signal into a nearby microphone.

2. The voice privacy protection method based on high-frequency sound signal mixing according to claim 1 is characterized in that: The specific implementation process of step 2 is: using speech synthesis technology to expand the user's personal corpus, first collecting audio samples from different speakers, and then adding emotion category labels to the collected data.

3. The voice privacy protection method based on high-frequency sound signal mixing according to claim 2 is characterized in that: By optimizing the text encoder and duration predictor of the VITS model to accept speaker ID and emotion category embeddings; The optimization of the text encoder is to add two embedding layers for speaker ID and emotion category to the input layer of the text encoder, and the two embedding layers convert the discrete speaker ID and emotion category into continuous vector representation respectively, and the word embedding of the text will be spliced ​​with the embedding vector of the speaker ID and emotion category; The duration predictor is optimized by increasing the number of layers and the number of nodes in each layer of the duration predictor according to the new input dimension, increasing the original number of layers from 2 to 4, and increasing the number of nodes in each layer from 256 to 512, to ensure that the model can effectively process the additional embedded information and enhance the expressiveness of the model.

4. The voice privacy protection method based on high-frequency sound signal mixing according to claim 1 is characterized in that: The specific implementation of step 3 includes the following steps: Step 3.1: Use a random acceleration factor α to accelerate the randomly selected speech segment; Step 3.2: Perform gap removal processing on the accelerated speech segment to remove speech gaps whose energy is continuously lower than a specific threshold; Step 3.3: Superimpose the processed speech segments in the time domain to generate mixed speech interference noise.

5. The voice privacy protection method based on high-frequency sound signal mixing according to claim 4 is characterized in that: In step 3.1, the randomly selected speech segment is converted into the frequency domain using Fourier transform. In the frequency domain, each frequency component has its own phase evolution over time. By adjusting the phase information and the step size of the phase evolution, the duration of the signal is shortened. Finally, an inverse Fourier transform is performed to convert the audio signal into the time domain, thereby obtaining accelerated audio that maintains the pitch.

6. The voice privacy protection method based on high-frequency sound signal mixing according to claim 4 is characterized in that: In step 3.2, the speech gaps whose energy is continuously below a specific threshold are removed by analyzing the sampling data in the accelerated audio stream frame by frame, calculating the short-time energy of the audio signal, and if a certain audio segment lasts longer than threshold A and the short-time energy of all the audio frames it contains is lower than threshold B, then the audio segment will be marked as silent; all audio parts marked as silent are removed from the audio stream.

7. The voice privacy protection method based on high-frequency sound signal mixing according to claim 1 is characterized in that: In step 4, the interference noise is modulated to a high frequency band of 17 kHz to 21 kHz using an upper sideband modulation technique.

8. The voice privacy protection method based on high-frequency sound signal mixing according to claim 1 is characterized in that: In step 5, voice activity detection technology is used to perform voice activity detection in real time; the controller collects environmental audio signals in real time, divides the audio signals into frame signals of preset length, analyzes each frame independently, and extracts energy distribution, spectrum tilt and peak features of each frame to distinguish between speech and non-speech features; If at least N consecutive frame signals are detected to have voice features, it is determined that user voice activity exists at this time; wherein N is a preset value.

9. The voice privacy protection method based on high-frequency sound signal mixing according to any one of claims 1 to 8, characterized in that: In step 5, an automatic interference control strategy is used to perform interference control; the specific implementation includes the following steps: Step 5.1: Perform voice activity detection in real time; when voice activity is detected, execute the following step 5.2; otherwise, execute the following step 5.3; Step 5.2: Turn on jamming & wait for voice command. Through the phoneme-level wake-up word detection technology, when the controller recognizes the first phoneme in the predefined wake-up word phoneme sequence, the controller instructs the jammer to switch to the stop jamming state while continuing to monitor subsequent phonemes until the entire phoneme sequence is recognized; if the wake-up word phoneme sequence is not detected, the jammer returns to the jamming state; The trained BiLSTM-CRF model is used for single phoneme recognition. During training, the features of the 10 frames before and after the current frame are combined with the features of the current frame as the features of the frame for training. After training, the wake-up word dataset is combined with a part of the original speech dataset to fine-tune the model. Step 5.3: Determine whether interference is in progress; if so, stop interference; if not, do nothing; determine whether the voice assistant is in follow-up mode or non-follow-up mode; when the voice assistant is in non-follow-up mode, use voice activity detection and a preset silent period strategy to recover from interference; when the voice assistant is in follow-up mode, once the voice assistant responds to user instructions, the controller resumes monitoring voice activity and resumes interference when voice activity is detected again.

10. A voice privacy protection device based on high-frequency sound signal mixing, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the voice privacy protection method based on high-frequency acoustic signal mixing as described in any one of claims 1 to 9.

Images