A method for designing speech interference noise based on human speech structure

Abstract

This invention discloses a method for designing speech interference noise based on human speech structure, comprising the following steps: (1) acquiring a large amount of speech data containing different speakers and different speech content, extracting voiceprint information, and constructing an initial speech dataset; (2) for each user, acquiring a small amount of user speech data, extracting voiceprint information, and matching the closest speech data in the initial speech dataset; (3) performing data augmentation on the matched speech data; (4) segmenting the augmented speech data using a phoneme segmentation algorithm to form a vowel dataset and a consonant dataset; (5) constructing three noise sequences based on the vowel dataset and the consonant dataset, superimposing them to obtain interference noise; (6) continuously generating random interference noise and playing it, injecting continuous interference noise into the recording to achieve continuous interference. Using this invention, interference noise can be made impossible to remove from speech, thereby avoiding the leakage of user privacy information.

Description

Technical Field

[0001] This invention belongs to the field of voice privacy protection, and in particular relates to a method for designing voice interference noise based on the structure of human voice. Background Technology

[0002] With the development of science and technology, devices with recording functions, such as mobile phones, smart TVs, and smart speakers, are becoming increasingly common in our lives. Due to the black-box nature of these smart devices, users cannot fully understand the internal programs running on them, posing a significant threat to user privacy. Attackers can control these devices to eavesdrop on users' voice information in their environment, and then use the rapidly developing deep learning-based speech recognition systems to identify the content of this voice information, thereby stealing users' privacy.

[0003] Therefore, effectively preventing eavesdropping has become a popular research direction. Some existing anti-eavesdropping products, such as Project Alias ​​and Paranoid Home Wave, can prevent microphones from recording by injecting white noise into them. However, these require knowledge of the specific microphone locations and the configuration of a noise emitter for each microphone, which greatly limits the application scenarios of the devices. Furthermore, research has found that using white noise to interfere with recording is not a reliable solution. Existing denoising methods, such as the deep learning-based speech denoising algorithm proposed by Xiang Hao et al. in "FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement," can effectively remove white noise interference in audio. This means that using white noise for interference cannot effectively prevent the leakage of voice privacy.

[0004] In recent years, scholars have proposed ultrasonic-based audio jamming schemes. The basic principle is to inject noise based on the nonlinearity of the device's microphone, thereby interfering with eavesdropping devices without disturbing the user in the environment. Yuxin Chen et al., in "Wearable Microphone Jamming," designed a wearable wristband with multiple ultrasonic transmitters that continuously emit ultrasonic waves to interfere with recording devices in the environment. Lingkun Li et al., in "Patronus: Preventing Unauthorized Speech Recordings with Support for Selective Unscrambling," designed an ultrasonic transmitter that can send frequency-modulated noise based on a pre-generated key, interfering with unauthorized recording devices while allowing authorized recording devices to record.

[0005] While the aforementioned interference methods can effectively inject noise into eavesdropping devices, the noise they use is overly simplistic, such as white noise or frequency-modulated noise. Existing denoising algorithms, such as FullSubNet based on deep learning, spectral subtraction based on noise features, and filtering methods, can remove this noise from speech. Therefore, current recording interference methods are insufficient to effectively prevent attackers from stealing users' private information from noisy recordings and do not meet current security requirements. Summary of the Invention

[0006] To address the shortcomings of existing voice eavesdropping and interference schemes, this invention provides a voice interference noise design method based on the structure of human speech. The generated interference noise can efficiently interfere with speech at low energy and maintain strong robustness, making the interfered speech unrecognizable by the human auditory system or machine speech recognition system. At the same time, existing speech enhancement and noise removal algorithms cannot effectively remove interference noise from the original speech, thereby avoiding the leakage of user privacy information.

[0007] A method for designing speech interference noise based on human speech structure includes the following steps:

[0008] (1) Obtain a large amount of speech data containing different speakers and different speech content, extract voiceprint information for each speaker in the speech data, and construct an initial speech dataset;

[0009] (2) For each user, a small amount of user voice data is obtained, and voiceprint information is extracted from the user voice data; based on the extracted user voiceprint information, the voiceprint information matching algorithm is used to match the closest voice data in the initial voice dataset generated in step (1);

[0010] (3) Perform data augmentation on the voice data obtained from step (2);

[0011] (4) The augmented speech data is segmented at the phoneme level using a phoneme segmentation algorithm to form vowel datasets and consonant datasets respectively.

[0012] (5) Construct three noise sequences based on the vowel dataset and the consonant dataset, where two sequences are concatenations of vowel data and one sequence is concatenation of consonant data; superimpose the three noise sequences to obtain the interference noise;

[0013] (6) Continuously generate random interference noise and play it, continuously injecting the interference noise into the recording to achieve continuous interference, making it difficult for the recording to steal privacy information.

[0014] Preferably, in steps (1) and (2), a neural network is used to extract voiceprint information. The input of the neural network is a continuous time-domain speech signal, and the output is a vector representing the voiceprint information. The neural network is represented as e = f(x), where x is a speech signal with a length greater than 1.6 seconds, e is the output voiceprint information, and the dimension is 1×256.

[0015] Preferably, in step (2), the voiceprint information matching algorithm adopts a cosine distance-based matching algorithm, specifically as follows:

[0016] Assuming the current user's voiceprint information is e t The initial speech dataset contains e voiceprint information for each speaker. i Where i∈[1,N], and N is the number of speakers in the initial speech dataset; then the closest speaker j in the matched dataset needs to satisfy the following expression:

[0017]

[0018] Where d(x,y) is the cosine distance between the two vectors.

[0019] Preferably, in step (2), the length of the user's voice data is 8 to 15 seconds, which is used to accurately extract the user's voiceprint information.

[0020] Preferably, in step (3), the augmentation algorithm based on speech emotion features performs data augmentation, which is divided into five augmentation methods: speech rate modification, average fundamental frequency modification, fundamental frequency curve modification, energy modification, and time sequence modification.

[0021] When modifying the speech rate, the speech rate modification parameter is randomly sampled from the uniform distribution U(0.3,1.8), with a value greater than 1 indicating acceleration and a value less than 1 indicating deceleration;

[0022] When modifying the average fundamental frequency, the average fundamental frequency modification parameter is randomly sampled from the uniform distribution U(0.9,1.1). A value greater than 1 indicates an increase in the fundamental frequency, and a value less than 1 indicates a decrease in the fundamental frequency.

[0023] When modifying the fundamental frequency curve, the fundamental frequency curve modification parameter is randomly sampled from the uniform distribution U(0.7,1.3). A value greater than 1 indicates that the original fundamental frequency curve is stretched, and a value less than 1 indicates that the original fundamental frequency curve is compressed.

[0024] When performing energy modification, the energy modification parameter is randomly sampled from the uniform distribution U(0.5,2), and the original audio signal s(t) is multiplied by the energy modification parameter;

[0025] When modifying the time sequence, simply reverse the order of the audio within the time domain.

[0026] Preferably, in step (4), the phoneme segmentation algorithm is an alignment algorithm based on Prosodylab-Aligner, and the specific segmentation process is as follows:

[0027] First, an open-source dataset such as aidatatang_200zh is used to train a speaker-independent acoustic model based on a Gaussian mixture model. Based on this model, the model is fine-tuned using data from each speaker in the initial speech dataset constructed in step (1), ultimately generating a dedicated acoustic model for each speaker. During segmentation, the corresponding speaker's acoustic model is selected, and the audio and corresponding text are input. The model outputs the types of phonemes in the audio and their corresponding timestamps in sequence. Based on these timestamps, each phoneme in the audio can be segmented and classified into vowels and consonants, forming vowel and consonant datasets.

[0028] The specific process of step (5) is as follows:

[0029] Vowels were randomly selected from the vowel dataset and spliced ​​together. A Hamming window of 25ms was used to smooth the splice. The resulting sequence was then accelerated to 1.1 times the original speed, resulting in the first noise signal.

[0030] Next, vowels are randomly selected from the vowel dataset, and the speed of each vowel is modified to α times the original speed, where α is a random number obtained by random sampling from a uniform distribution U(0.3, 1.8). Each vowel is resampled. The vowels with modified speeds are concatenated, and blank intervals are inserted between the vowels. The interval length is a random sampling of a uniform distribution U(0.001, 0.1) seconds to obtain the second noise signal.

[0031] Then, consonants are randomly selected from the consonant dataset and spliced ​​together. The splice is smoothed using a Hamming window with a length of 25ms to obtain the third noise signal.

[0032] Finally, the three noise signals are directly superimposed to obtain the final interference noise.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. Compared with existing speech interference noise (such as white noise), the interference noise designed in this invention can achieve a stronger interference effect under the same energy.

[0035] 2. Compared with existing speech interference noise, the noise designed in this invention has stronger robustness and is more difficult to be removed by existing denoising algorithms.

[0036] 3. Compared with existing targetless speech interference noise, the noise designed in this invention is generated for each user, which is more targeted and has a better interference effect.

[0037] 4. Based on the diversity of the corpus and the speech data augmentation algorithm proposed in this invention, the interference noise repetition rate designed in this invention is very low, with greater diversity and stronger interference universality. Attached Figure Description

[0038] Figure 1 This is an overall flowchart of a speech interference noise design method based on human speech structure according to an embodiment of the present invention;

[0039] Figure 2 This is a design block diagram for generating interference noise using vowel and consonant datasets in an embodiment of the present invention;

[0040] Figure 3 The error rate of speech containing perturbation noise under different speech recognition models;

[0041] Figure 4 The error rate of speech recognition under different speech recognition models is calculated by comparing speech containing perturbation noise before and after processing with a speech denoising algorithm. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0043] Currently, the issue of voice privacy leaks has received widespread attention. Attackers can control widely distributed smart devices to record and steal a target's voice privacy information. Existing voice jamming noise has drawbacks such as poor interference effect and low robustness, and cannot effectively protect user privacy.

[0044] Based on this, the present invention provides a method for designing speech interference noise based on the structure of human speech. The designed speech interference noise can interfere with recordings in real-world scenarios, making it difficult for attackers to extract the target's private information from the disturbed speech. Furthermore, the noise is robust, making it difficult for attackers to remove the interference noise from the recording using existing denoising algorithms.

[0045] like Figure 1 As shown, a speech interference noise design method based on human speech structure includes the following steps:

[0046] S1, Construct the speech dataset.

[0047] The speech dataset should be as rich as possible, broadly covering speakers of different ages, genders, accents, and emotions, and the content should be as diverse as possible. Public datasets such as LibriSpeech and GigaSpeech can be utilized. Deep learning methods should be used to compute the voiceprint information of each speaker in the obtained speech dataset.

[0048] S2, User Registration.

[0049] Record 10 seconds of the user's voice data using a device with recording function, such as a mobile phone, and extract the user's voiceprint information based on this data using the same method as in S1.

[0050] S3, acquisition of the closest voice data.

[0051] The speech data of the speaker most similar to the user's voiceprint information is obtained from the constructed speech dataset. The similarity is defined as the cosine distance between the voiceprint information; the larger the cosine distance, the lower the similarity, and vice versa.

[0052] The matching algorithm based on cosine distance is as follows: assuming the current user's voiceprint information is e t The voiceprint information of each speaker in the database is e i Where i∈[1,N], and N is the number of speakers in the speech dataset. The closest speaker j in the database must satisfy the following expression: Where d(x,y) is the cosine distance between the two vectors.

[0053] S4, Voice Augmentation.

[0054] Augmentation is performed on the matched speech data based on speech emotion features. Five augmentation methods are used: speech rate modification, average fundamental frequency modification, fundamental frequency curve modification, energy modification, and temporal order modification. Assuming the original audio signal is s(t), the specific augmentation algorithm is as follows:

[0055] The speech rate modification parameter α is randomly sampled from a uniform distribution U(0.3,1.8), with a value greater than 1 indicating acceleration and a value less than 1 indicating deceleration. There are two methods for speech rate modification. The first is to use the ffmpeg toolkit, which can directly modify the speech rate. The resulting speech s1(t) = ffmpeg.a_speed(s(t),α). Its advantage is good conversion quality, but its disadvantage is relatively slow speed. The other method is based on a phase vocoder. First, the speech is converted to a frequency domain signal, then the spectrum is interpolated frame by frame in the frequency domain, and finally converted back to the time domain. The resulting speech s1(t) = PhaseVocoder(s(t),α). Its advantage is fast conversion speed, but its disadvantage is poorer conversion quality.

[0056] The base frequency modification parameter α is randomly sampled from a uniform distribution U(0.9,1.1). A value greater than 1 increases the base frequency, while a value less than 1 decreases it. The modification method involves first using the aforementioned speech rate modification method to restore the speech rate to its original value. The result is s1(t) multiplied by 1, then the obtained audio is interpolated to the original value. times, get

[0057] The fundamental frequency curve modification parameter α is obtained by random sampling from a uniform distribution U(0.7,1.3). A value greater than 1 stretches the original fundamental frequency curve, while a value less than 1 compresses it. The specific operation method is to first extract the fundamental frequency curve f0 = world.harvest(s(t)) and the average fundamental frequency of the speech based on the world vocoder. Then modify the fundamental frequency curve of the audio. Then, based on the world vocoder, calculate the audio period parameter sp = world.cheaptrick(s(t), f0). ′ The aperiodic parameter ap = world.d4c(s(t),f′0). Finally, the modified speech s3(t) is synthesized based on the world vocoder: s3(t) = world.synthesize(f′0,sp,ap).

[0058] The energy modification parameter α is obtained by random sampling from a uniform distribution U(0.5,2). The modified speech is s4(t) = αs(t).

[0059] The time setting is changed to directly reverse the audio sequence in the time domain.

[0060] S5, phoneme segmentation.

[0061] Based on the Prosodylab-Aligner algorithm, the augmented speech data is segmented into individual vowels and consonants using the corresponding speaker's acoustic model, forming a set of vowels and a set of consonants.

[0062] S6, interference noise generation.

[0063] The noise generation algorithm continuously generates speech interference noise. For example... Figure 2 As shown, the specific process is as follows:

[0064] Vowels are randomly selected from the obtained vowel dataset and concatenated. A Hamming window of 25ms is used for smoothing at the concatenation point, and the resulting sequence is accelerated to 1.1 times its original speed, yielding the first noise signal. Next, vowels are randomly selected from the vowel dataset, and the speed of each vowel is modified by a factor of α, where α is a random number randomly sampled from a uniform distribution U(0.3, 1.8). Each vowel is resampled. The speed-modified vowels are then concatenated, and blank intervals are inserted between them, with an interval length of U(0.001, 0.1) seconds of random sampling, yielding the second noise signal. Consonants are then randomly selected from the consonant dataset and concatenated, with a Hamming window of 25ms used for smoothing at the concatenation point, yielding the third noise signal. Finally, the three noise signals are directly superimposed to obtain the final interference noise.

[0065] S7, noise transmission.

[0066] There are several options for noise transmission. You can use a regular speaker to transmit interfering noise, or you can use ultrasonic transmission to transmit interfering noise without disturbing other speakers in the environment. Users can choose according to their needs.

[0067] To verify the effectiveness of the present invention, experiments were conducted on the above-mentioned speech interference noise design method based on human speech structure.

[0068] Experiment 1 aims to verify the interference effect of the designed noise at different signal-to-noise ratios (SNRs) and compare it with traditional white noise interference. The noise and the original speech were mixed at different energy ratios (SNRs between -5 and 5). The noisy speech data was then used in a speech recognition model to calculate the word error rate (WER, a predictive error rate that measures the difference between the recognized result and the actual text; a higher value indicates a greater difference). Three speech recognition models (Amazon Speech Recognition, iFlytek Speech Recognition, and Google Speech Recognition) were tested in this experiment, and the results are as follows: Figure 3 As shown. In these three models, except for one case (using Amazon speech recognition with a signal-to-noise ratio of 5), the noise interference effect of the present invention on speech is better than the existing white noise interference method.

[0069] Experiment 2 verifies the robustness of the designed interference noise against existing denoising algorithms and compares it with the robustness of traditional white noise interference. The interference noise and the original speech are mixed at different energy ratios (signal-to-noise ratio between -5 and 5), and then existing speech denoising algorithms are used to process the noisy speech. Finally, the audio before and after processing is input into three speech recognition models for recognition, and the recognition results before and after denoising are compared. This experiment tested three speech recognition models (Tencent Speech Recognition, DeepSpeech Speech Recognition, and Wenet Speech Recognition), and the results are as follows: Figure 4 As shown in the figure. Among these three models, the interference noise designed in this invention exhibits stronger robustness compared to existing white noise interference, and the accuracy of speech recognition after being disturbed does not improve after being processed by the speech denoising algorithm.

[0070] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for designing speech interference noise based on human speech structure, characterized in that, Includes the following steps: (1) Obtain a large amount of speech data containing different speakers and different speech content, extract voiceprint information for each speaker in the speech data, and construct an initial speech dataset; (2) For each user, a small amount of user voice data is obtained, and voiceprint information is extracted from the user voice data; based on the extracted user voiceprint information, the voiceprint information matching algorithm is used to match the closest voice data in the initial voice dataset generated in step (1); (3) Perform data augmentation on the speech data obtained in step (2); specifically, perform data augmentation based on speech emotion features, which is divided into five augmentation methods: speech rate modification, average fundamental frequency modification, fundamental frequency curve modification, energy modification and time sequence modification. When modifying speech rate, the speech rate modification parameters are from a uniform distribution. The values ​​are obtained from random sampling; a value greater than 1 indicates acceleration, and a value less than 1 indicates deceleration. When performing average fundamental frequency modification, the average fundamental frequency modification parameters change from a uniform distribution. The values ​​are obtained by random sampling, and a value greater than 1 indicates an increase in the fundamental frequency, while a value less than 1 indicates a decrease in the fundamental frequency. When modifying the fundamental frequency curve, the fundamental frequency curve modification parameters are changed from a uniform distribution. The original fundamental frequency curve is obtained by random sampling. A value greater than 1 indicates that the original fundamental frequency curve is stretched, and a value less than 1 indicates that the original fundamental frequency curve is compressed. When performing energy modification, the energy modification parameters change from a uniform distribution. The original audio signal was obtained by random sampling. Multiply by the energy modification parameter; When modifying the time sequence, simply reverse the order of the audio in the time domain; (4) The augmented speech data is segmented at the phoneme level using a phoneme segmentation algorithm to form vowel datasets and consonant datasets respectively; (5) Construct three noise sequences based on the vowel and consonant datasets, where two sequences are concatenations of vowel data and one sequence is concatenation of consonant data; superimpose the three noise sequences to obtain the interference noise; the specific process is as follows: Vowels were randomly selected from the vowel dataset and spliced ​​together. A Hamming window of 25ms was used to smooth the splice. The resulting sequence was then accelerated to 1.1 times the original speed, resulting in the first noise signal. Then, randomly select vowels from the vowel dataset, and modify the velocity of each vowel to its original value. times, For each vowel, a random number is obtained by random sampling from a uniform distribution U(0.3, 1.8). The vowels after speed modification are concatenated and blank intervals are inserted between the vowels. The interval length is a random sampling of a uniform distribution U(0.001, 0.1) seconds to obtain the second noise signal. Then, consonants are randomly selected from the consonant dataset and spliced ​​together. The splice is smoothed using a Hamming window with a length of 25ms to obtain the third noise signal. Finally, the three noise signals are directly superimposed to obtain the final interference noise; (6) Continuously generate random interference noise and play it, continuously injecting the interference noise into the recording to achieve continuous interference and prevent the recording from being eavesdropped.

2. The speech interference noise design method based on human speech structure according to claim 1, characterized in that, In both steps (1) and (2), a neural network is used to extract voiceprint information. The input of the neural network is a continuous time-domain speech signal, and the output is a vector representing the voiceprint information. The neural network is represented as follows: , For speech signals longer than 1.6 seconds, The output voiceprint information has the following dimensions: .

3. The speech interference noise design method based on human speech structure according to claim 1, characterized in that, In step (2), the voiceprint information matching algorithm adopts a cosine distance-based matching algorithm, specifically as follows: Assuming the current user's voiceprint information is The initial speech dataset contains the voiceprint information of each speaker. ,in , The initial number of speakers in the speech dataset; then the most similar speaker in the matched dataset. The following expression must be satisfied: in, Let be the cosine distance between two vectors. .

4. The speech interference noise design method based on human speech structure according to claim 1, characterized in that, In step (2), the length of the user's voice data is 8 to 15 seconds.

5. The speech interference noise design method based on human speech structure according to claim 1, characterized in that, In step (4), the phoneme segmentation algorithm is an alignment algorithm based on Prosodylab-Aligner, and the specific segmentation process is as follows: First, an acoustic model based on a Gaussian mixture model and independent of the speaker is trained using an open-source dataset; the acoustic model is fine-tuned based on the data of each speaker in the initial speech dataset constructed in step (1), and finally a special acoustic model is generated for each speaker. During the segmentation process, the acoustic model of the corresponding speaker is first selected, the audio and the corresponding text are input, and the model outputs the types of phonemes in the audio and the corresponding timestamps in sequence. Based on this timestamp, each phoneme in the audio is segmented and classified into vowels and consonants according to its type, forming vowel datasets and consonant datasets.

Images