A Voiceprint Privacy Protection Method Based on Optimizing Codec Parameters

Abstract

The present invention discloses a voiceprint privacy protection method based on codec parameter optimization, belonging to the field of voice privacy protection. This method optimizes the internal characteristic parameters of the audio codec, introduces a formant conversion function, performs offset transformation, separation transformation, and bandwidth transformation on the formant features of the original audio signal, and generates a reconstructed audio according to the results of the transformation, so as to prevent malicious third parties from using the audio for privacy information mining (for example, voiceprint recognition). However, under the same set of audio codec parameters, the reconstructed audio can still be used for specific applications, such as speech content recognition. The method of the present invention embeds privacy protection in the voice generation process, does not require the introduction of additional hardware, only needs to introduce conversion function code, etc. in the source code of the codec, and the possibility of being maliciously tampered is greatly reduced; since voice coding is a necessary step for the storage and transmission of audio signals in the digital domain, this method is applicable to most voice usage scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of voice privacy protection, and relates to a voiceprint privacy protection method based on codec parameter optimization. Background Art

[0002] Speaker recognition and speech recognition technologies are widely used in intelligent voice devices, such as mobile phone voice assistants, smart speakers, smart homes, etc. Users activate the voice system with specific voice commands and issue subsequent commands for the device to perform related operations. Although the application of intelligent voice makes people's lives more convenient, the audio recorded by these devices contains user privacy information. Once a malicious third party obtains the audio with privacy information, it may lead to the leakage of the user's voiceprint, causing people to worry about the privacy security of the intelligent voice system.

[0003] Most existing methods need to transmit the voice signal with sensitive information from voice transmission devices such as microphones to the software layer for desensitization processing, but the original voice may be stolen at various links during the data transmission process. Therefore, how to more reliably protect voice privacy information is a technical problem that the industry urgently needs to solve. Summary of the Invention

[0004] To overcome the above problems, the present invention provides a voiceprint privacy protection method based on codec parameter optimization. This method utilizes the characteristic that the codec parameters in existing voice transmission devices are adjustable, and through multi-objective optimization, the voice content recorded after parameter adjustment is difficult to be recognized by privacy-sensitive algorithms (such as speaker recognition), but can still be recognized by some privacy-insensitive algorithms (such as speech content recognition), realizing the function of protecting the speaker's identity without affecting the normal application of the voice.

[0005] The present invention is implemented by adopting the following technical solutions:

[0006] A voiceprint privacy protection method based on codec parameter optimization, the method comprising the following steps:

[0007] Step 1: Define formant conversion functions. The formant features of a frame of original audio signal are respectively subjected to offset transformation, separation transformation, and bandwidth transformation through three formant conversion functions, defined as follows:

[0008] F1(ω i ) = ω i + ω i (ξ1 - 1)(π - ω i )π i = 1, …, p

[0009] F2(ω i ) = ω i +(ξ2 - 1)sin(2πωi ) / p i = 1, …, p

[0010]

[0011] where ω i is the i-th line spectral pair frequency parameter of the original audio signal, p is the LPC order of the CELP filter, δ(ω i ) is the line spectral pair spacing, ξ1, ξ2, ξ3 are conversion coefficients, and F1(.), F2(.), F3[.] are three formant conversion functions that respectively implement offset transformation, separation transformation, and bandwidth transformation functions;

[0012] Step 2: Generate multiple groups of conversion coefficients for population initialization of the genetic algorithm and store the conversion coefficients in a configuration file;

[0013] Step 3: Modify the CELP codec, insert the formant conversion function into the source code of the CELP encoder, and generate an executable file;

[0014] Step 4: Optimize each group of coefficients in the configuration file using the genetic algorithm, and the specific method is as follows:

[0015] Step 4.1: Use the executable file to encode and decode the speech dataset, respectively identify the encoded and decoded speech dataset using a speaker recognition model and a speech content recognition model, evaluate the model recognition results using the general metrics corresponding to each model, and obtain the normalized scores corresponding to each group of conversion coefficients;

[0016] Step 4.2: Sort the multiple groups of conversion coefficients according to the normalized scores, and determine whether the iteration times have been reached. If the iteration times have been reached, output the conversion coefficients stored in the current configuration file as the optimal conversion coefficients. Otherwise, continue to execute the following steps;

[0017] Step 4.3: Perform selection, crossover, and mutation operations on the population to obtain a new population as the population for the next iteration;

[0018] Step 4.4: Write the conversion coefficients corresponding to the new population into the configuration file and recompile the CELP codec to obtain a new executable file, and return to Step 4.1;

[0019] Step 5: Randomly select a group from the optimal conversion coefficients and set it in the CELP codec to obtain a speech codec with privacy protection function;

[0020] Step 6: Use the speech codec with privacy protection function to perform offset transformation, separation transformation, and bandwidth transformation on the formant features of the original audio signal, and generate a reconstructed audio according to the transformed results.

[0021] Furthermore, the line spectral pair spacing in step 1 is expressed as follows:

[0022] δ(ω i )=ω i+1 -ω i i = 0, …, p

[0023] Additionally, ω0 = 0, ω p+1 = 1.

[0024] Furthermore, the value ranges of the conversion coefficients ξ1, ξ2, and ξ3 in step 1 are [0, 2].

[0025] Furthermore, step 3 includes:

[0026] Step 3.1: Add the conversion function code source file, configuration file reading code source file, and header file to the CELP encoder source code; modify the compilation configuration file of the CELP codec;

[0027] Step 3.2: Compile the CELP encoder and decoder respectively according to the compilation configuration file to obtain executable files.

[0028] Furthermore, the genetic algorithm in step 4 adopts the NSGA-II multi-objective genetic algorithm, and represents the genotype of an individual as a set of three conversion coefficients:

[0029]

[0030] where P is the population size, g (k) represents the k-th individual, represents the k-th set of conversion coefficients.

[0031] Furthermore, in step 4, the calculation formula for the normalized score is as follows:

[0032]

[0033] where D is the speech dataset, D sub is a subset randomly selected from D with N audio segments, x i is the i-th audio sample in the subset, s (k) (x i ) is the evaluation score of the i-th audio sample under the k-th set of conversion coefficients, and S (k) is the normalized score of the k-th set of conversion coefficients.

[0034] Furthermore, the calculation method of the evaluation score is:

[0035] s (k) (x i ) = k1s cos_dist(k) (x i ) + k2s WER (k) (x i )

[0036] where s cos_dist represents the cosine distance between the original speech calculated by the speaker recognition model and the speech processed by the codec, and s WER represents the word error rate output by the speech content recognition model, and k1 and k2 are fractional weights respectively.

[0037] Furthermore, the formant features of the original audio signal are subjected to offset transformation, separation transformation, and bandwidth transformation, and the result after transformation is expressed as:

[0038] ω′ i = F3[δ(F2(F1(ω i )))]

[0039] where ω′ i is the result of the transformation of the i-th line spectral pair frequency parameter of the formant feature of the original audio signal.

[0040] The beneficial effects of the present invention are:

[0041] (1) The present invention only needs to make minor modifications to the existing CELP codec, and then the formant conversion algorithm for hiding voiceprint information can be deployed in the CELP codec, without affecting the normal recognition of speech content at the same time. It is applicable to various speech application scenarios, such as video conferencing, Internet telephony, and smart home, etc. The application range is wide, and it can fundamentally curb the privacy infringement problem caused by the abuse of speaker recognition.

[0042] (2) Since the CELP codec is generally hardware-isolated from the downstream processors required to implement speech recognition, the security is strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of the voiceprint privacy protection method based on codec parameter optimization shown in the embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0044] In order to facilitate the understanding and implementation of this invention by those of ordinary skill in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0045] The method flow of the embodiments of the present invention, as Figure 1 shown, a voiceprint privacy protection method based on codec parameter optimization includes the following steps:

[0046] Step 1: Take the LPC order p = 10 and define the formant conversion function:

[0047] F1(ω i ) = ω i + ω i (ξ1 - 1)(π - ω i )π i = 1, …, 10

[0048] F2(ω i ) = ω i +(ξ2 - 1)sin(2πω i ) / 10 i = 1, …, 10

[0049]

[0050] where ω0 = 0, ω 11 = 1, and the line spectral pair spacing:

[0051] δ(ω i ) = ω i+1 - ω i i = 0, …, 10

[0052] Step 2: Initialize the population size p = 50 and generate the initial coefficients using a uniform distribution within the space of the conversion coefficients:

[0053]

[0054] Save the above 150 coefficients in the configuration file param.txt.

[0055] Step 3: Modify the G.729 encoder as an example of the CELP encoder, and the specific steps are as follows:

[0056] 3.1) According to the definition in Step 1, write the LSF conversion function Lsf_trans(), and insert the LSF conversion function Lsf_trans() between the Lsp_Lsf2() function and the Lsp_qua_cs() function in the G.729 encoder source code. The conversion coefficients are read from the configuration file param.txt;

[0057] 3.2) Modify the coder.mak and decoder.mak files, and compile the G.729 codec to obtain the executable files coder.hex and decoder.hex.

[0058] Step 4: Use the NSGA-II genetic algorithm to optimize the conversion parameters, and the specific steps are as follows:

[0059] 4.1) Use the coder.hex and decoder.hex executable files to encode and decode the Librispeech - test - clean speech dataset. Identify the encoded and decoded speech datasets using the ECAPA - TDNN speaker recognition model and the crdnn - rnn speech content recognition model respectively. Use the cosine distance between the original speech and the encoded / decoded speech as the privacy protection score, and use the word error rate WER as the speech usability score.

[0060] 4.2) Set the number of iterations to 5000 times. Sort the effectiveness of 50 groups of conversion coefficients according to the cosine distance and WER scores, and determine whether the number of iterations has been reached. If the number of iterations has been reached, output the conversion coefficients in the current configuration file param.txt; otherwise, continue with the following steps.

[0061] 4.3) Perform selection, crossover, and mutation operations on the population to obtain a new population as the population for the next iteration.

[0062] 4.4) Write the conversion coefficients corresponding to the new population into the configuration file param.txt and re - compile the G.729 codec to obtain new executable files coder.hex and decoder.hex, then return to step 4.1).

[0063] Step 5: Randomly select a group from the optimal conversion coefficients and set it in the G.729 codec to obtain a speech codec with privacy protection function.

[0064] Taking a microphone with a codec as an example, the working principle of the present invention for realizing privacy protection is as follows: After the codec in the microphone receives the original audio signal, it extracts the line - spectral pair frequency parameter ω i , and based on the formant conversion function defined in step 1, converts the line - spectral pair frequency parameter ω i to obtain a new line - spectral pair frequency parameter ω′ i = F3[δ(F2(F1(ω i )))]; Then further synthesize the reconstructed audio from the new line - spectral pair frequency parameter ω′ i . Since the conversion coefficients in the formant conversion function are the optimized results through the speaker recognition model and the speech content recognition model, the present invention can protect the voiceprint privacy without affecting the normal recognition of speech content, and is applicable to various speech application scenarios such as video conferencing, Internet telephony, and smart home, etc., with a wide range of applications.

[0065] It should be noted that the line - spectral pair frequency parameter ω is extracted from the original audio signal. iThe process is well-known in the art and will not be elaborated here. Similarly, the parts not described in detail in this specification all belong to the prior art.

[0066] It should be understood that the above description of the embodiments is relatively detailed, and it should not be considered as a limitation to the protection scope of the invention patent. Under the inspiration of the present invention, those of ordinary skill in the art can also make substitutions or modifications without departing from the scope protected by the claims of the present invention, and all fall within the protection scope of the present invention. The scope of protection claimed by the present invention should be subject to the appended claims.

Claims

1. A voiceprint privacy protection method based on codec parameter optimization, characterized in that The method includes the following steps: Step 1: Define formant conversion functions. The formant feature of a frame of original audio signal is respectively subjected to offset transformation, separation transformation and bandwidth transformation through three formant conversion functions, which are defined as follows: F1(ω i ) = ω i + ω i (ξ1 - 1)(π - ω i )πi = 1, …, p F2(ω i ) = ω i + (ξ2 - 1)sin(2πω i ) / pi = 1, …, p where ω i is the i-th line spectral pair frequency parameter of the original audio signal, p is the LPC order of the CELP filter, δ(ω i ) is the line spectral pair spacing, ξ1, ξ2, ξ3 are conversion coefficients, and F1(.), F2(.), F3[.] are three formant conversion functions that respectively implement offset transformation, separation transformation, and bandwidth transformation functions; Step 2: Generate multiple groups of conversion coefficients for the population initialization of the genetic algorithm, and store the conversion coefficients in a configuration file; Step 3: Modify the CELP codec, insert the formant conversion function into the source code of the CELP encoder, and generate an executable file; Step 4: Optimize each group of coefficients in the configuration file using the genetic algorithm, and the specific method is as follows: Step 4.1: Use the executable file to encode and decode the speech data set. The encoded and decoded speech data set is respectively identified using a speaker recognition model and a speech content recognition model, and the recognition results of the models are evaluated using the general metrics corresponding to each model to obtain the normalized score corresponding to each group of conversion coefficients; Step 4.2: Sort the multiple groups of conversion coefficients according to the normalized scores, and determine whether the iteration times are reached. If the iteration times are reached, output the conversion coefficients stored in the current configuration file as the optimal conversion coefficients, otherwise continue to execute the following steps; Step 4.3: Perform selection, crossover and mutation operations on the population to obtain a new population as the population for the next iteration; Step 4.4: Write the conversion coefficients corresponding to the new population into the configuration file and recompile the CELP codec to obtain a new executable file, and return to Step 4.1; Step 5: Randomly select a group from the optimal conversion coefficients and set it in the CELP codec to obtain a speech codec with privacy protection function; Step 6: Use the speech codec with privacy protection function to perform offset transformation, separation transformation and bandwidth transformation on the formant feature of the original audio signal, and generate a reconstructed audio according to the transformed result.

2. The voiceprint privacy protection method based on codec parameter optimization according to claim 1, wherein The line spectral pair spacing of the said Step 1 is expressed as follows: δ(ω i ) = ω i+1 -ω i i = 0, …, p Additionally, define ω0 = 0, ω p+1 = 1.

3. The voiceprint privacy protection method based on codec parameter optimization according to claim 1, characterized in that The value ranges of the conversion coefficients ξ1, ξ2, ξ3 in the said Step 1 are [0, 2].

4. The voiceprint privacy protection method based on codec parameter optimization according to claim 1, characterized in that The said Step 3 includes: Step 3.1: Add a conversion function source file, a configuration file reading source file and a header file to the CELP encoder source code; modify the compilation configuration file of the CELP codec; Step 3.2: Compile the CELP encoder and decoder respectively according to the compilation configuration file to obtain an executable file.

5. The voiceprint privacy protection method based on codec parameter optimization according to claim 1, characterized in that The genetic algorithm in the said Step 4 adopts the NSGA-II multi-objective genetic algorithm, and represents the genotype of an individual as a set of three conversion coefficients: where P is the population size, and g (k) represents the k-th individual, and represents the k-th set of conversion coefficients.

6. The voiceprint privacy protection method based on codec parameter optimization according to claim 1, characterized in that In the said Step 4, the calculation formula of the normalized score is as follows: Among them, D is the speech dataset, and D sub is a subset randomly selected from D with N audio segments, and x i is the i-th audio sample in the subset, and s (k) (x i ) is the evaluation score of the i-th audio sample under the k-th set of conversion coefficients, and S (k) is the normalized score of the k-th set of conversion coefficients.

7. The voiceprint privacy protection method based on codec parameter optimization according to claim 6, characterized in that The calculation method of the said evaluation score is: s (k) (x i ) = k1s cos_dist (k) (x i ) + k2s WER (k) (x i ) Among them, s cos_dist represents the cosine distance between the original speech calculated by the speaker recognition model and the speech processed by the codec, and s WER represents the word error rate output by the speech content recognition model, and k1 and k2 are score weights respectively.

8. The voiceprint privacy protection method based on codec parameter optimization according to claim 1, characterized in that The offset transformation, separation transformation and bandwidth transformation are performed on the formant feature of the original audio signal, and the transformed result is expressed as: ω i ′ = F3[δ(F2(F1(ω i )))] Among them, ω i ′ is the result after transformation of the i-th line spectrum pair frequency parameter of the formant feature of the original audio signal.

Images