A speaker privacy protection method and system capable of preserving voice emotion

Abstract

The present invention belongs to the technical field of intelligent digital signal processing, and discloses a speaker privacy protection method and system capable of retaining speech emotion, the method comprising: obtaining the speaker's original audio and inputting it into an end-to-end emotional speech fusion model; using the fusion model to infer the original audio, changing the voiceprint information in the original audio, and obtaining a pseudo speaker audio retaining the emotional state; the fusion model comprises a priori encoder and a waveform decoder, the priori encoder receives the original audio and the speaker's identity ID, calculates and uses the fundamental frequency coding, content coding, emotion coding and speaker ID coding of the original audio to generate a priori distribution mean and variance; the waveform decoder restores the priori distribution mean and variance into the pseudo speaker audio. Using the model fusion method, the unseen pseudo speaker audio is generated, and the "out-of-distribution" problem of the reasoning process is avoided, and the problem of mismatch between emotion retention and pseudo speaker information is solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent digital signal processing, and in particular to a speaker privacy protection method and system capable of retaining voice emotions. Background Art

[0002] As a medium rich in information, speech contains a large amount of personal (i.e., personally identifiable) information, such as the speaker's identity, age and gender, health status, personality, race or ethnicity, geographic background, and even social identity and economic status, which makes speech a unique biomarker. In many cases, speakers want to hide their identities, such as when sharing sensitive content or unwilling to disclose personal information. As a result, speech privacy protection becomes particularly important, and speech anonymization technology has emerged. One of the main methods of speech anonymization is voice conversion (VC). This technology converts the speaker's voice into the voice of another person without changing the language or intonation content. Modern deep neural network speech conversion models can convert the voice of any speaker into the voice of the target speaker.

[0003] The Chinese invention patent with publication number CN117153162A discloses a method and device for protecting voice privacy. The invention constructs an anonymous timbre library of the target domain; replaces the user's input voice with the corresponding voice of the target timbre through voice conversion technology, forms a replacement pronunciation feature and constructs a target anonymous timbre pronunciation library; and performs voice anonymization processing based on the replacement pronunciation feature. This approach can hide the identity information of the original speaker while roughly ensuring the integrity of the semantics. However, in order to convey the information of the original speech more accurately and richly, the emotions contained in the voice should also be retained, and this invention does not have this function. In fact, existing voice anonymization methods often find it difficult to balance the contradiction between privacy protection and emotional retention while maintaining emotional expression. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the prior art and to propose a speaker privacy protection method and system that can preserve voice emotions.

[0005] On the one hand, a speaker privacy protection method capable of preserving speech emotion is provided, comprising:

[0006] Obtain the speaker's original speech audio and input it into the end-to-end emotion speech fusion model;

[0007] The end-to-end emotional speech fusion model is used to infer the original speech audio, change the voiceprint information in the original speech audio, and obtain the pseudo speaker speech audio that retains the emotional state of the original speech audio;

[0008] The end-to-end emotion speech fusion model includes: a priori encoder and a waveform decoder, and the use of the end-to-end emotion speech fusion model to infer the original speech audio includes:

[0009] Step 1: The prior encoder receives the original speech audio and the speaker identification number ID, calculates and uses the fundamental frequency coding, content coding, emotion coding and speaker ID coding of the original speech audio to generate the prior distribution mean and variance, where the speaker ID coding contains the voiceprint information;

[0010] Step 2: The waveform decoder re-parameterizes the prior distribution mean and variance into latent variables, performs decoding operations on the latent variables, and upsamples the decoded latent variables to restore them to pseudo speaker speech audio in the time domain.

[0011] In an improved speaker privacy protection method capable of retaining speech emotion, the prior encoder comprises: a fundamental frequency extraction module, a speaker encoder, a content encoder, an emotion encoder and a self-attention encoder; the step 1 comprises:

[0012] The fundamental frequency extraction module extracts the fundamental frequency in the original speech audio and encodes it to obtain a fundamental frequency encoding sequence with a time sequence;

[0013] The speaker encoder encodes the speaker ID to obtain a speaker ID encoding sequence with a time sequence;

[0014] The content encoder performs content encoding on the original speech audio to obtain a phoneme posterior probability sequence with a time sequence, wherein the phoneme posterior probability sequence is used to represent semantic content information in the speech;

[0015] The emotion encoder performs emotion encoding on the original speech audio to obtain an emotion vector latent space encoding sequence with time order;

[0016] The self-attention encoder uses multi-head attention to calculate the attention weight of each position in the fundamental frequency coding sequence, the speaker ID coding sequence, the phoneme posterior probability sequence and the emotion vector latent space coding sequence to capture the dependencies between the positions in the sequence, concatenates the dependencies and integrates them through linear transformation, and calculates the prior distribution mean and variance of the integrated dependencies through the feedforward network layer.

[0017] In an improved speaker privacy protection method capable of preserving speech emotion, the fundamental frequency extraction module extracts the fundamental frequency in the original speech audio in the following manner:

[0018] The baseband extraction module first uses formula (1) to calculate the difference function d of the original speech audio t(τ), and then use formula (2) to normalize the cumulative mean of the difference function d′t(τ), and then use the parabolic interpolation of formula (3) to improve the pitch period estimation The accuracy of the base frequency estimation f is finally derived using formula (4) 0 ;

[0019]

[0020]

[0021] Where t is time, x(n) is the original speech audio, τ is the time delay, N is the window size, n is the index in the time domain, τ 1 The starting time value.

[0022] In an improved speaker privacy protection method capable of preserving speech emotion, the waveform decoder includes: a reparameterized convolution layer and a transposed convolution stack layer, and the step 2 includes:

[0023] The reparameterized convolution layer converts the prior distribution mean and variance into latent variables through a reparameterization operation;

[0024] The transposed convolution stack layer restores the latent variables to audio sequence signals at the sampling point level through an upsampling structure based on a convolutional neural network, thereby obtaining pseudo speaker speech in the time domain.

[0025] In an improved speaker privacy protection method that can retain speech emotions, the end-to-end emotional speech fusion model is obtained by weighted summation of multiple fine-tuned single-speaker end-to-end emotional speech conversion models; the fine-tuned single-speaker end-to-end emotional speech conversion model is obtained by fine-tuning the trained multi-speaker end-to-end emotional speech conversion model using the original speech audio of the single speaker.

[0026] In an improved speaker privacy protection method capable of preserving speech emotions, in the training stage, an end-to-end emotional speech conversion model includes: a priori encoder, a posteriori encoder and a waveform decoder;

[0027] The end-to-end emotional speech conversion model for multiple speakers is obtained by training the end-to-end emotional speech conversion model in the following way:

[0028] Construct training and test sets: For each speech, pack the original speech audio, the corresponding speaker ID, and the corresponding linear spectrum into a tuple. According to the set capacity ratio, divide all tuples into training and test sets. The linear spectrum is obtained by performing time-frequency transformation on the original audio. There are multiple speaker IDs in both the training and test sets.

[0029] A tuple in the training set is input into the prior encoder and the posterior encoder respectively to obtain the prior distribution mean and variance and the posterior distribution mean and variance; the posterior encoder uses the posterior distribution mean and variance to verify the correctness of the prior distribution mean and variance, and outputs the verified correct prior distribution mean and variance to the waveform decoder; the waveform decoder receives the verified correct prior distribution mean and variance through the feedforward network, re-parameterizes the prior distribution mean and variance into latent variables, performs decoding operations on the latent variables, and upsamples the decoded latent variables to restore them to the pseudo speaker speech sound in the time domain. frequency; then compare the pseudo speaker speech audio with the original speech audio to determine whether the loss of the pseudo speaker speech audio is less than a set value, and output the result to the feedforward network when it is not less than the set value, so as to optimize and update the network weights; repeatedly input other tuples into the end-to-end emotional speech conversion model for training until the loss of the pseudo speaker speech audio is less than the set value, and obtain a trained multi-speaker end-to-end emotional speech conversion model, wherein the prior distribution mean and variance and the posterior distribution mean and variance are kept consistent through the Kulbeck-Leibler KL divergence loss constraint function.

[0030] In an improved speaker privacy protection method capable of preserving speech emotions, a trained multi-speaker end-to-end emotional speech conversion model is fine-tuned in the following way to obtain a fine-tuned single-speaker end-to-end emotional speech conversion model:

[0031] The training set is reduced to tuples with only one speaker ID. Following the method of training a multi-speaker end-to-end emotional speech conversion model, the original network weight training is continued to obtain a fine-tuned single-speaker end-to-end emotional speech conversion model.

[0032] In an improved speaker privacy protection method capable of preserving speech emotion, a tuple is input into a posterior encoder to obtain a posterior distribution mean and variance, including:

[0033] The posterior encoder encodes the linear spectrum in a tuple through a variational autoencoder to generate a posterior latent space variable, which is output as the mean and variance of the posterior distribution.

[0034] In an improved speaker privacy protection method capable of preserving speech emotion, each single speaker end-to-end emotion speech conversion model has multiple weight nodes;

[0035] The weighted summation of multiple trained single-speaker end-to-end emotion speech conversion models is performed as follows:

[0036] For each weight node involved in the reasoning process, weighted summation is performed on the parameters of the weight node of multiple trained single-speaker end-to-end emotion speech conversion models to obtain the fused parameters of the weight node.

[0037] On the other hand, a system based on the above-mentioned speaker privacy protection method capable of retaining speech emotions is provided, comprising: an input module and an inference module, wherein:

[0038] The input module is used to obtain the original speech audio of the speaker and input it into the end-to-end emotion speech fusion model;

[0039] The inference module is used to use the end-to-end emotional speech fusion model to infer the original speech audio, change the voiceprint information in the original speech audio, and obtain the pseudo speaker speech audio that retains the emotional state of the original speech audio;

[0040] Wherein, the end-to-end emotion speech fusion model includes: a priori encoder and waveform decoder;

[0041] The prior encoder is used to receive the original speech audio and the speaker identification number ID, calculate and use the fundamental frequency coding, content coding, emotion coding and speaker ID coding of the original speech audio to generate the prior distribution mean and variance, wherein the speaker ID coding contains the voiceprint information;

[0042] The waveform decoder is used to re-parameterize the prior distribution mean and variance into latent variables, perform decoding operations on the latent variables, and upsample the decoded latent variables to restore them into pseudo speaker speech audio in the time domain.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] 1. A fully end-to-end emotional speech conversion model can achieve better sound quality. Compared with the method of digital signal processing, the method of the embodiment of the present invention uses a vocoder process of fitting real data at the sampling level through a neural network to generate an effect that is more in line with human hearing.

[0045] 2. Encode the content with a priori encoder to obtain content features. The content features contain almost no speaker information, but there is a part of the residual emotion caused by the incomplete decoupling of the content and emotion, which can be used further;

[0046] 3. Model fusion is used to generate unseen pseudo-speaker audio and avoid the "out-of-distribution" problem in the reasoning process through the node weighted summation method of the complete model. That is, the node weighted summation method of the complete model ensures that the pseudo-speaker used in the reasoning process is still within the training distribution, and the model does not need to have the ability to generalize emotions. This solves the problem of mismatch between emotion retention and pseudo-speaker information, thereby achieving the goal of balancing emotion retention and anonymity. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1is a flow chart of a speaker privacy protection method capable of retaining voice emotion according to embodiment 1 of the present invention;

[0048] Figure 2 is a schematic structural diagram of a priori encoder according to Embodiment 1 of the present invention;

[0049] Figure 3 is a schematic diagram of the structure of a waveform decoder according to Embodiment 1 of the present invention;

[0050] Figure 4 is a schematic diagram of the structure of an end-to-end speech conversion model in the training process of Example 1 of the present invention;

[0051] Figure 5 is a schematic diagram of the structure of a posterior encoder according to embodiment 1 of the present invention;

[0052] Figure 6 This is an overall framework diagram of a speaker privacy protection method capable of retaining voice emotions according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0053] The solution of the embodiment of the present invention constructs a speaker privacy protection method and system that can retain speech emotions. First, a multi-speaker end-to-end emotional speech conversion model is trained on a very large number of multi-speaker original speech audios. Then, the original speech audios of the speaker identities ID1, ID2... are used to independently fine-tune the end-to-end emotional speech conversion models M1, M2... of the single speaker on the multi-speaker end-to-end emotional speech conversion model. Finally, the models are fused to obtain an end-to-end emotional speech fusion model for use. By using the powerful representation learning ability of deep learning, the speaker ID, speech emotion, semantic content and other parts are decoupled and then resynthesized. Finally, the audio result that can protect the speaker identity and fully retain the speech emotion is output, which balances the contradiction between privacy protection and emotion retention.

[0054] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0055] Example 1

[0056] The embodiment of the present invention provides a speaker privacy protection method that can preserve voice emotions, such as Figure 1 As shown, the following steps are included:

[0057] Step 101: obtaining the original speech audio of the speaker and inputting it into the end-to-end emotion speech fusion model;

[0058] The original voice audio of the speaker here is also the original voice audio to be protected.

[0059] Preferably, the end-to-end emotional speech fusion model is obtained by weighted summing of multiple fine-tuned single-speaker end-to-end emotional speech conversion models;

[0060] The above weighting requires a weight list, which is obtained by inputting the number of speaker IDs in the speaker pool into a unit with a random weight list generation function, and then generating a weight list according to the number of speaker IDs. The weight list can be a one-dimensional random vector with the sum of any elements normalized to one, but cannot be a one-hot vector.

[0061] The fine-tuned single-speaker end-to-end emotional speech conversion model is obtained by fine-tuning the trained multi-speaker end-to-end emotional speech conversion model using the original speech audio of the single speaker.

[0062] Step 102: using the end-to-end emotional speech fusion model to infer the original speech audio, and changing the voiceprint information in the original speech audio;

[0063] The end-to-end emotion speech fusion model includes: a priori encoder and a waveform decoder. The use of the end-to-end emotion speech fusion model to infer the original speech audio includes the following two steps:

[0064] Step 1: The prior encoder receives the original speech audio and the speaker identification number ID, calculates and uses the fundamental frequency coding, content coding, emotion coding and speaker ID coding of the original speech audio to generate the prior distribution mean and variance, where the speaker ID coding contains the voiceprint information;

[0065] Step 2: The waveform decoder re-parameterizes the prior distribution mean and variance into latent variables, performs decoding operations on the latent variables, and upsamples the decoded latent variables to restore them to pseudo speaker speech audio in the time domain.

[0066] Here, the speaker ID containing the voiceprint information is encoded and decoded in combination with other information using a priori encoder and waveform decoder, thereby achieving partial decoupling and resynthesis of the original speech audio, thereby changing the voiceprint information in the original speech audio.

[0067] Specifically, the a priori encoder includes: a fundamental frequency extraction module, a speaker encoder, a content encoder, a sentiment encoder and a self-attention encoder; the above step 1 includes:

[0068] The baseband extraction module extracts the baseband in the original speech audio using a pure signal processing method and encodes it to obtain a baseband encoding sequence with a time sequence.

[0069] The speaker encoder encodes the speaker ID to obtain a speaker ID encoding sequence with a time order; specifically, the speaker encoder may encode the speaker ID using a one-hot vector embedding method.

[0070] The content encoder performs content encoding on the original speech audio to obtain a phoneme posterior probability sequence with a time order, and the phoneme posterior probability sequence is used to represent the semantic content information in the speech; specifically, a pre-trained CTC / transformer structured automatic speech recognition model can be used to input audio and output the probability distribution before the decoding layer, which is also called a phoneme posterior probability graph (PPG), which is used to represent the semantic content information in the speech.

[0071] Specifically, the phoneme posterior probability map generated by automatic speech recognition is used as the content feature. This content feature contains almost no speaker information, but there is some emotional residue caused by incomplete decoupling of content and emotion, which can continue to be used.

[0072] The emotion encoder performs emotion encoding on the original speech audio to obtain an emotion vector latent space encoding sequence with a time sequence. Specifically, a wav2vec model pre-trained on a large-scale emotion dataset can be used to input audio and output a 768-dimensional emotion vector latent space encoding (Emotion-to-Vectors, E2V).

[0073] The self-attention encoder uses multi-head attention to calculate the attention weight of each position in the fundamental frequency coding sequence, the speaker ID coding sequence, the phoneme posterior probability sequence and the emotion vector latent space coding sequence to capture the dependencies between the positions in the sequence, concatenates the dependencies and integrates them through linear transformation, and calculates the prior distribution mean and variance of the integrated dependencies through the feedforward network layer.

[0074] When the content encoder uses a pre-trained CTC / transformer structured automatic speech recognition model and the emotion encoder uses a pre-trained wav2vec model, the structural diagram of the above prior encoder is as follows: Figure 2 The modules, models and encoders are shown in the rounded squares in the figure. Figure 2 The processing process of the input original speech audio and speaker ID by the prior encoder is also shown.

[0075] Specifically, the fundamental frequency extraction module extracts the fundamental frequency in the original speech audio in the following manner:

[0076] The baseband extraction module first uses formula (1) to calculate the difference function d of the original speech audio t(τ), and then use formula (2) to normalize the cumulative mean of the difference function d′t(τ), and then use the parabolic interpolation of formula (3) to improve the pitch period estimation The accuracy of the base frequency estimation f is finally derived using formula (4) 0 ;

[0077]

[0078]

[0079] Where t is time, x(n) is the original speech audio signal, τ is the time delay, N is the window size, n is the index in the time domain, τ 1 The starting time value.

[0080] The structural diagram of the waveform decoder is as follows Figure 3 As shown in the rounded square layers in the figure, it includes: re-parameterized convolution layer and transposed convolution stack layer. The above step 2 includes:

[0081] The reparameterized convolution layer converts the prior distribution mean and variance into latent variables through a reparameterization operation;

[0082] The transposed convolution stack layer restores the latent variables to audio sequence signals at the sampling point level through an upsampling structure based on a convolutional neural network, thereby obtaining pseudo speaker speech in the time domain.

[0083] The convolutional neural network part here can adopt the ResNet structure, which is also called the feedforward network.

[0084] Figure 3 The waveform decoder shown in includes a discriminator in addition to the reparameterized convolution layer and the transposed convolution stack layer. It should be noted that the waveform decoder does not need to include a discriminator during inference, but it does need a discriminator during training. The processing of the discriminator on the input audio will be explained during the training process.

[0085] Step 103: The end-to-end emotional speech fusion model outputs a pseudo speaker speech audio that retains the emotional state of the original speech audio, that is, an anonymized result.

[0086] The above describes the reasoning process of the end-to-end emotional speech conversion model. The following describes the training process of the end-to-end emotional speech conversion model:

[0087] During the training process, the structural diagram of the end-to-end emotional speech conversion model is as follows Figure 4 As shown, it includes: a priori encoder, a posteriori encoder and a waveform decoder, wherein the structural diagram of the posteriori encoder is as shown Figure 5As shown, it includes a variational autoencoder. The prior encoder and waveform decoder are responsible for generating audio during inference, and the posterior encoder is responsible for constraining and verifying the audio decoupling and compression coding capabilities of the prior encoder.

[0088] The end-to-end emotional speech conversion model for multiple speakers is obtained by training the end-to-end emotional speech conversion model in the following way:

[0089] Construct training and test sets: For each speech, pack the original speech audio, the corresponding speaker ID, and the corresponding linear spectrum into a tuple. According to the set capacity ratio, divide all tuples into training and test sets. The linear spectrum is obtained by performing time-frequency transformation on the original audio, such as short-time Fourier transform. There are multiple speaker IDs in both the training and test sets.

[0090] A tuple in the training set is input into the prior encoder and the posterior encoder respectively to obtain the prior distribution mean and variance and the posterior distribution mean and variance; the posterior encoder uses the posterior distribution mean and variance to verify the correctness of the prior distribution mean and variance, and outputs the verified correct prior distribution mean and variance to the waveform decoder; the waveform decoder receives the verified correct prior distribution mean and variance through the feedforward network, re-parameterizes the prior distribution mean and variance into latent variables, performs decoding operations on the latent variables, and upsamples the decoded latent variables to restore them to the pseudo speaker speech audio in the time domain ; Then compare the pseudo speaker speech audio with the original speech audio to determine whether the loss of the pseudo speaker speech audio is less than the set value. When it is not less than the set value, output the result to the feedforward network to optimize and update the network weights; Repeat the input of other tuples into the end-to-end emotional speech conversion model for training until the loss of the pseudo speaker speech audio is less than the set value, and obtain a trained multi-speaker end-to-end emotional speech conversion model, in which the prior distribution mean and variance and the posterior distribution mean and variance are kept consistent through the Kulbeck-Leibler KL divergence loss constraint function.

[0091] Specifically, the trained multi-speaker end-to-end emotional speech conversion model is fine-tuned in the following way to obtain a fine-tuned single-speaker end-to-end emotional speech conversion model:

[0092] The training set is reduced to tuples with only one speaker ID. Following the method of training a multi-speaker end-to-end emotional speech conversion model, the original network weight training is continued to obtain a fine-tuned single-speaker end-to-end emotional speech conversion model.

[0093] Specifically, a tuple is input into the posterior encoder to obtain the mean and variance of the posterior distribution, including:

[0094] The posterior encoder encodes the linear spectrum in a tuple through a variational autoencoder to generate a posterior latent space variable, which is output as the mean and variance of the posterior distribution.

[0095] The above linear spectrum can be obtained by short-time Fourier transform of the original audio and calculated using the following formula (5):

[0096]

[0097] Among them, X k (ω) is the linear spectrum, x(n) is the original speech audio, h(·) is the selected window function, k is the number of frames, n is the index in the time domain, w is the angular frequency, and j is the imaginary unit.

[0098] After the above training, the a priori encoder, posterior encoder and waveform decoder contained in the end-to-end emotional speech conversion model have obtained parameters, and the a priori encoder and the waveform decoder excluding the discriminator can be used in the reasoning process.

[0099] Specifically, each single speaker end-to-end emotional speech conversion model has multiple weight nodes; multiple trained single speaker end-to-end emotional speech conversion models are weighted and summed in the following manner:

[0100] For each weight node involved in the reasoning process, weighted summation is performed on the parameters of the weight node of multiple trained single-speaker end-to-end emotion speech conversion models to obtain the fused parameters of the weight node.

[0101] A neural network model contains many weight nodes. When the data is calculated, it passes through each weight node according to the network rules. The fusion of the model is actually the fusion of the parameters on each node.

[0102] Assume there are five models M1 to M5, each with 100 nodes N 1,1 ~N 5,100 , the i-th node parameter N of the new model M* of the fusion target *,i =p 1 N 1,i +p 2 N 2,i +…+p 5 N 5,i .

[0103] Here, it should be noted that the posterior encoder is only used in the training process, which is equivalent to a "true situation of the pseudo-speaker's speech" to verify and constrain the prior encoding results.

[0104] The overall framework diagram of the speaker privacy protection method capable of retaining speech emotion according to an embodiment of the present invention is as follows: Figure 6 As shown in the figure, the first step is to collect the original speech audio of multiple speakers. Then the linear spectrum features are extracted from the original speech audio. Next, an end-to-end emotional speech conversion model is constructed and trained. Here, speaker-independent pre-trained emotion encoding is used and integrated into the emotional speech conversion model, thereby introducing emotion into the subsequent input end-to-end emotional speech fusion model. Next, the end-to-end emotional speech conversion model is fine-tuned for a single speaker. Next, multiple fine-tuned single-speaker end-to-end emotional speech conversion models are node-fused to obtain an end-to-end emotional speech fusion model. Finally, the original speech audio to be protected is input into the end-to-end emotional speech fusion model, and the inference output is anonymized, that is, the pseudo-speaker speech audio. Pseudo speakers are generated by model fusion, thereby alleviating the mismatch problem between pseudo speakers and target emotions.

[0105] The embodiment of the present invention is applicable to speech data processing scenarios that require a high degree of anonymity and high emotional retention, and provides an effective solution for speaker privacy protection. The method of the embodiment of the present invention is used to obtain an end-to-end emotional speech fusion model, which can make the input speech sound like another pseudo-speaker by changing its embedded voiceprint information, thereby achieving the purpose of hiding identity and protecting privacy. The method of the embodiment of the present invention can retain the content and emotional expression of the input speech without damage, and restore its emotional expressiveness to the greatest extent.

[0106] Example 2

[0107] Based on the same inventive concept as that of Example 1, Example 2 of the present invention provides a system for speaker privacy protection method capable of retaining speech emotion, comprising: an input module and an inference module; wherein:

[0108] The input module is used to obtain the original speech audio of the speaker and input it into the end-to-end emotion speech fusion model;

[0109] The inference module is used to use the end-to-end emotional speech fusion model to infer the original speech audio, change the voiceprint information in the original speech audio, and obtain the pseudo speaker speech audio that retains the emotional state of the original speech audio;

[0110] The end-to-end emotion speech fusion model includes: a priori encoder and a waveform decoder, including:

[0111] The prior encoder is used to receive the original speech audio and the speaker identification number ID, calculate and use the fundamental frequency coding, content coding, emotion coding and speaker ID coding of the original speech audio to generate the prior distribution mean and variance, wherein the speaker ID coding contains the voiceprint information;

[0112] The waveform decoder is used to re-parameterize the prior distribution mean and variance into latent variables, perform decoding operations on the latent variables, and upsample the decoded latent variables to restore them into pseudo speaker speech audio in the time domain.

[0113] In the scheme of the embodiment of the present invention, the original speech audio is input into a speaker privacy protection system that can retain speech emotions, and a pseudo speaker speech is inferred and output. The output speech sounds like a completely unrelated new speaker and has anonymization characteristics, while retaining the emotional state of the original input speech.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A speaker privacy protection method capable of preserving speech emotion, comprising: Obtain the speaker's original speech audio and input it into the end-to-end emotion speech fusion model; The end-to-end emotional speech fusion model is used to infer the original speech audio, change the voiceprint information in the original speech audio, and obtain the pseudo speaker speech audio that retains the emotional state of the original speech audio; The end-to-end emotion speech fusion model includes: a priori encoder and a waveform decoder, and the use of the end-to-end emotion speech fusion model to infer the original speech audio includes: Step 1: The prior encoder receives the original speech audio and the speaker identification number ID, calculates and uses the fundamental frequency coding, content coding, emotion coding and speaker ID coding of the original speech audio to generate the prior distribution mean and variance, where the speaker ID coding contains the voiceprint information; Step 2: The waveform decoder re-parameterizes the prior distribution mean and variance into latent variables, performs decoding operations on the latent variables, and upsamples the decoded latent variables to restore them to pseudo speaker speech audio in the time domain.

2. The speaker privacy protection method capable of preserving speech emotion according to claim 1, characterized in that: The a priori encoder includes: a fundamental frequency extraction module, a speaker encoder, a content encoder, a sentiment encoder and a self-attention encoder; the step 1 includes: The fundamental frequency extraction module extracts the fundamental frequency in the original speech audio and encodes it to obtain a fundamental frequency encoding sequence with a time sequence; The speaker encoder encodes the speaker ID to obtain a speaker ID encoding sequence with a time sequence; The content encoder performs content encoding on the original speech audio to obtain a phoneme posterior probability sequence with a time sequence, wherein the phoneme posterior probability sequence is used to represent semantic content information in the speech; The emotion encoder performs emotion encoding on the original speech audio to obtain an emotion vector latent space encoding sequence with time order; The self-attention encoder uses multi-head attention to calculate the attention weight of each position in the fundamental frequency coding sequence, the speaker ID coding sequence, the phoneme posterior probability sequence and the emotion vector latent space coding sequence to capture the dependencies between the positions in the sequence, concatenates the dependencies and integrates them through linear transformation, and calculates the prior distribution mean and variance of the integrated dependencies through the feedforward network layer.

3. The speaker privacy protection method capable of preserving voice emotion according to claim 2, characterized in that: The fundamental frequency extraction module extracts the fundamental frequency in the original speech audio in the following manner: The baseband extraction module first uses formula (1) to calculate the difference function d of the original speech audio t (τ), and then use formula (2) to normalize the cumulative mean of the difference function d′ t (τ), and then use the parabolic interpolation of formula (3) to improve the pitch period estimation The accuracy of the fundamental frequency is finally derived using formula (4); Among them, t is time, x(n) is the original speech audio, τ is the time delay, N is the window size, n is the index in the time domain, and τ1 is the starting time value.

4. The speaker privacy protection method capable of preserving speech emotion according to claim 1, characterized in that: The waveform decoder includes: a re-parameterized convolution layer and a transposed convolution stack layer, and the step 2 includes: The reparameterized convolution layer converts the prior distribution mean and variance into latent variables through a reparameterization operation; The transposed convolution stack layer restores the latent variables to audio sequence signals at the sampling point level through an upsampling structure based on a convolutional neural network, thereby obtaining pseudo speaker speech in the time domain.

5. The speaker privacy protection method capable of preserving voice emotion according to claim 1, characterized in that: The end-to-end emotional speech fusion model is obtained by weighted summation of multiple fine-tuned single-speaker end-to-end emotional speech conversion models; the fine-tuned single-speaker end-to-end emotional speech conversion model is obtained by fine-tuning the trained multi-speaker end-to-end emotional speech conversion model using the original speech audio of the single speaker.

6. The speaker privacy protection method capable of preserving speech emotion according to claim 5, characterized in that: In the training phase, the end-to-end emotion speech conversion model includes: a priori encoder, a posteriori encoder, and a waveform decoder; The end-to-end emotional speech conversion model for multiple speakers is obtained by training the end-to-end emotional speech conversion model in the following way: Construct training and test sets: For each speech, pack the original speech audio, the corresponding speaker ID, and the corresponding linear spectrum into a tuple. According to the set capacity ratio, divide all tuples into training and test sets. The linear spectrum is obtained by performing time-frequency transformation on the original audio. There are multiple speaker IDs in both the training and test sets. A tuple in the training set is input into the prior encoder and the posterior encoder respectively to obtain the prior distribution mean and variance and the posterior distribution mean and variance; the posterior encoder uses the posterior distribution mean and variance to verify the correctness of the prior distribution mean and variance, and outputs the verified correct prior distribution mean and variance to the waveform decoder; the waveform decoder receives the verified correct prior distribution mean and variance through the feedforward network, re-parameterizes the prior distribution mean and variance into latent variables, performs decoding operations on the latent variables, and upsamples the decoded latent variables to restore them to the pseudo speaker speech sound in the time domain. frequency; then compare the pseudo speaker speech audio with the original speech audio to determine whether the loss of the pseudo speaker speech audio is less than a set value, and output the result to the feedforward network when it is not less than the set value, so as to optimize and update the network weights; repeatedly input other tuples into the end-to-end emotional speech conversion model for training until the loss of the pseudo speaker speech audio is less than the set value, and obtain a trained multi-speaker end-to-end emotional speech conversion model, wherein the prior distribution mean and variance and the posterior distribution mean and variance are kept consistent through the Kulbeck-Leibler KL divergence loss constraint function.

7. The speaker privacy protection method capable of preserving speech emotion according to claim 6, characterized in that: Fine-tune the trained multi-speaker end-to-end emotional speech conversion model in the following way to obtain a fine-tuned single-speaker end-to-end emotional speech conversion model: The training set is reduced to tuples with only one speaker ID. Following the method of training a multi-speaker end-to-end emotional speech conversion model, the original network weight training is continued to obtain a fine-tuned single-speaker end-to-end emotional speech conversion model.

8. The speaker privacy protection method capable of preserving speech emotion according to claim 6, characterized in that: Feed a tuple into the posterior encoder to get the mean and variance of the posterior distribution, including: The posterior encoder encodes the linear spectrum in a tuple through a variational autoencoder to generate a posterior latent space variable, which is output as the mean and variance of the posterior distribution.

9. The speaker privacy protection method capable of preserving voice emotion according to claim 5, characterized in that: Each single speaker end-to-end emotion speech conversion model has multiple weight nodes; The weighted summation of multiple trained single-speaker end-to-end emotion speech conversion models is performed as follows: For each weight node involved in the reasoning process, weighted summation is performed on the parameters of the weight node of multiple trained single-speaker end-to-end emotion speech conversion models to obtain the fused parameters of the weight node.

10. A system based on the speaker privacy protection method capable of preserving voice emotion according to claim 1, characterized in that: include: Input module and reasoning module, where: The input module is used to obtain the original speech audio of the speaker and input it into the end-to-end emotion speech fusion model; The inference module is used to use the end-to-end emotional speech fusion model to infer the original speech audio, change the voiceprint information in the original speech audio, and obtain the pseudo speaker speech audio that retains the emotional state of the original speech audio; Wherein, the end-to-end emotion speech fusion model includes: a priori encoder and waveform decoder; The prior encoder is used to receive the original speech audio and the speaker identification number ID, calculate and use the fundamental frequency coding, content coding, emotion coding and speaker ID coding of the original speech audio to generate the prior distribution mean and variance, wherein the speaker ID coding contains the voiceprint information; The waveform decoder is used to re-parameterize the prior distribution mean and variance into latent variables, perform decoding operations on the latent variables, and upsample the decoded latent variables to restore them into pseudo speaker speech audio in the time domain.

Images