Voice anonymization method based on content-related frame-level speaker voiceprint modeling

Abstract

The invention discloses a voice anonymization method based on content-related frame-level speaker voiceprint modeling. An automatic voice recognition acoustic model is used for extracting bottleneck features from source voice and reference voice to serve as representation of semantic content; introducing an attention module to align content vectors between the reference voice and the source voice; a Mel spectrogram is extracted from the reference voice, frame-level pseudo speaker voiceprint representation is obtained through a speaker voiceprint encoder, then the frame-level pseudo speaker voiceprint representation serves as a key value to be input into the introduced attention module, and the weighted sum of voiceprint embedding in the reference voice is achieved through an attention mechanism; and inputting the semantic content vector and the fundamental frequency value extracted from the source voice and the frame-level pseudo speaker voiceprint representation in the target voice into a voice generation decoder to generate anonymous voice. According to the method, the personal identity information of the speaker can be more thoroughly removed from the anonymized voice, the uniqueness of the voice of the pseudo speaker is improved, and the quality and the understandability of the synthesized voice are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to a speech anonymization method based on content-related frame-level speaker voiceprint modeling. Background Art

[0002] In recent years, with the application of deep learning technology in speech technology, various speech technologies have experienced rapid development, including automatic speech recognition (ASR), speaker recognition (speaker recognition), text to speech synthesis (TTS), voice conversion (VC), etc., which have made rapid progress. On the other hand, speech data contains a lot of personal information, such as the speaker's identity, gender, age, religion, region, etc. The process of collecting speech data has caused serious privacy issues, especially the speaker's identity information. Voice privacy infringement is becoming an important security issue. With the widespread application of deep learning, advanced speech processing technology makes it possible to extract personal information from user voice data. At the same time, speech synthesis and voice conversion technology can generate voice data that is highly close to the real voice. This allows the user's timbre and speaking style to be used to forge voice, thereby achieving reputation attacks, fraud and other malicious acts against the user himself.

[0003] As a privacy protection technology, speech anonymization technology aims to remove the speaker's identity information from speech signals while retaining semantic content and speech quality and meeting downstream task requirements. Most of the existing mainstream speech anonymization frameworks use independently trained speaker voiceprint representation extractors to extract sentence-level speaker embeddings from speech, which are used to construct pseudo speaker embeddings and train back-end speech synthesis models. Studies have shown that the existing mean-constructed pseudo speaker embedding method makes the features of pseudo speakers tend to cluster, weakening the uniqueness and differences of pseudo speakers, making it easier for attack models to identify the correspondence between the original speech and the anonymized speech; in addition, the constructed pseudo speaker embedding is not in the same distribution as the speaker voiceprint representation used for speech synthesis model training, which will seriously damage the speech quality when it is input into the synthesis model to generate anonymized speech. Summary of the invention

[0004] The purpose of the present invention is to provide a speech anonymization method based on content-related frame-level speaker voiceprint modeling, which can make the anonymized speech more thoroughly remove the speaker's personal identity information, enhance the uniqueness of the pseudo-speaker's voice, and improve the quality and intelligibility of the synthesized speech.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A speech anonymization method based on content-related frame-level speaker voiceprint modeling, the method comprising:

[0007] Step 1: Use the automatic speech recognition acoustic model to extract bottleneck features from the source speech and the reference speech as representations of the semantic content;

[0008] Step 2: Introduce an attention module to align the content vectors between the reference speech and the source speech, and provide context-dependent attention weights;

[0009] Step 3: extract the mel-spectrogram from the reference speech, pass it through the speaker voiceprint encoder, obtain the frame-level pseudo speaker voiceprint representation, and then input it into the introduced attention module as a key value. The weighted sum of the voiceprint embedding in the reference speech is realized through the attention mechanism, and it is used as the frame-level pseudo speaker voiceprint representation in the target speech;

[0010] Step 4: Input the semantic content vector and fundamental frequency value extracted from the source speech, as well as the frame-level pseudo speaker voiceprint representation in the target speech, into the speech generation decoder to generate anonymous speech.

[0011] It can be seen from the technical solution provided by the present invention that the above method can make the anonymized speech more thoroughly remove the personal identity information of the speaker, improve the uniqueness of the pseudo-speaker's speech, and improve the quality and comprehensibility of the synthesized speech, thereby improving the anonymization method's ability to protect the privacy of the speech and meet the purpose of downstream speech tasks. Compared with traditional methods, it has obvious improvements in removing identity information in speech, the uniqueness of pseudo-speaker speech and speech quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A flow chart of a method for speech anonymization based on content-related frame-level speaker voiceprint modeling provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] like Figure 1 FIG. 1 is a flow chart of a method for speech anonymization based on content-related frame-level speaker voiceprint modeling provided by an embodiment of the present invention. The method includes:

[0016] Step 1: Use the automatic speech recognition (ASR) acoustic model (AM) to extract bottleneck features from the source speech and the reference speech as representations of the semantic content;

[0017] In step 1, the automatic speech recognition acoustic model ASR is trained on the selected speech dataset, such as LibriSpeechtrain-clean-100 and train-other-500 speech datasets. The automatic speech recognition acoustic model takes the 40-dimensional speech Mel Frequency Cepstral Coefficients (MFCC) and the speaker identity features extracted from the source speech and the reference speech as input, and performs classification tasks on phonemes;

[0018] The main body of the automatic speech recognition acoustic model is a 17-layer factor time-delay neural network (TDNN-F), with the 256-dimensional output of the last layer as the extracted bottleneck feature with a time shift of 20ms.

[0019] The original speech is used as the source speech, and certain speakers are selected from the predetermined speaker pool. Each speaker randomly selects several speech pieces, called group speech, which are used as reference speech.

[0020] Step 2: Introduce an attention module to align the content vectors between the reference speech (Key) and the source speech (Query), and provide context-dependent attention weights;

[0021] In this step, the introduced attention module is expressed as:

[0022]

[0023] Where Q is the query, which represents the semantic features of the source speech; K is the key, which represents the semantic features of the reference speech; d k is the dimension of K.

[0024] Step 3: extract the Mel-spectrogram from the reference speech, pass it through the speaker voiceprint encoder, obtain the frame-level pseudo speaker voiceprint representation, and then input it into the introduced attention module as the key value. The weighted sum of the voiceprint embedding in the reference speech is realized through the attention mechanism, and it is used as the frame-level pseudo speaker voiceprint representation in the target speech;

[0025] In step 3, the speaker voiceprint encoder uses the main structure before the pooling layer of the ECAPA-TDNN model. The Mel-spectrogram is time-shifted by 10ms. The latent variables are generated through one-dimensional convolution and activation function, and then the speaker identity information in the speech is extracted through the three-layer compression excitation residual block SE-ResBlock.

[0026] The convolution layer in the residual block SE-ResBlock uses an increasing expansion coefficient to achieve multi-scale information fusion on speech duration, and then downsamples the features by 1 / 2 through a layer of convolution to achieve alignment with semantic features;

[0027] The frame-level pseudo speaker voiceprint representation representing the pseudo speaker is derived from the reference speech based on the attention weight, which can realize context-dependent pseudo speaker voiceprint representation.

[0028] Frame-level pseudo speaker voiceprint representation is used to adjust the voiceprint characteristics in the source speech.

[0029] Step 4: Input the semantic content vector and fundamental frequency value extracted from the source speech, as well as the frame-level pseudo speaker voiceprint representation in the target speech, into the speech generation decoder to generate anonymous speech.

[0030] In step 4, the decoder uses HiFi-GAN decoder as the synthesis backend, which is essentially the generator of HiFi-GAN v1, consisting of 4 layers of transposed convolutions, each of which contains a multi-receptive field fusion module MRF, expressed as:

[0031] y=HiFi-GAN(C,S tgt )(2)

[0032] Where y represents the generated anonymous speech waveform; C represents the semantic content vector extracted from the source speech; S tgt Represents the frame-level pseudo speaker voiceprint representation in the target speech;

[0033] In the loss cost function of anonymous speech generation, the HiFi-GAN loss cost function in speech encoding and decoding is adopted, which is expressed as L HIFI-GAN ;

[0034] In addition, a classification layer is applied to the frame-level pseudo speaker voiceprint representation extracted from the reference speech, and the cross entropy loss is calculated, denoted as L CE ;

[0035] The loss function L is defined as the weighted sum of these two functions as follows:

[0036] L=αL HIFI-GAN +(1-α)L CE (3)

[0037] Where α represents the weight to balance the two loss functions.

[0038] In addition, in the specific implementation, in the training process of the anonymization model of the present application, the training can be divided into two stages: in the first stage, a single voice is used as both the source voice and the reference voice to obtain a preliminary neural network parameter model; in the second stage, given a training speaker, a voice is used as the source voice, and N voices greater than 1 are randomly selected and used as reference voices to further train the model.

[0039] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.

[0040] In order to verify the effectiveness of the method proposed in the embodiment of the present invention, the following experiment was designed:

[0041] 1. Experimental data and configuration

[0042] The LibriTTStrain-clean-100 dataset is used to train the decoder model for speech synthesis; the LibriSpeechtrain-clean-360 dataset is used to train and evaluate the models for voiceprint confirmation and speech recognition. Speech recognition uses an end-to-end recognition model; the voiceprint recognition model extracts X-vectors and uses probabilistic linear discriminant analysis (PLDA) for back-end scoring and discrimination. The predefined speaker pool contains 1,160 speakers from LibriTTStrain-other-500. The registration set and test set are composed of the development set and test set of LibriSpeech and VCTK.

[0043] In the method proposed in the present invention, a frame shift of 20 milliseconds is used when extracting semantic vectors and frame-level speaker embeddings. The speaker voiceprint encoder uses the encoder in the ECAPA-TDNN architecture, which consists of 2 one-dimensional convolutional layers and 3 SE-Res2Blocks, and finally generates a 512-dimensional speaker embedding. The decoder for speech generation is an enhanced HiFi-GAN decoder, which consists of 3 residual blocks (Resblocks) and 2 upsampling convolutional layers.

[0044] In the second phase of training the anonymized model, 15 speech clips were randomly selected from each training speaker as reference. During speech anonymization (inference), 20 speech clips were randomly selected from each group speaker as group speech (reference speech).

[0045] 2. Experimental results compared with mainstream systems

[0046] This application uses three mainstream systems for comparative experiments:

[0047] VPC22 system: The baseline system proposed by the VPC22 competition officials uses a pre-trained X-vector extractor to extract the X-vector of each group speaker, calculates its mean to construct a pseudo speaker embedding.

[0048] PSD: Predict the feature distribution of group speakers and construct pseudo speaker embeddings by random sampling in space.

[0049] NAC: A speech anonymization method based on a neural network audio codec. It builds pseudo-speaker embeddings through an external prompt library and combines speech coding to complete the speech anonymization process.

[0050] First, the difference between de-identified information and anonymized speech:

[0051] Speaker voice verification (ASV) is used to evaluate the ability to remove speaker identity information and voice differences, and is tested in two scenarios: ignorant and semi-informed. In the ignorant scenario, the original voice is used for ASV model training and speaker registration, while the anonymized voice is used for testing; in the semi-informed scenario, the anonymized voice is used for ASV model training, speaker registration, and testing. In order to evaluate the voice uniqueness between pseudo speakers, the training data of the ASV model is anonymized at the speaker level.

[0052] Table 1 below shows the results of the four comparison methods. The equal error rate (EER) is used to measure the performance of the system. The higher the EER, the better the performance of the anonymization method.

[0053] Table 1 EERs (%) of voiceprint confirmation obtained by four comparison methods

[0054]

[0055]

[0056] It can be observed from Table 1 above that the method proposed in the embodiment of the present invention achieves about 50% EERs in each test set of the ignorant scenario and achieves the highest average EER compared with other methods, which proves its effectiveness in removing identity information. In the semi-informed scenario, the method proposed in the present application obtains EERs significantly higher than other methods, showing improved performance in pseudo-speaker uniqueness. It can be seen from the experimental results that by using the frame level to model the differences in the speaker's voiceprint, the diversity of the learned speaker's voiceprint attributes is enhanced compared with the sentence-level modeling speaker voiceprint representation used in the comparison method.

[0057] Then perform voice quality analysis:

[0058] The quality and intelligibility of the generated anonymized speech are evaluated to evaluate the practicality of various anonymization methods. First, a Mean Opinion Score (MOS) test is performed to measure the quality of the anonymized speech. 12 paid speakers are asked to rate the naturalness of the speech on a scale of 1 to 5 with a step size of 0.5. A higher score indicates a better performance. As shown in Table 2, the method proposed in the present invention achieves a higher MOS score than the other three methods, which shows the effectiveness of frame-level pseudo-speaker learning in improving the quality of anonymized speech.

[0059] Table 2 Mean opinion scores (MOS, with 95% confidence intervals) obtained on the recorded and anonymized speech of the four comparison methods

[0060]

[0061] The intelligibility of the anonymized speech is measured by the word error rates (WERs) obtained in the automatic speech recognition (ASR) evaluation, as shown in Table 3 below. For each speech type, the ASR model is trained using the corresponding speech, whether it is the original recording or the speech anonymized using the respective method. It can be observed that the method proposed in this application achieves lower WERs compared to the NAC method, and the WERs are comparable to those of the VPC22 and NAC methods, with relative changes within about 5%. These results demonstrate the effectiveness of the algorithm proposed in this application in maintaining the intelligibility of the original speech.

[0062] Table 3 Speech recognition word error rate on the development set (dev) and test set (test) of LibriSpeech (libri) and VCTK (vctk) datasets

[0063]

[0064]

[0065] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0066] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A speech anonymization method based on content-related frame-level speaker voiceprint modeling, characterized in that: The method comprises: Step 1: Use the automatic speech recognition acoustic model to extract bottleneck features from the source speech and the reference speech as representations of the semantic content; Step 2: Introduce an attention module to align the content vectors between the reference speech and the source speech, and provide context-dependent attention weights; Step 3: extract the mel-spectrogram from the reference speech, pass it through the speaker voiceprint encoder, obtain the frame-level pseudo speaker voiceprint representation, and then input it into the introduced attention module as a key value. The weighted sum of the voiceprint embedding in the reference speech is realized through the attention mechanism, and it is used as the frame-level pseudo speaker voiceprint representation in the target speech; Step 4: Input the semantic content vector and fundamental frequency value extracted from the source speech, as well as the frame-level pseudo speaker voiceprint representation in the target speech, into the speech generation decoder to generate anonymous speech.

2. The speech anonymization method based on content-related frame-level speaker voiceprint modeling according to claim 1 is characterized in that: In step 1, an automatic speech recognition acoustic model ASR is trained on the selected speech dataset. The automatic speech recognition acoustic model takes the 40-dimensional speech Mel frequency cepstral coefficients and the speaker identity features extracted from the source speech and the reference speech as input, and performs a classification task on the phonemes; The main body of the automatic speech recognition acoustic model is a 17-layer factor time-delay neural network, with the 256-dimensional output of the last layer as the bottleneck feature for extraction; The original speech is used as the source speech, and certain speakers are selected from the predetermined speaker pool. Each speaker randomly selects several speech pieces, called group speech, which are used as reference speech.

3. The speech anonymization method based on content-related frame-level speaker voiceprint modeling according to claim 1 is characterized in that: In step 2, the introduced attention module is expressed as: Where Q is the query, which represents the semantic features of the source speech; K is the key, which represents the semantic features of the reference speech; d k is the dimension of K.

4. The method for speech anonymization based on content-related frame-level speaker voiceprint modeling according to claim 1, characterized in that: In step 3, the speaker voiceprint encoder adopts the main structure before the pooling layer of the ECAPA-TDNN model, generates latent variables through one-dimensional convolution and activation function, and then extracts the speaker identity information in the reference speech through the three-layer compressed excitation residual block SE-ResBlock; The convolution layer in the residual block SE-ResBlock uses an increasing expansion coefficient to achieve multi-scale information fusion on speech duration; then a layer of convolution is used to downsample the features by 1 / 2 to achieve alignment with the semantic features; Among them, the frame-level pseudo speaker voiceprint representation representing the pseudo speaker is derived from the reference speech based on the attention weight, which can realize context-dependent pseudo speaker voiceprint representation.

5. The speech anonymization method based on content-related frame-level speaker voiceprint modeling according to claim 1 is characterized in that: In step 3, the frame-level pseudo speaker voiceprint representation is used to adjust the voiceprint characteristics in the source speech.

6. The speech anonymization method based on content-related frame-level speaker voiceprint modeling according to claim 1 is characterized in that: In step 4, the decoder uses HiFi-GAN decoder as the synthesis backend, which consists of 4 layers of transposed convolutions, each of which contains a multi-receptive field fusion module MRF, expressed as: and=HiFi-GAN(C,S tgt )(2) Where y represents the generated anonymous speech waveform; C represents the semantic content vector extracted from the source speech; S tgt Represents the frame-level pseudo speaker voiceprint representation in the target speech; In the loss cost function of anonymous speech generation, the HiFi-GAN loss cost function in speech encoding and decoding is adopted, which is expressed as L HIFI-GAN ; In addition, a classification layer is applied to the frame-level pseudo speaker voiceprint representation extracted from the reference speech, and the cross entropy loss is calculated, denoted as L CE ; The loss function L is defined as the weighted sum of these two functions as follows: L=αL HIFI-GAN +(1-α)L CE (3) Where α represents the weight to balance the two loss functions.

Images