A voiceprint elimination method, device, and electronic equipment
Patent Information
- Application Number
- CN202611232221.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-03
AI Technical Summary
Existing arbitrary-to-arbitrary speech conversion methods based on diffusion models result in synthesized speech that is not clear and natural enough, and pose a security threat to voiceprint recognition.
By acquiring and preprocessing the speech dataset, constructing forward and backward diffusion processes, and utilizing a denoising decoder and autoencoder to optimize the loss function value, a second denoising model and encoding model are generated to eliminate speakerprints.
The generated speech is clearer and more natural, eliminating the threat of voiceprint recognition, breaking through the limitations of language transcription, improving the ease of use and generalization of the method, and protecting the integrity of the speech content.
Smart Images
Figure CN123033507A_ABST
Abstract
Description
Technical Field
[0001] This application relates to artificial intelligence, and to, but is not limited to, a method, apparatus, and electronic device for voiceprint removal. Background Technology
[0002] With the development of biometric identification technology, an increasing number of intelligent automated tools are using human biometrics for identity verification. Voiceprint is a typical biometric feature, and humans use it to determine the identity of others in daily communication. However, the security of service systems using voiceprint recognition poses a threat, and how to eliminate voiceprints has become a problem that needs to be solved.
[0003] The existing technology for arbitrary-to-arbitrary speech conversion based on the diffusion model is DiffVC, but the synthesized speech is not clear and natural enough. Summary of the Invention
[0004] In view of this, embodiments of this application provide a method, apparatus, and electronic device for eliminating voiceprints.
[0005] The technical solution of this application embodiment is implemented as follows: This application provides a method for voiceprint removal, the method comprising: acquiring a speech dataset; preprocessing the speech dataset to obtain a first spectrum and a first audio feature; obtaining a first loss function value based on the first spectrum and a first denoising model; obtaining a second denoising model based on the first loss function value; obtaining a second loss function value based on the first audio feature and a first encoding model; obtaining a second encoding model based on the second loss function value; acquiring speech to be processed; processing the speech to be processed, the second denoising model, and the second encoding model to obtain a denoising result; and removing the voiceprint based on the denoising result.
[0006] Optionally, obtaining a first loss function value based on the first spectrum and the first denoising model includes: obtaining a noisy spectrum based on the first spectrum; processing the noisy spectrum using the first denoising model to obtain estimated noise; and obtaining the first loss function value based on the estimated noise.
[0007] Optionally, obtaining the noisy spectrum based on the first spectrum includes: calculating the noisy spectrum by applying a forward diffusion equation to the first spectrum; the forward diffusion equation is defined as follows: ;in, For time steps, For noise scheduling functions, and These are two independent Wiener processes. It is the average of the Gaussian priors.
[0008] Optionally, obtaining the first loss function value based on the estimated noise includes: acquiring the actual noise; and calculating the first loss function value based on the estimated noise and the actual noise.
[0009] Optionally, obtaining the first loss function value based on the estimated noise includes: processing the estimated noise based on the backdiffusion equation to obtain the first loss function value; the backdiffusion equation is defined as follows: ;in, For time steps, For noise scheduling functions, and These are two independent Wiener processes. It is the average of the Gaussian priors.
[0010] Optionally, based on the first audio features and the first coding model, a second loss function value is obtained, including:
[0011]
[0012]
[0013] in, For convergent smooth L1 reconstruction loss, It is the cosine distance loss that reconstructs the similarity between the original embedding and the original embedding. To constrain the spatial distribution of latent variables, KL divergence is used.
[0014] Optionally, after obtaining the denoising result, the process includes: processing the denoising result using the back diffusion equation to obtain an updated spectrum, and eliminating the voiceprint based on the updated spectrum.
[0015] Optionally, processing is performed based on the speech to be processed, the second denoising model, and the second encoding model to obtain a denoising result, including: preprocessing the speech to be processed to obtain a second audio feature; and processing is performed based on the second audio feature, the second denoising model, and the second encoding model to obtain the denoising result.
[0016] A voiceprint removal device, comprising: an acquisition unit, an analysis unit, and a processing unit; the acquisition unit for acquiring a speech dataset; the analysis unit for preprocessing the speech dataset to obtain a first spectrum and a first audio feature; obtaining a first loss function value based on the first spectrum and a first denoising model; obtaining a second denoising model based on the first loss function value; obtaining a second loss function value based on the first audio feature and a first encoding model; and obtaining a second encoding model based on the second loss function value; the processing unit for acquiring speech to be processed; processing the speech to be processed, the second denoising model, and the second encoding model to obtain a denoising result; and removing the voiceprint based on the denoising result.
[0017] An electronic device includes: a memory for storing at least one set of instructions; a processor for acquiring a speech dataset; preprocessing the speech dataset to obtain a first spectrum and a first audio feature; obtaining a first loss function value based on the first spectrum and a first denoising model; obtaining a second denoising model based on the first loss function value; obtaining a second loss function value based on the first audio feature and a first encoding model; obtaining a second encoding model based on the second loss function value; acquiring speech to be processed; processing the speech to be processed, the second denoising model, and the second encoding model to obtain a denoising result; and eliminating voiceprints based on the denoising result.
[0018] This application provides a method, apparatus, and electronic device for voiceprint removal. First, a speech dataset is acquired. Then, the speech dataset is preprocessed to obtain a first spectrum and a first audio feature. Next, based on the first spectrum and a first denoising model, a first loss function value is obtained. Then, based on the first loss function value, a second denoising model is obtained. Next, based on the first audio feature and a first encoding model, a second loss function value is obtained. Finally, based on the second loss function value, a second encoding model is obtained. The speech to be processed is then acquired. Finally, processing is performed on the speech to be processed, the second denoising model, and the second encoding model to obtain a denoising result. The voiceprint is removed based on the denoising result. In other words, by processing with the second denoising model and the second encoding model, the synthesized speech after voiceprint removal is clearer and more natural. Attached Figure Description
[0019] Figure 1 A flowchart illustrating the voiceprint removal method provided in this application embodiment; Figure 2 Another flowchart of the voiceprint elimination method provided in the embodiments of this application; Figure 3 Another flowchart of the voiceprint elimination method provided in the embodiments of this application; Figure 4Another flowchart of the voiceprint elimination method provided in the embodiments of this application; Figure 5 Another flowchart of the voiceprint elimination method provided in the embodiments of this application; Figure 6 Another flowchart of the voiceprint elimination method provided in the embodiments of this application; Figure 7 Another flowchart of the voiceprint elimination method provided in the embodiments of this application; Figure 8 A schematic diagram of the voiceprint cancellation device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structural composition of the electronic device provided in the embodiments of this application. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0021] Please refer to Figure 1 , Figure 2 ,in, Figure 1 A flowchart illustrating an implementation of a voiceprint removal method provided in this application embodiment may include: Step S101: Obtain the speech dataset; Step S102: Preprocess the speech dataset to obtain the first spectrum and the first audio features; Step S103: Based on the first spectrum and the first denoising model, obtain the first loss function value; Step S104: Obtain the second denoising model based on the first loss function value; Step S105: Based on the first audio features and the first coding model, obtain the value of the second loss function; Step S106: Obtain the second encoding model based on the second loss function value; Step S107: Obtain the speech to be processed; Step S108: Process the speech to be processed, the second denoising model, and the second coding model to obtain the denoising result, and eliminate the voiceprint based on the denoising result.
[0022] The speech dataset is any high-quality, clear multi-speaker speech dataset, without the need for transcription and without strict language requirements. In this embodiment, the publicly available VCTK English dataset is used. The first spectrum is the Mel spectrum. The first audio features include at least speaker embeddings, self-supervised content representations, and other audio features. The first denoising model can be a denoising decoder for performing backdiffusion. The second denoising model can be an optimized and trained denoising decoder. The first encoding model can be an autoencoder for reconstructing speaker embeddings. The second encoding model can be an optimized and trained autoencoder, with its decoding network layer used as the speaker embedding generator.
[0023] Specifically, in step one, obtain a speech dataset. Obtain any high-quality, clear multi-speaker speech dataset without transcribing text and without strict language requirements. In this embodiment, the publicly available VCTK English dataset is used.
[0024] Step two involves preprocessing the speech dataset to obtain audio features such as Mel spectrum, self-supervised content representation, and speaker embedding.
[0025] (1) Resample the source speech to 22.05kHz and calculate the Mel spectrum with 80 filter banks (i.e. 80 dimensions) with a window size of 1024 and a step size of 256.
[0026] (2) The source speech is resampled to the working sampling rate of the pre-trained self-supervised model and input into the model. The state of the last hidden layer is extracted as the self-supervised content representation. In this embodiment, the open-source self-supervised model ContentVec is selected, with a working sampling rate of 16kHz. The 768-dimensional hidden state of the last Transformer layer is extracted, and the downsampling rate along the time axis is 320.
[0027] (3) Input the source speech into the pre-trained speaker verification network to obtain a fixed 256-dimensional speaker embedding.
[0028] Step 3: Construct the forward diffusion and backward diffusion processes, as well as the denoising decoder used to perform backward diffusion.
[0029] The forward diffusion process is defined by a stochastic differential equation with infinite time steps:
[0030] Similarly, the reverse diffusion process is defined as:
[0031] in For time steps, For noise scheduling functions, In this embodiment, and Set to 0.05 and 20.0. and These are two independent Wiener processes, with Gaussian noise added using simulation. It is the average of the Gaussian priors, assuming The forward process distributes the data. Convert to To ensure the flexibility of the method, It was set to 0.
[0032] For a noise reduction decoder, such as Figure 3 As shown, the denoising decoder employs a U-Net structure with multiple stacked network blocks and skip connections. Each network block mainly consists of residual blocks and linear attention layers. The input to the denoising decoder is a noisy Mel-frequency spectrum. Self-monitoring content representation and time step The 130-channel feature set consists of: a noisy spectrum as a single channel; a self-supervised content representation interpolated and projected into a content channel with the same shape as the noisy spectrum; and speaker embeddings and time-step embeddings concatenated, projected, and broadcast into 128 conditional channels with the same shape as the input noisy spectrum. The 130×80×T feature input to the decoder undergoes two rounds of downsampling and two rounds of upsampling. Each network block extracts features from the input, and skip connections are used to fuse multi-scale information. The final output is an estimated noise of shape 1×80×T. Subtracting this noise from the noisy spectrum yields the denoising result for a single pass.
[0033] Step 4: Calculate the noisy spectrum and corresponding noise at a certain time step in the forward process based on the Mel spectrum obtained in Step 2.
[0034] Based on a forward diffusion process with fixed parameters, given It can be obtained through sampling. And the corresponding Gaussian noise:
[0035] The denoising decoder is specifically trained to minimize the weighted L2 loss of the back-diffusion process:
[0036]
[0037] Step 5: Use a denoising decoder, with the noisy spectrum from Step 4 as the main input, and the self-supervised content representation, speaker embedding from Step 2, and time step from Step 4 as conditions to estimate the noise.
[0038] Step 6: Calculate the loss function value based on the noise estimated in Step 5 and the corresponding actual noise in Step 3, and optimize the denoising decoder.
[0039] Step 7: Repeat steps 4 to 6, iterating multiple times to obtain the trained denoising decoder.
[0040] Step 8: Construct an autoencoder. Input the speaker embedding from Step 2 and output the reconstructed speaker embedding. Calculate the loss function value based on the reconstructed embedding and the real embedding, and optimize the autoencoder. After multiple iterations, the trained autoencoder is obtained. Use its decoding network layer as the speaker embedding generator.
[0041] Step 9: Preprocess the source speech to be processed, extract the self-supervised content representation, and use the speaker embedding generator to synthesize the target speaker embedding; Step 10: Input the Gaussian noise, the self-supervised content representation of the source speech from Step 9, and the target speaker embedding into the trained denoising decoder and perform one step of back diffusion. Output the denoising result of this step, replace the Gaussian noise with the denoising result of this step, and continue to perform back diffusion. After multiple iterations, the Mel spectrum of the changed voiceprint is obtained. Step 11: Input the Mel spectrum obtained in Step 10 into the vocoder to obtain the voiceprint-cancelled speech.
[0042] Please refer to Figure 4 The method in this embodiment may include obtaining a first loss function value based on a first spectrum and a first denoising model, including: Step S201: Obtain the noisy spectrum based on the first spectrum; Step S202: Process the noisy spectrum using the first denoising model to obtain the estimated noise; Step S203: Obtain the first loss function value based on the estimated noise.
[0043] Specifically, the obtained Mel spectrum is used to calculate the noisy spectrum and corresponding noise at a certain time step in the forward process. Then, a denoising decoder is used, with the noisy spectrum in step four as the main input, and the self-supervised content representation, speaker embedding in step two, and the time step in step four as conditions to estimate the noise. The estimated noise and the corresponding actual noise in step three are used to calculate the loss function value, and the denoising decoder is optimized.
[0044] The method in this embodiment may include: obtaining a noisy spectrum based on a first spectrum, including: calculating the noisy spectrum based on the first spectrum using a forward diffusion equation; the forward diffusion equation is defined as follows: ;in, For time steps, For noise scheduling functions, and These are two independent Wiener processes. It is the average of the Gaussian priors.
[0045] Please refer to Figure 5 The method in this embodiment may include: obtaining a first loss function value based on the estimated noise, including: Step S301: Obtain the actual noise; Step S302: Calculate the first loss function value based on the estimated noise and the actual noise.
[0046] Specifically, the actual noise is obtained in step three, and then the loss function value is calculated based on the actual noise and the estimated noise.
[0047] The method in this embodiment may include: obtaining a first loss function value based on the estimated noise, including: processing the estimated noise based on the back diffusion equation to obtain the first loss function value; the back diffusion equation is defined as follows: ;in, For time steps, For noise scheduling functions, and These are two independent Wiener processes. It is the average of the Gaussian priors.
[0048] The method in this embodiment may include: obtaining a second loss function value based on a first audio feature and a first coding model, including:
[0049]
[0050]
[0051]
[0052] in, For convergent smooth L1 reconstruction loss, It is the cosine distance loss that reconstructs the similarity between the original embedding and the original embedding. To constrain the spatial distribution of latent variables, KL divergence is used.
[0053] For details, please refer to Figure 6 (1) The constructed autoencoder adopts a multilayer perceptron structure with a single hidden layer. In this embodiment, the input speaker embedding is 256-dimensional, the number of hidden layer units is set to 384, and the latent variable distribution is set to a 64-dimensional Gaussian distribution. When a speaker embedding is input, the encoding perceptron first maps it to 384 dimensions, and then maps it to a 64-dimensional mean vector. And a 64-dimensional covariance vector This represents the distribution of latent variables, from which a latent vector is sampled. Then, the decoding perceptron maps it to 384 dimensions and then back to 256 dimensions to obtain the reconstructed speaker embedding.
[0054] (2) The autoencoder is trained with the goal of reconstructing the speaker embedding. The loss function of the autoencoder is designed as follows:
[0055]
[0056]
[0057]
[0058] in, For a smoother L1 reconstruction loss that converges more easily, This is to ensure the cosine distance loss used to reconstruct the similarity between the original embedding and the embeddedness in the speaker embedding space. To constrain the spatial distribution of latent variables, KL divergence is used.
[0059] (3) After training the autoencoder, the speaker embedding generator is obtained by taking its decoding perceptron. The latent space vector is sampled from the 64-dimensional Gaussian distribution and input into the speaker embedding generator to obtain a pseudo speaker embedding that is close to the real speaker embedding distribution.
[0060] The method in this embodiment may include: after obtaining the denoising result, processing the denoising result through the back diffusion equation to obtain an updated spectrum, and eliminating the voiceprint based on the updated spectrum.
[0061] Please refer to Figure 7 The method in this embodiment may include: processing based on the speech to be processed, a second denoising model, and a second coding model to obtain a denoising result, including: Step S401: Preprocess the speech to be processed to obtain the second audio features; Step S402: Process based on the second audio features, the second denoising model, and the second coding model to obtain the denoising result.
[0062] The second audio feature may include: a self-supervised content representation of the speech to be processed, and a speaker embedding of the speech to be processed.
[0063] The method in this embodiment may include: estimating the noise in the noisy spectrum using a denoising decoder based on a diffusion probability model, conditionally using the self-supervised content representation of the source speech, speaker embedding, and time step; training the denoising decoder by calculating a loss function with the actual noise; then reconstructing the speaker embedding using an autoencoder, training the autoencoder by calculating a loss function with the real speaker embedding, and using its decoding network layer as the speaker embedding generator; finally, extracting the self-supervised content representation of the speech to be processed, synthesizing a pseudo speaker embedding using the trained speaker embedding generator, inputting it into the trained denoising decoder, and synthesizing the Mel spectrum for speaker removal after multiple iterations, which is then converted into speech by a vocoder. This invention helps users conveniently remove speakerprints from source speech without compromising the integrity and naturalness of the speech content.
[0064] Compared with existing technologies, the advantages of this application are: 1. Traditional speech encoders require training on both text and speech modal data. The training process has high data requirements and is limited by language. In contrast, this proposal trains a self-supervised encoder based on speaker embedding, which only requires source speech data. This breaks through the limitations of transcription language and has strong generalization ability.
[0065] 2. Traditional speech representation uses Mel spectrum, which results in the loss of information such as speech content and prosody. This proposal uses speaker embedding, self-supervised content representation, and Mel spectrum for fusion representation, which better protects the integrity of more fine-grained effective information in speech.
[0066] 3. Traditional speech conversion methods rely on the source audio of the target speaker as reference audio, while this proposal uses speaker embedding to represent the speaker's identity, which does not rely on reference audio and improves the ease of use of the method.
[0067] Please refer to Figure 8 The apparatus of this embodiment may include the following structure: Acquisition unit 501 is used to acquire the speech dataset; Analysis unit 502 is used to preprocess the speech dataset to obtain a first spectrum and a first audio feature; obtain a first loss function value based on the first spectrum and a first denoising model; obtain a second denoising model based on the first loss function value; obtain a second loss function value based on the first audio feature and a first coding model; and obtain a second coding model based on the second loss function value. The processing unit 503 is used to acquire the speech to be processed; process the speech to be processed, the second denoising model, and the second coding model to obtain the denoising result; and eliminate the voiceprint based on the denoising result.
[0068] Please refer to Figure 9This embodiment of the present application also discloses an electronic device, which includes at least one processor 601, and at least one memory 602 and a bus 603 connected to the processor 601; wherein the processor 601 and the memory 602 communicate with each other through the bus 603; the processor 601 is used to call program instructions in the memory 602 to execute the above-mentioned voiceprint elimination method.
[0069] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for eliminating voiceprints, characterized in that, The method includes: Obtain the speech dataset; The speech dataset is preprocessed to obtain a first spectrum and a first audio feature; Based on the first spectrum and the first denoising model, the first loss function value is obtained; Based on the first loss function value, a second denoising model is obtained; Based on the first audio features and the first coding model, the second loss function value is obtained; Based on the second loss function value, a second encoding model is obtained; Acquire the speech to be processed; The speech to be processed is processed based on the second denoising model and the second encoding model to obtain a denoising result, and the voiceprint is eliminated based on the denoising result.
2. The method according to claim 1, characterized in that, Based on the first spectrum and the first denoising model, the first loss function value is obtained, including: Based on the first spectrum, the noisy spectrum is obtained; The noisy spectrum is processed using the first denoising model to obtain an estimated noise level. The first loss function value is obtained based on the estimated noise.
3. The method according to claim 2, characterized in that, Based on the first spectrum, a noisy spectrum is obtained, including: The noisy spectrum is obtained by calculating the first spectrum based on the forward diffusion equation; The forward diffusion equation is defined as follows: ； in, For time steps, For noise scheduling functions, and These are two independent Wiener processes. It is the average of the Gaussian priors.
4. The method according to claim 3, characterized in that, Based on the estimated noise, the first loss function value is obtained, including: Obtain the actual noise; The first loss function value is obtained by calculating based on the estimated noise and the actual noise.
5. The method according to claim 4, characterized in that, Based on the estimated noise, the first loss function value is obtained, including: The estimated noise is processed based on the back diffusion equation to obtain the first loss function value; The reverse diffusion equation is defined as follows: ； in, For time steps, For noise scheduling functions, and These are two independent Wiener processes. It is the average of the Gaussian priors.
6. The method according to claim 5, characterized in that, Based on the first audio features and the first coding model, the second loss function value is obtained, including: in, For convergent smooth L1 reconstruction loss, It is the cosine distance loss that reconstructs the similarity between the original embedding and the original embedding. To constrain the spatial distribution of latent variables, KL divergence is used.
7. The method according to claim 6, characterized in that, After obtaining the denoising result, the process includes: The denoising result is processed by the back diffusion equation to obtain an updated spectrum, and the speaker is eliminated based on the updated spectrum.
8. The method according to claim 1, characterized in that, Based on the speech to be processed, the second denoising model, and the second encoding model, a denoising result is obtained, including: The speech to be processed is preprocessed to obtain second audio features; The denoising result is obtained by processing the second audio feature, the second denoising model, and the second encoding model.
9. A voiceprint elimination device, characterized in that, The device includes: an acquisition unit, an analysis unit, and a processing unit. The acquisition unit is used to acquire a voice dataset; The analysis unit is configured to preprocess the speech dataset to obtain a first spectrum and a first audio feature; obtain a first loss function value based on the first spectrum and a first denoising model; obtain a second denoising model based on the first loss function value; obtain a second loss function value based on the first audio feature and a first encoding model; and obtain a second encoding model based on the second loss function value. The processing unit is used to acquire the speech to be processed; to process the speech, the second denoising model, and the second encoding model to obtain a denoising result; and to eliminate the voiceprint based on the denoising result.
10. An electronic device, characterized in that, include: Memory, used to store at least one set of instructions; The processor is used to acquire the speech dataset; The speech dataset is preprocessed to obtain a first spectrum and a first audio feature; Based on the first spectrum and the first denoising model, the first loss function value is obtained; Based on the first loss function value, a second denoising model is obtained; Based on the first audio features and the first coding model, the second loss function value is obtained; Based on the second loss function value, a second encoding model is obtained; Acquire the speech to be processed; The speech to be processed is processed based on the second denoising model and the second encoding model to obtain a denoising result, and the voiceprint is eliminated based on the denoising result.