A high-fidelity voice conversion system and method

Abstract

The present invention discloses a high-fidelity voice conversion system and method, which relates to the technical field of voice recognition and voice synthesis. The system includes an audio acquisition module, a voice recognition module, a prosody encoder, a decoder and a vocoder; the audio acquisition module acquires the input audio of the source speaker and the reference audio of the target speaker; the voice recognition module extracts the content information and intonation information in the input audio; the prosody encoder extracts the prosody information of the reference audio; the decoder generates a mel spectrum according to the content information, intonation information and prosody information; the vocoder converts the mel spectrum into audio. The present invention can convert any input audio to the target timbre, completely retain the content and rhythm of the input audio, and has high-fidelity sound quality after conversion, which can greatly meet the voice change scenarios of various scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition and speech synthesis, and more particularly to a high-fidelity speech conversion system and method. Background Art

[0002] Voice conversion refers to the technology of converting one person's words into another person's voice while keeping the content unchanged. A vivid example is Conan's voice-changing bow. There are many application scenarios for voice conversion. In movie dubbing, there is a significant difference between the voice of the dubbing artist and the actor himself, and the audience always expects to hear the actor's own voice. At this time, the voice conversion technology can be used to convert the voice of the dubbing artist into a voice with the actor's own voice, thereby enhancing the viewing effect of the movie; in some call scenarios, when it is inconvenient for the speaker to reveal his or her identity information, the voice conversion technology can be used to convert the voice into another person's voice to achieve anonymity; in the field of live broadcasting, some anchors do not have a good voice due to vocal cord problems. The voice conversion technology is used to convert the anchor's voice into other pleasant human voices to achieve the purpose of beautifying the voice and improving the live broadcast effect.

[0003] Speech conversion has a long history of development. Traditional speech conversion methods change the timbre of the original audio by changing the fundamental frequency, formant and other signal processing methods of the audio, which can achieve specific voice-changing purposes, but it takes a lot of time to optimize the parameters and the effect is unstable. Speech conversion technology based on deep learning has greatly improved the effect of speech conversion in terms of the richness of voice-changing scenes and the naturalness of the converted audio. There are still many problems to be solved in the existing speech conversion technology. Although speech conversion based on speech recognition and speech synthesis can achieve good conversion sound quality and timbre, the rhythm and content of the original audio are difficult to guarantee; speech conversion based on generative adversarial networks or variational autoencoders can completely retain the content and rhythm of the original audio, but the original timbre is difficult to eliminate and the sound quality is often distorted.

[0004] Therefore, how to improve the fidelity effect of speech conversion is a problem that technical personnel in this field need to solve urgently. Summary of the invention

[0005] In view of this, the present invention provides a high-fidelity speech conversion system and method, which can customize the conversion timbre by providing reference audio, convert any input audio to the target timbre, completely retain the content and rhythm of the input audio, and have high-fidelity sound quality after conversion, which can greatly meet the voice changing scenarios of various scenarios.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A high-fidelity speech conversion system, including an audio acquisition module, a speech recognition module, a prosody encoder, a decoder and a vocoder;

[0008] An audio acquisition module, which acquires the input audio of the source speaker and the reference audio of the target speaker;

[0009] Speech recognition module, extracting content information and intonation information from input audio;

[0010] A prosody encoder, which extracts prosody information of the reference audio;

[0011] The decoder generates a mel-spectrogram based on the content information, intonation information, and prosody information;

[0012] Vocoder, which converts the mel-spectrogram into audio.

[0013] Preferably, the speech recognition module includes a content encoder and a intonation extractor; the wenet algorithm model is used as the content encoder, a large-scale speech recognition data set is collected to train the content encoder, and after the training converges, the input audio is input into the content encoder to obtain the phoneme posterior probability, and the phoneme posterior probability is used as the content information of the audio; the intonation extractor uses praat-parselmouth to extract the fundamental frequency information in the input audio and quantizes it to obtain the intonation information.

[0014] Preferably, the prosody information includes overall prosody information and latent feature prosody information; the prosody encoder includes a coarse-grained prosody encoder and a fine-grained prosody encoder; the coarse-grained prosody encoder includes multiple convolution layers and multiple pooling layers, and obtains the overall prosody and timbre information of the target speaker from the reference audio, that is, a one-dimensional vector is obtained from the reference audio through multiple convolution layers and pooling layers as the overall prosody information; the fine-grained prosody encoder includes multiple convolution layers and multiple pooling layers, and obtains multi-dimensional latent features from the reference audio as latent feature prosody information; the number of pooling layers of the coarse-grained prosody encoder is greater than the number of pooling layers of the fine-grained prosody encoder. The overall prosody information includes the prosody and timbre of the target speaker, and the latent feature prosody information is specific to the detailed characteristics such as the pronunciation and stress of each word of the target speaker.

[0015] Preferably, the content information, intonation information, overall prosody information and latent feature prosody information are summed or concatenated in the decoder to generate a mel-spectrogram; the latent feature prosody information is used as a query of the attention mechanism and is sent to the decoder for attention together with the content information.

[0016] Preferably, the decoder adopts a Transformer structure.

[0017] Preferably, the vocoder uses a HIFI-GAN model to map the mel-spectrogram to audio.

[0018] Preferably, the content encoder comprises a Transformer Encoder layer, the input audio is input into the Transformer Encoder layer, and the posterior probability of the phoneme is obtained as the content information.

[0019] A high-fidelity voice conversion method comprises the following steps:

[0020] Step 1: Get the input audio of the source speaker and the reference audio of the target speaker;

[0021] Step 2: Extract content information and intonation information from the input audio, and extract the prosody information of the target speaker from the reference audio;

[0022] Step 3: Generate a Mel-spectrogram based on content information, intonation information, and prosody information;

[0023] Step 4: Convert the Mel-spectrogram to audio.

[0024] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a high-fidelity speech conversion system and method, which uses the posterior probability of phonemes (PPG) in speech recognition to solve the problem of source speaker identity information leakage, and designs rhythm extraction modules of different granularities to ensure the timbre similarity between the converted audio and the target speaker as much as possible, and, in order to stabilize the tone after conversion, extract the fundamental frequency from the input audio as additional information to guide the conversion, thereby learning the reference audio of the target speaker, and converting the input audio to be converted into audio with the timbre of the target speaker. The speech conversion system and method involved in the present invention solve the problems of low timbre similarity and unnatural sound quality in current speech conversion technology, and can better meet the needs of speech voice change in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0026] Figure 1 The accompanying drawing is a schematic structural diagram of a high-fidelity speech conversion system provided by the present invention;

[0027] Figure 2 The accompanying drawing is a schematic diagram of the content encoder training process provided by the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Example 1

[0030] The embodiment of the present invention discloses a high-fidelity speech conversion system, such as Figure 1 As shown, it includes an audio acquisition module, a speech recognition module, a prosody encoder, a decoder and a vocoder;

[0031] An audio acquisition module, which acquires the input audio of the source speaker and the reference audio of the target speaker;

[0032] Speech recognition module, extracting content information and intonation information from input audio;

[0033] A prosody encoder, which extracts prosody information of the reference audio;

[0034] The decoder generates a mel-spectrogram based on the content information, intonation information, and prosody information;

[0035] Vocoder, which converts the mel-spectrogram into audio.

[0036] Furthermore, the speech recognition module includes a content encoder and a intonation extractor; the wenet algorithm model is used as the content encoder, and the content encoder is trained with a large-scale speech recognition data set. After the training converges, the input audio is input into the content encoder to obtain the phoneme posterior probability, and the phoneme posterior probability is used as the content information of the audio; the intonation extractor uses praat-parselmouth to extract the fundamental frequency information in the input audio and quantizes it to obtain the intonation information.

[0037] The phonetic posteriorgrams (PPG) in speech recognition are used as content information, that is, the probability distribution of each frame of speech on each phoneme is used as content information. The speech recognition algorithm wenet is used to perform pre-training on large-scale speech data using phonemes as recognition units.

[0038] Since PPG is close to text, it inevitably lacks intonation information. It is difficult to retain the intonation of the original audio by converting only with PPG. Even after conversion, the pronunciation of some words may be abnormal due to inaccurate intonation. The present invention adopts praat-parselmouth to extract fundamental frequency information and quantize it. The fundamental frequency is generally used to measure intonation.

[0039] Furthermore, the prosody information includes overall prosody information and latent feature prosody information; the prosody encoder includes a coarse-grained prosody encoder and a fine-grained prosody encoder; the coarse-grained prosody encoder includes multiple layers of convolutional layers and multiple layers of pooling layers, which obtain the overall prosody and timbre information of the target speaker from the reference audio, and obtain a one-dimensional vector from the reference audio through multiple layers of convolutional layers and pooling layers as the overall prosody information; the fine-grained prosody encoder includes multiple layers of convolutional layers and multiple layers of pooling layers, which obtain multi-dimensional latent features from the reference audio as the latent feature prosody information; the number of pooling layers of the coarse-grained prosody encoder is greater than the number of pooling layers of the fine-grained prosody encoder.

[0040] Furthermore, the content information, intonation information, overall rhythm information and latent feature rhythm information are summed or concatenated in the decoder to generate a Mel-spectrogram; the latent feature rhythm information is used as the query of the attention mechanism and is sent to the decoder for attention together with the content information.

[0041] Furthermore, the decoder adopts the Transformer structure, which integrates the input intonation, content and prosody information and finally generates the converted acoustic features, namely the Mel-spectrogram.

[0042] Prosody is an implicit feature of speech, which can be understood as the style of speech, such as the ups and downs of speech, timbre style (rough and dull, crisp and loud or other), etc., which is difficult to describe with specific numerical values. The present invention divides prosody into two types: coarse-grained prosody and fine-grained prosody; coarse-grained prosody represents the overall style of the audio, including the overall speech emotion, speaking speed, etc.; fine-grained prosody represents more delicate information of the audio, such as the changes in the weight of each word, the connection between the tone of the previous moment and the next moment, etc. On the one hand, prosody information can better assist the decoder in reconstructing the Mel spectrum, and on the other hand, it can better simulate the style of the target speaker, making the converted audio more like what the target speaker said.

[0043] The coarse-grained prosody encoder obtains the overall prosody and timbre information of the speaker from the reference audio, and obtains a one-dimensional vector from the reference audio through multiple layers of convolution and pooling layers. The vector is summed or concatenated with the content information for subsequent Mel spectrum generation. The fine-grained prosody encoder obtains multi-dimensional latent features from the reference audio through multiple layers of convolution and pooling layers. The latent features are used as queries of the attention mechanism, and are paid attention to with the content information and then sent to the decoder to generate speech acoustic features.

[0044] Furthermore, the vocoder uses the HIFI-GAN model to map the mel-spectrogram to audio, and is trained on multiple speakers to achieve the purpose of being universal for all speakers.

[0045] Furthermore, the vocoder can continue to make fine adjustments to the target speaker's audio by adjusting the parameters of the vocoder.

[0046] Furthermore, the content encoder includes a Transformer Encoder layer, and the input audio is input to the Transformer Encoder layer to obtain the posterior probability of the phoneme, that is, to obtain the content information.

[0047] like Figure 2 As shown in the figure, the probability distribution of each phoneme in each frame output by the Transformer Encoder of the wenet algorithm model is PPG. Pre-training with large-scale data enables PPG to have the ability to generalize to various input audios. PPG has natural advantages as content information for speech conversion. First, it completely contains all the text content of the speech, while retaining the same speech speed information as the original audio. In addition, PPG is very close to text, and text does not contain any speaker information. Therefore, PPG basically and completely filters out the timbre information of the original audio, which is very ideal content information for speech conversion.

[0048] Example 2

[0049] In a specific embodiment, a high-fidelity speech conversion method includes the following steps:

[0050] S1: Get the input audio of the source speaker and the reference audio of the target speaker;

[0051] S2: extracts content information and intonation information from the input audio, and extracts the prosody information of the target speaker from the reference audio;

[0052] S3: Generate a mel-spectrogram based on content information, intonation information and prosody information;

[0053] S4: Convert the Mel-spectrogram to audio.

[0054] First, the content information and intonation information are extracted from the input audio, and the prosody information of the target speaker is extracted from the reference audio. The decoder combines the above extracted information with the speaker information to generate the converted Mel spectrogram, and the vocoder converts the generated spectrogram into audio. During the training phase of the entire system,

[0055] The input audio and the reference audio can be the same audio or different audios from the same speaker. After pre-training on large-scale speech data, fine-tuning is performed on the target speaker's data to achieve better conversion results. The input audio is the audio to be converted, and the reference audio is the audio of the target speaker, so that the audio to be converted is converted into audio with the timbre of the target speaker.

[0056] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0057] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high-fidelity speech conversion system, characterized in that: It includes audio acquisition module, speech recognition module, prosody encoder, decoder and vocoder; An audio acquisition module, which acquires the input audio of the source speaker and the reference audio of the target speaker; The speech recognition module extracts content information and intonation information from the input audio; the speech recognition module includes a content encoder and an intonation extractor; the wenet algorithm model is used as the content encoder, and a large-scale speech recognition data set is collected to train the content encoder. After the training converges, the input audio is input into the content encoder to obtain the phoneme posterior probability, and the phoneme posterior probability is used as the content information; the intonation extractor uses praat-parselmouth to extract the fundamental frequency information in the input audio and quantize it to obtain the intonation information; A prosody encoder extracts prosody information of the reference audio; the prosody information includes overall prosody information and latent feature prosody information; The prosody encoder includes a coarse-grained prosody encoder and a fine-grained prosody encoder; the coarse-grained prosody encoder includes multiple convolutional layers and multiple pooling layers, and obtains the overall prosody and timbre information of the target speaker from the reference audio as the overall prosody information, and the overall prosody information represents an overall style of the audio, including the overall speech emotion and speech speed; the fine-grained prosody encoder includes multiple convolutional layers and multiple pooling layers, and obtains multi-dimensional latent features from the reference audio as latent feature prosody information, and the latent feature prosody information represents the style of the speech, including the timbre style of the speech, the stress of each word, and the connection between the tone of the previous moment and the next moment; the number of pooling layers of the coarse-grained prosody encoder is greater than the number of pooling layers of the fine-grained prosody encoder; The decoder generates a mel-spectrogram based on the content information, intonation information, and prosody information; The decoder integrates the input intonation, content, and prosody information to finally generate the converted acoustic features; Prosody information assists the decoder in reconstructing the Mel spectrum and simulating the style of the target speaker; the content information, intonation information, overall prosody information and latent feature prosody information are summed or concatenated in the decoder to generate a Mel spectrum graph; the overall prosody information is summed or concatenated with the content information for subsequent generation of the Mel spectrum; the latent feature prosody information is used as the query of the attention mechanism and is sent to the decoder for attention together with the content information; Vocoder, which converts the mel-spectrogram into audio.

2. A high-fidelity speech conversion system according to claim 1, characterized in that: The decoder adopts the Transformer structure.

3. A high-fidelity speech conversion system according to claim 1, characterized in that: The vocoder uses the HIFI-GAN model to map the Mel-spectrogram to audio.

4. A high-fidelity speech conversion system according to claim 1, characterized in that: The content encoder includes a Transformer Encoder layer, and the input audio is input to the Transformer Encoder layer to obtain the posterior probability of the phoneme as the content information.

5. A high-fidelity speech conversion method, characterized in that: A high-fidelity speech conversion system applied to any one of claims 1 to 4, comprising the following steps: Step 1: Get the input audio of the source speaker and the reference audio of the target speaker; Step 2: Extract content information and intonation information from the input audio, and extract the prosody information of the target speaker from the reference audio; Step 3: Generate a Mel-spectrogram based on content information, intonation information, and prosody information; Step 4: Convert the Mel-spectrogram to audio.

Images