A method and system for sound reproduction

Abstract

The present invention provides a method and system for voice replication, the method comprising S1, pre-training a speaker encoder: preparing a speaker encoder and pre-training it; S2, pre-training a speech synthesis model: preparing a speech synthesis model and pre-training it; S3, audio acquisition: providing a user with a sample text containing a certain amount of text, and collecting a sample audio of the user reading the sample text; S4, training a synthesizer: fine-tuning the synthesizer; the speech synthesis model formed by the combination of the trained synthesizer and the vocoder is used to realize the voice replication of a specific user. The present invention can quickly use a very small amount of samples to realize the user's voice replication and speech synthesis, and can be widely used in the fields of audio reading, intelligent customer service, and intelligent vehicle-mounted.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and in particular to a method and system for sound reproduction. Background Art

[0002] With the development of speech synthesis technology, people have begun to pursue more personalized and familiar voices. Speech synthesis services with fixed voices can no longer meet the needs of actual scenarios. The birth of sound reproduction technology meets this practical need.

[0003] Voice replication technology is developed from speech synthesis technology. It can use a small number of samples to quickly replicate the user's voice and achieve high-quality synthesis effects. However, existing voice replication technology requires a large number of training samples, has high requirements for audio quality, and has unstable synthesis effects and poor robustness, which needs to be improved and perfected. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for sound replication in order to achieve rapid replication of the user's voice and speech synthesis using a very small amount of samples in order to solve the problems in the prior art.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for sound reproduction, characterized by comprising:

[0007] S1. Pre-training speaker encoder: prepare a speaker encoder and pre-train it;

[0008] The speaker encoder includes a neural network structure for extracting audio features of a speaker;

[0009] S2. Pre-training speech synthesis model: preparing a speech synthesis model and pre-training it;

[0010] The speech synthesis model includes a synthesizer and a vocoder, wherein the synthesizer is used to convert text into a Mel spectrum, and the vocoder is used to convert the Mel spectrum into a waveform;

[0011] The synthesizer includes a text encoder and a feature decoder; the text encoder is used to extract text features of the text, and the feature decoder is used to generate a Mel spectrum according to the extracted audio features and text features;

[0012] S3, audio collection: providing a sample text containing a certain amount of text to the user, and collecting a sample audio of the user reading the sample text;

[0013] S4. Train the synthesizer: Input the sample audio into the pre-trained speaker encoder to output the user's audio features; input the sample text into the text encoder to output the text features; input the audio features of the sample audio and the text features of the sample text into the feature decoder together to generate a sample Mel spectrum to fine-tune the synthesizer; the speech synthesis model formed by the combination of the trained synthesizer and the vocoder is used to achieve speech reproduction of a specific user.

[0014] Furthermore, between S3 and S4, it also includes: S31, data preprocessing: performing denoising, volume normalization and silence removal processing on the audio collected in S3, and performing text regularization processing on the sample text.

[0015] Furthermore, S3 also includes:

[0016] Background noise detection: Uses an audio energy calculation algorithm to assess whether the user's environment is quiet enough. When the detected energy is higher than the set threshold, the user is prompted to record in a quiet environment.

[0017] Speaker volume detection: First, denoise the audio recorded by the user, then detect the audio energy. When the detected energy is lower than the set threshold, the user is prompted to increase the volume and re-record.

[0018] Pronunciation accuracy detection: The audio recorded by the user is converted into text and then compared with the sample text. When the accuracy is lower than the set threshold, the user is prompted to re-record according to the sample text.

[0019] Furthermore, the neural network structure in the speaker encoder includes at least one of CNN (convolutional neural network), RNN (recurrent neural network), LSTM (long short-term memory network) and transformer network.

[0020] Furthermore, the neural network structure in the speaker encoder is composed of 3 layers of LSTM stacked together, and each LSTM layer has 256 units;

[0021] The speaker encoder further includes a preprocessing unit and a normalization processing unit;

[0022] The preprocessing unit is used to process the audio into a Mel spectrum and then input it into the neural network structure;

[0023] The normalization processing unit is used to normalize the output of the neural network structure.

[0024] Furthermore, in S1, the speaker encoder is pre-trained using multi-speaker data or a voiceprint recognition dataset; after the pre-training is completed, the network parameters of the speaker encoder are no longer updated.

[0025] Furthermore, in the speech synthesis model, the synthesizer adopts the Tacotron or FastSpeech network structure, and the vocoder adopts the WaveNet, WaveGlow, MelGAN or LPCNet network structure.

[0026] Furthermore, in S2, the speech synthesis model is pre-trained using multi-speaker speech synthesis data; after the training is completed, the network parameters of the vocoder are no longer updated, and the network parameters of the synthesizer are fine-tuned according to the training in S4.

[0027] A sound reproduction system, used for performing sound reproduction according to the method described above, comprising a speaker encoder, a speech synthesis model and an audio acquisition module;

[0028] The speaker encoder is used to extract the audio features of the speaker;

[0029] The speech synthesis model includes a synthesizer and a vocoder, wherein the synthesizer is used to convert text into a Mel spectrum, and the vocoder is used to convert the Mel spectrum into a waveform; the synthesizer includes a text encoder and a feature decoder; the text encoder is used to extract text features of the text, and the feature decoder is used to generate a Mel spectrum according to the extracted audio features and text features;

[0030] The audio collection module is used to provide a sample text containing a certain amount of text to the user, and to collect a sample audio of the user reading the sample text.

[0031] Furthermore, it also includes a pronunciation detection module, and the pronunciation detection module includes:

[0032] Background noise detection module: used to use the audio energy calculation algorithm to evaluate whether the user's environment is quiet enough. When the detected energy is higher than the set threshold, the user is prompted to record in a quiet environment;

[0033] Speaker volume detection module: used to first denoise the audio recorded by the user, and then detect the audio energy. When the detected energy is lower than the set threshold, the user is prompted to increase the volume and re-record;

[0034] Pronunciation accuracy detection module: used to convert the audio recorded by the user into text, and then compare it with the sample text. When the accuracy is lower than the set threshold, the user is prompted to re-record according to the sample text.

[0035] The present invention provides a method and system for voice replication, including the entire process of detection, replication, synthesis, etc., which can adapt to different usage scenarios and users. The present invention can quickly use a very small number of samples to achieve user voice replication and speech synthesis, and can be widely used in audio reading, intelligent customer service, intelligent car-mounted and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a sound replication method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1 As shown, the present invention provides a method for sound reproduction, comprising:

[0039] S1. Pre-training speaker encoder: prepare a speaker encoder and pre-train it;

[0040] Since voice reproduction requires a very small number of samples to train a speech synthesis model, accurately extracting the speaker's features (Speaker embedding) from a small number of samples is the first step in voice reproduction, which is the role of the speaker encoder. In order to obtain speaker features that can distinguish the timbre of different people, the speaker encoder needs to have strong discrimination capabilities to distinguish different speakers, so speaker verification tasks are usually used to train the model.

[0041] Specifically, the speaker encoder includes a neural network structure for extracting the audio features of the speaker; the neural network structure includes at least one of CNN (convolutional neural network), RNN (recurrent neural network), LSTM (long short-term memory network) and transformer network. In this embodiment, the neural network structure in the speaker encoder is composed of 3 layers of LSTM stacked, and each layer of LSTM has 256 units.

[0042] The speaker encoder further includes a preprocessing unit and a normalization processing unit;

[0043] The preprocessing unit is used to process the audio into a Mel spectrum and then input it into the neural network structure;

[0044] The normalization processing unit is used to normalize the output of the neural network structure.

[0045] In S1, the speaker encoder is pre-trained using multi-speaker data or a voiceprint recognition dataset; after pre-training, the network parameters of the speaker encoder are no longer updated.

[0046] S2. Pre-training speech synthesis model: preparing a speech synthesis model and pre-training it;

[0047] The essence of sound reproduction is still a task of converting text into audio, following the same process of text to Mel spectrum and then to audio as speech synthesis.

[0048] In this embodiment, the speech synthesis model includes a synthesizer and a vocoder, the synthesizer is used to convert the text into a Mel spectrum, and the vocoder is used to convert the Mel spectrum into a waveform;

[0049] The synthesizer includes a text encoder and a feature decoder; the text encoder is used to extract text features of the text, and the feature decoder is used to generate a Mel spectrum according to the extracted audio features and text features.

[0050] The synthesizer adopts Tacotron or FastSpeech network structure, and the vocoder adopts WaveNet, WaveGlow, MelGAN or LPCNet network structure. In this embodiment, Tacotron2 is used as the synthesizer, and HiFi-GAN is used as the vocoder.

[0051] In S2, the speech synthesis model is pre-trained using multi-speaker speech synthesis data; after the training is completed, the network parameters of the vocoder are no longer updated, and the network parameters of the synthesizer are fine-tuned according to the training in S4.

[0052] S3. Audio collection: Provide the user with a sample text containing a certain amount of text (10-30 sentences, generally 30 sentences), and collect sample audio of the user reciting the sample text.

[0053] Furthermore, S3 also includes:

[0054] Background noise detection: Uses an audio energy calculation algorithm to assess whether the user's environment is quiet enough. When the detected energy is higher than the set threshold, the user is prompted to record in a quiet environment.

[0055] Speaker volume detection: First, denoise the audio recorded by the user, then detect the audio energy. When the detected energy is lower than the set threshold, the user is prompted to increase the volume and re-record.

[0056] Pronunciation accuracy detection: The audio recorded by the user is converted into text and then compared with the sample text. When the accuracy is lower than the set threshold, the user is prompted to re-record according to the sample text.

[0057] Audio collection can be implemented through online recording or file uploading. Specifically, online recording can be implemented through APP, mini-programs or web applications, and background noise detection, speaker volume detection and pronunciation accuracy detection can be performed online in real time. After successful recording, the audio and text pairs will be retained for standby use. The file uploading method generally only requires speaker volume detection and pronunciation accuracy detection.

[0058] As an improvement, after S3, S31, data preprocessing, is also included: denoising, volume normalization and silence removal are performed on the audio collected in S3, and text regularization is performed on the sample text.

[0059] S4. Train the synthesizer: Input the sample audio into the pre-trained speaker encoder to output the user's audio features; input the sample text into the text encoder to output the text features; input the audio features of the sample audio and the text features of the sample text into the feature decoder together to generate a sample Mel spectrum to fine-tune the synthesizer; the speech synthesis model formed by the combination of the trained synthesizer and the vocoder can be used to achieve voice reproduction of a specific user.

[0060] Furthermore, the present invention also provides a sound replication system for performing sound replication according to the above-mentioned method, comprising a speaker encoder, a speech synthesis model and an audio acquisition module.

[0061] Wherein, the speaker encoder is used to extract the audio features of the speaker;

[0062] The speech synthesis model includes a synthesizer and a vocoder, wherein the synthesizer is used to convert text into a Mel spectrum, and the vocoder is used to convert the Mel spectrum into a waveform; the synthesizer includes a text encoder and a feature decoder; the text encoder is used to extract text features of the text, and the feature decoder is used to generate a Mel spectrum according to the extracted audio features and text features;

[0063] The audio collection module is used to provide a sample text containing a certain amount of text to the user, and to collect a sample audio of the user reading the sample text.

[0064] Furthermore, the sound replication system further includes a pronunciation detection module, and the pronunciation detection module includes:

[0065] Background noise detection module: used to use the audio energy calculation algorithm to evaluate whether the user's environment is quiet enough. When the detected energy is higher than the set threshold, the user is prompted to record in a quiet environment;

[0066] Speaker volume detection module: used to first denoise the audio recorded by the user, and then detect the audio energy. When the detected energy is lower than the set threshold, the user is prompted to increase the volume and re-record;

[0067] Pronunciation accuracy detection module: used to convert the audio recorded by the user into text, and then compare it with the sample text. When the accuracy is lower than the set threshold, the user is prompted to re-record according to the sample text.

[0068] The present invention provides a method and system for voice replication, including the entire process of detection, replication, synthesis, etc., which can adapt to different usage scenarios and users. The present invention can quickly use a very small number of samples to achieve user voice replication and speech synthesis, and can be widely used in audio reading, intelligent customer service, intelligent car-mounted and other fields.

[0069] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for sound reproduction, characterized in that: include: S1. Pre-training speaker encoder: prepare a speaker encoder and pre-train it; The speaker encoder includes a neural network structure for extracting audio features of a speaker; S2. Pre-training speech synthesis model: preparing a speech synthesis model and pre-training it; The speech synthesis model includes a synthesizer and a vocoder, wherein the synthesizer is used to convert text into a Mel spectrum, and the vocoder is used to convert the Mel spectrum into a waveform; The synthesizer includes a text encoder and a feature decoder; the text encoder is used to extract text features of the text, and the feature decoder is used to generate a Mel spectrum according to the extracted audio features and text features; S3, audio collection: providing a sample text containing a certain amount of text to the user, and collecting a sample audio of the user reading the sample text; S4, training the synthesizer: input the sample audio into the pre-trained speaker encoder to output the user's audio features; input the sample text into the text encoder to output the text features; input the audio features of the sample audio and the text features of the sample text into the feature decoder together to generate a sample Mel spectrum, so as to fine-tune the synthesizer; The speech synthesis model formed by the trained synthesizer and vocoder is used to achieve speech reproduction of a specific user; Also included in S3: Background noise detection: Uses an audio energy calculation algorithm to assess whether the user's environment is quiet enough. When the detected energy is higher than the set threshold, the user is prompted to record in a quiet environment. Speaker volume detection: First, denoise the audio recorded by the user, then detect the audio energy. When the detected energy is lower than the set threshold, the user is prompted to increase the volume and re-record. Pronunciation accuracy detection: The audio recorded by the user is converted into text, and then compared with the sample text. When the accuracy is lower than the set threshold, the user is prompted to re-record according to the sample text; The neural network structure in the speaker encoder includes at least one of CNN (convolutional neural network), RNN (recurrent neural network), LSTM (long short-term memory network) and transformer network; The neural network structure in the speaker encoder is composed of 3 layers of LSTM stacked together, and each LSTM layer has 256 units; The speaker encoder further includes a preprocessing unit and a normalization processing unit; The preprocessing unit is used to process the audio into a Mel spectrum and then input it into the neural network structure; The normalization processing unit is used to normalize the output of the neural network structure; In S1, the speaker encoder is pre-trained using multi-speaker data or voiceprint recognition datasets; after pre-training, the network parameters of the speaker encoder are no longer updated; In S2, the speech synthesis model is pre-trained using multi-speaker speech synthesis data; after the training is completed, the network parameters of the vocoder are no longer updated, and the network parameters of the synthesizer are fine-tuned according to the training in S4.

2. The method for sound reproduction according to claim 1, characterized in that: Between S3 and S4, it also includes: S31, data preprocessing: performing denoising, volume normalization and silence removal processing on the audio collected in S3, and performing text regularization processing on the sample text.

3. The method for sound reproduction according to claim 1, characterized in that: In the speech synthesis model, the synthesizer adopts the Tacotron or FastSpeech network structure, and the vocoder adopts the WaveNet, WaveGlow, MelGAN or LPCNet network structure.

4. A sound reproduction system, used for performing sound reproduction according to any one of the methods of claims 1 to 3, characterized in that: Includes speaker encoder, speech synthesis model and audio acquisition module; The speaker encoder is used to extract the audio features of the speaker; The speech synthesis model includes a synthesizer and a vocoder, wherein the synthesizer is used to convert text into a Mel spectrum, and the vocoder is used to convert the Mel spectrum into a waveform; the synthesizer includes a text encoder and a feature decoder; the text encoder is used to extract text features of the text, and the feature decoder is used to generate a Mel spectrum according to the extracted audio features and text features; The audio collection module is used to provide a sample text containing a certain amount of text to the user, and to collect a sample audio of the user reading the sample text.

5. The sound reproduction system according to claim 4, characterized in that: It also includes a pronunciation detection module, which includes: Background noise detection module: used to use the audio energy calculation algorithm to evaluate whether the user's environment is quiet enough. When the detected energy is higher than the set threshold, the user is prompted to record in a quiet environment; Speaker volume detection module: used to first denoise the audio recorded by the user, and then detect the audio energy. When the detected energy is lower than the set threshold, the user is prompted to increase the volume and re-record; Pronunciation accuracy detection module: used to convert the audio recorded by the user into text, and then compare it with the sample text. When the accuracy is lower than the set threshold, the user is prompted to re-record according to the sample text.

Images