Speech synthesis method based on speaker feature control
By integrating speaker identity feature control in the speech synthesis network, using spectrum encoding and duration modeling, the problem of low speech synthesis efficiency in the prior art is solved, and a fast and accurate speech synthesis effect is achieved.
Patent Information
- Application Number
- CN202310525575.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-12
AI Technical Summary
The existing speech synthesis technology requires a large number of target speaker corpus and a long migration time to achieve personalized speech synthesis, which fails to effectively consider the design of the network model, resulting in low speech synthesis efficiency.
Using a speech synthesis method based on speaker feature control, the cascading speaker identity feature control network is designed to integrate the speaker identity information into the encoder and decoder of the speech synthesis network, and uses spectrum encoding, phoneme encoding and location information. Perform duration modeling to achieve efficient control of voice.
It realizes the rapid replica of the voiceprint information of the speaker with fewer migration samples, and generates the voice of the speaker whose voice is closer to the reference audio. The migration speed is fast and can generate the voice of the corresponding speaker in actual applications.
Smart Images

Figure CN118942439A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology in the field of speech processing, in particular to a speech synthesis method based on speaker feature control. Background Art
[0002] At present, speech synthesis technology generally performs transfer learning on pre-trained models on large-scale data sets, and synthesizes the speech of a specific speaker through fine-tuning. No targeted consideration is given to the design of the network model. This results in existing technical solutions requiring more corpus of the target speaker and a longer migration time to achieve system customization for personalized speech synthesis. Summary of the invention
[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a speech synthesis method based on speaker feature control, which does not require an additional speaker switching unit to store the speaker's identity information. The speaker's identity information can be implicitly learned directly from the reference audio, and the speaker's identity information can be integrated into the encoder and decoder of the speech synthesis network through a designed speaker identity feature control network. The method can more directly and efficiently achieve the regulation of finer-grained features of the synthesized speech, thereby achieving the control of the speaker's identity, and has good performance in both subjective and objective indicators.
[0004] The present invention is achieved through the following technical solutions:
[0005] The present invention relates to a speech synthesis method based on speaker feature control. In an offline stage, a coding-decoding architecture neural network including a cascaded speaker identity feature control network is constructed. Reference audio after spectrum encoding, text information after phoneme encoding, and encoded position information are used as training samples to perform pitch and intensity prediction training based on duration modeling. In an online stage, the text to be processed is input to obtain the corresponding synthesized speech.
[0006] The encoder and decoder in the encoding-decoding architecture neural network are both implemented using four cascaded speaker identity feature control networks, each of which includes: an attention unit, a layer normalization unit, a gated recurrent unit, and a gated convolution unit, wherein: the attention unit is based on the input feature dimension X∈R E×T , the self-attention network processes the correlation of the input sequence and obtains the attention output h attn ∈R E×T The layer normalization unit performs layer normalization operation based on the information output by attention to obtain the feature frame h attn (t), the gated recurrent unit represents the speaker identity vector ω∈R E , decode the output of the attention unit frame by frame to obtain the feature frame hrnn (t)= After that, the decoded results of each frame are concatenated and the feature vector h is output. rnn ∈R E×T ; The gated convolution unit represents the speaker identity vector c∈∈R E , after expansion processing and attention output h attn ∈R E×T After convolution obtains the filtered features, the outputs of the gated recurrent unit and the gated convolutional unit are added together to output feature information containing the speaker identity representation vector at the speech level and frame level as the output of the speaker identity feature control network. Technical Effects
[0007] The present invention uses a dual-branch network to perform feature control on the frame level and speech level of the spectrum features of the generated speech, so that the generated speech sounds closer to the speaker's timbre of the input reference audio; based on gated convolution and gated recurrent units, respectively, it can better constrain the frame-level features of the generated speech spectrum while realizing feature constraints on the speech spectrum at the speech level. Compared with the prior art, the present invention requires only fewer migration samples. In the test phase, each speaker only needs 15 samples to replicate a person's voiceprint information; it has a faster migration speed, and fine-tuning 5,000 parameter weight updates on the pre-trained model can achieve the replication of a person's voice. In the actual application stage, it can generate the corresponding speaker's voice for any text input. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 It is a flow chart of the present invention;
[0009] Figure 2 A schematic diagram of a speaker identity feature control network in an embodiment;
[0010] Figure 3 A comparison diagram of the predicted spectrum and the actual spectrum of the embodiment;
[0011] In the figure: the upper figure is the spectrum diagram corresponding to the real speech, and the lower figure is the predicted spectrum diagram. DETAILED DESCRIPTION
[0012] like Figure 1 As shown, this embodiment involves a speech synthesis method based on speaker feature control, in which an encoding-decoding architecture neural network including a cascaded speaker identity feature control network is constructed in an offline stage, and reference audio after spectral encoding, text information after phoneme encoding, and encoded position information are used as training samples to perform pitch and intensity prediction training based on duration modeling; and in the online stage, the text to be processed is input to obtain the corresponding synthesized speech.
[0013] The spectrum encoding means that after processing the spectrum corresponding to the input reference audio in the spectrum and time series dimensions, the corresponding speaker identity representation vector is obtained by taking the average in the time series dimension after attention.
[0014] The phoneme encoding means that after the input text is normalized, it is processed by a character-to-phoneme tool to obtain a phoneme time sequence with a length of T.
[0015] The encoded position information refers to: the position of the input feature in the entire sequence represented by the position embedding encoding using the cosine function, specifically: and Among them: pos refers to the position of the word in the sequence, i refers to the i-th component of the position embedding code, and the dimension d of the position embedding code is model Set to 256.
[0016] The pitch and intensity prediction based on duration modeling refers to: calculating the fundamental frequency of the speech frame corresponding to the input speech as its pitch information, and calculating the L2 norm of the short-time Fourier transform of the speech frame corresponding to the input speech as its energy information; after realizing the correspondence between phonemes and speech through automatic annotation tools, taking the average of the pitch and energy of the speech frame corresponding to the phoneme as the pitch and intensity information corresponding to each phoneme.
[0017] The automatic annotation tool is implemented by, but not limited to, the open source Montreal Forced Aligner, and extracts the temporal correspondence between phonemes and speech through the pre-trained model and dictionary provided by the MFA official website.
[0018] The input speech is the reference audio in the training phase and the information predicted by the corresponding model in the online phase.
[0019] The encoding-decoding architecture neural network includes: an encoder and a decoder both implemented by four cascaded speaker identity feature control networks.
[0020] like Figure 2 As shown, each speaker identity feature control network includes: attention unit, layer normalization unit, gated recurrent unit, gated convolution unit, where: the attention unit is based on the input feature dimension X∈R E×T , the self-attention network processes the correlation of the input sequence and obtains the attention output h attn ∈R E×T The layer normalization unit performs layer normalization operation based on the information output by attention to obtain the feature frame h attn (t), the gated recurrent unit represents the speaker identity vector ω∈R E, decode the output of the attention unit frame by frame to obtain the feature frame After that, the decoded results of each frame are concatenated and the feature vector h is output. rnn ∈R E×T ; The gated convolution unit represents the speaker identity vector ω∈R E , after expansion processing and attention output h attn ∈R E×T After convolution obtains the filtered features, the outputs of the gated recurrent unit and the gated convolution unit are added together to output feature information containing the speaker identity representation vector at the speech level and the frame level as the output of the final speaker identity feature control network of the present invention.
[0021] The gated recurrent unit is implemented based on a gated recurrent unit (GRU).
[0022] The gated convolutional branch is implemented based on the gated convolutional network (Gated Convolutional Networks)
[0023] The extended processing is: the feature obtained according to the speaker identity representation vector ω is ω(CNN)∈R E×T , E is the vector dimension representing each feature, and T is the sequence length of the input feature.
[0024] The sequence in the encoder is at the phoneme level, and its sequence length is the phoneme length corresponding to the text; the sequence in the decoder is at the speech frame level, and its sequence length is the frame length of the corresponding speech.
[0025] The post-screening characteristics are specifically: h cnn =Sigmoid(Conv(h attn +ω))*conv(h attn ), where: conv is the convolution operation, activation function The input value is constrained to the range of 0 to 1 to achieve the screening of main road branch features.
[0026] The training specifically includes: using the normalized text and corresponding speech of the LibriTTS data set as training samples, and at the same time extracting the spectrum, fundamental frequency and energy information corresponding to the speech, as well as the alignment information of the corresponding speech after the input text is converted into phonemes. In the training stage, the predicted spectrum is decoded by the decoder to obtain the predicted corresponding duration, fundamental frequency and energy information. The predicted value and the true labeled value are calculated by the loss function and back propagated to realize the gradient update in the network training stage. The fundamental frequency prediction and energy prediction networks are obtained by superimposing multiple convolutional layers and normalization layers, and the predicted fundamental frequency or energy embedding representation is obtained by the linear layer in the last layer. The duration prediction network is obtained by superimposing multiple convolutional layers and normalization layers, and the predicted value is obtained by the linear layer in the last layer.
[0027] The loss function Loss = Loss mel +Losse nergy +Loss f0 +Loss dur ,in: and are the predicted spectrum, energy, fundamental frequency and duration information respectively; y mel ,y energy ,y f0 and dur They are the true labeled spectrum, energy, fundamental frequency and duration information respectively, MAE is the mean absolute error, and MSE is the mean square error.
[0028] After specific practical experiments, the experiment was carried out on an experimental test platform (CPU: Intel(R) Xeon(R) Gold 6226R CPU@2.90GHz, GPU: NVIDIA GeForce RTX 3090), using the Pytorch deep learning framework. The train-clean-360 subset of the LibriTTS public dataset was used as the dataset for the experiment of the present invention. In the inference stage, the HIFI-GAN vocoder was used to convert the predicted spectrogram into a speech waveform.
[0029] This embodiment uses four indicators for testing, including two subjective indicators and two objective indicators. The subjective indicators include Mean Opinion Score (MOS) and Similarity Mean Opinion Score (SMOS). The objective indicators include Mel-cepstral Distortion (MCD) and Speaker Encode Cosine Similarity (SECS). The objective indicators are statistically analyzed to obtain the mean of all results. The final test results are shown in Table 1.
[0030] Table 1 Test on LibriTTS public dataset
[0031] Compared with the existing technology, this method can achieve efficient speech synthesis quality with smaller transfer samples, and the number of iterations required for fine-tuning is also small.
[0032] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.
Claims
1. A speech synthesis method based on speaker feature control, characterized in that: In the offline stage, a speaker identity feature control network module is constructed, and a neural network based on an encoder-decoder architecture is constructed based on the speaker identity feature control network. The reference audio after spectrum encoding, the text information after phoneme encoding, and the encoded position information are used as training samples to perform pitch and intensity prediction training based on duration modeling. And after inputting the text to be processed in the online stage, the corresponding synthesized speech is obtained; The encoding-decoding architecture neural network includes: an encoder and a decoder both implemented by a cascaded speaker identity feature control network, each speaker identity feature control network includes: an attention unit, a layer normalization unit, a gated recurrent unit, and a gated convolution unit, wherein: the attention unit is based on the input feature dimension X∈R E×T , the self-attention network processes the correlation of the input sequence and obtains the attention output h attn ∈R E×T The layer normalization unit performs layer normalization operation based on the information output by attention to obtain the feature frame h attn (t), the gated recurrent unit represents the speaker identity vector ω∈R E , decode the output of the attention unit frame by frame to obtain the feature frame After that, the decoded results of each frame are concatenated and the feature vector h is output. rnn ∈R E×T ; The gated convolution unit represents the speaker identity vector ω∈R E , after expansion processing and attention output h attn ∈R E×T After convolution obtains the filtered features, the outputs of the gated recurrent unit and the gated convolutional unit are added together to output feature information containing the speaker identity representation vector at the speech level and frame level as the output of the speaker identity feature control network.
2. The method for speech synthesis based on speaker feature control according to claim 1, characterized in that: The spectrum encoding means: after processing the spectrum corresponding to the input reference audio in the spectrum and time series dimension, taking the average in the time series dimension after attention to obtain the corresponding speaker identity representation vector; The phoneme encoding means: after the input text is normalized, it is processed by a character-to-phoneme tool to obtain a phoneme time sequence; The encoded position information refers to: the position of the input feature in the entire sequence represented by the position embedding encoding using the cosine function, specifically: and Among them: pos refers to the position of the word in the sequence, i refers to the i-th component of the position embedding code, and the dimension d of the position embedding code is model Set to 256; The encoder and decoder implemented by the cascaded speaker identity feature control network refer to encoders and decoders implemented by four cascaded speaker identity feature control networks.
3. The method for speech synthesis based on speaker feature control according to claim 1, characterized in that: The pitch and intensity prediction based on duration modeling refers to: calculating the fundamental frequency of the speech frame corresponding to the input speech as its pitch information, and calculating the L2 norm of the short-time Fourier transform of the speech frame corresponding to the input speech as its energy information; after realizing the correspondence between phonemes and speech through automatic annotation tools, taking the average of the pitch and energy of the speech frame corresponding to the phoneme as the pitch and intensity information corresponding to each phoneme.
4. The method for speech synthesis based on speaker feature control according to claim 3, characterized in that: The automatic annotation tool is implemented using Montreal Forced Aligner, which extracts the temporal correspondence between phonemes and speech through the pre-trained model and dictionary provided by the MFA official website.
5. The method for speech synthesis based on speaker feature control according to claim 3, characterized in that: The input speech is the reference audio in the training phase and the information predicted by the corresponding model in the online phase.
6. The method for speech synthesis based on speaker feature control according to claim 1, characterized in that: The gated recurrent unit is implemented based on a gated recurrent unit (GRU); and the gated convolutional branch is implemented based on a gated convolutional network (Gated Convolutional Networks).
7. The method for speech synthesis based on speaker feature control according to claim 1, characterized in that: The extended processing is: the feature obtained according to the speaker identity representation vector ω is ω(CNN)∈R E×T , E is the vector dimension representing each feature, and T is the sequence length of the input feature.
8. The method for speech synthesis based on speaker feature control according to claim 1, characterized in that: The sequence in the encoder is at the phoneme level, and its sequence length is the phoneme length corresponding to the text; the sequence in the decoder is at the speech frame level, and its sequence length is the frame length of the corresponding speech.
9. The method for speech synthesis based on speaker feature control according to claim 1, characterized in that: The post-screening characteristics are specifically: h cnn =Sigmoid(conv(h attn +ω))*conv(h attn ), where: conv is the convolution operation, activation function 10. The method for speech synthesis based on speaker feature control according to claim 1, characterized in that: The training specifically includes: using the normalized text and corresponding speech of the LibriTTS data set as training samples, and at the same time extracting the spectrum, fundamental frequency and energy information corresponding to the speech, as well as the alignment information of the corresponding speech after the input text is converted into phonemes. In the training stage, the predicted spectrum is obtained by decoding through a decoder, as well as the predicted corresponding duration, fundamental frequency and energy information, and the predicted value and the true labeled value are calculated by loss function and back propagated to realize the gradient update of the network training stage, wherein the fundamental frequency prediction and energy prediction networks are obtained by superimposing multiple convolutional layers and normalization layers, and the predicted fundamental frequency or energy embedding representation is obtained by a linear layer in the last layer, wherein the duration prediction network is obtained by superimposing multiple convolutional layers and normalization layers, and the predicted value is obtained by a linear layer in the last layer; The loss function Loss = Loss mel +Loss energy +Loss f0 +Loss dur ,in: and are the predicted spectrum, energy, fundamental frequency and duration information respectively; y mel ,y energy ,y f0 and dur They are the true labeled spectrum, energy, fundamental frequency and duration information respectively, MAE is the mean absolute error, and MSE is the mean square error.
