Method for synthesizing singing cavity

Abstract

The invention discloses a singing play cavity synthesis method, relates to the field of singing play cavity synthesis, solves the problems that data is deficient due to the lack of annotation data in existing singing play synthesis, and local feature modeling of data by a voice synthesis model and a singing sound synthesis model is insufficient, and provides the following scheme. The method comprises the following steps of: sampling hidden representation similar to the output characteristics of an encoder from a codebook of EnCodec by utilizing the strong capability of the neural audio codec EnCodec, and reconstructing the hidden representation into audio, so that a song play data set is successfully expanded, and more data support is provided for model training; according to the method, the FG-GAN model is proposed, a GConformer module is proposed, and a grouping depth separable convolution module is proposed on the basis of a Conformer architecture, so that the receptive field of the model to a spectrogram is expanded, and the capture capability of the model to local features is enhanced. In addition, the FG-GAN also provides a frame-level implicit representation modeling module based on a stream model, so as to better simulate the change of audio in a phoneme or pitch.

Description

Technical Field

[0001] The invention relates to the field of Taiwanese opera singing synthesis, and in particular to a Taiwanese opera singing synthesis method. Background Art

[0002] Taiwanese opera has a long history. It originated in the southern Fujian region during the Song and Yuan dynasties, and was later introduced to Taiwan and other regions, where it was developed and inherited. It is known as a type of "Southern Fujian opera". After hundreds of years of evolution and development, Taiwanese opera has integrated elements such as music, singing, dance, drama, and rap, forming a unique performance style and artistic characteristics. Taiwanese opera is usually performed on stage, with one or more actors playing various roles. The actors express the plot through singing, reciting, acting, and fighting, and are accompanied by exquisite costumes, props, and scenery, as well as specific musical accompaniment. During the performance, the actors will use special lines and singing methods, such as reciting, singing lyrics, and chanting, to create a unique charm and atmosphere of opera.

[0003] The Taiwanese opera carries the traditional culture of Fujian Province in China, integrating the artistic elements of different ethnic groups such as Fujian, Hakka, and Han people. The Taiwanese opera originated in Fujian Province, so the cultural elements of Fujian occupy an important position in it. Fujian's traditional music, opera, singing methods, etc. have had a profound impact on the formation and development of the Taiwanese opera. In Fujian Province, the Hakka people are an important ethnic group, and their cultural elements are also deeply integrated into the Taiwanese opera. For example, in some Taiwanese opera repertoires, the characteristics of Hakka culture, such as dialects, costumes, customs, etc., can be seen. As the main ethnic group in southern China, the Han culture also occupies an important position in the Taiwanese opera. The traditional music, dance, and opera performance methods of the Han nationality have provided a wealth of expression forms and artistic elements for the Taiwanese opera. With the changes of history, foreign cultures have gradually been integrated into the Taiwanese opera, including some music, dance, and drama forms of ethnic minorities, enriching the artistic connotation and expression forms of the Taiwanese opera. As an excellent folk art form, the Taiwanese opera was initially performed by folk artists, and later gradually became a performance form for professional actors, but it still retains a strong folk color.

[0004] Gezaixi synthesis is to generate natural and smooth opera singing voices through technical means according to given lyrics and music score information. In recent years, speech synthesis technology based on deep learning has developed rapidly. This technological opportunity has made it possible to digitally inherit and innovate the cultural treasure of Gezaixi. Similarly, as a derivative task of speech synthesis, the rapid development of singing synthesis has laid a solid foundation for achieving high-quality Gezaixi synthesis. Existing singing synthesis technology can synthesize realistic speech when there is sufficient annotated data, but simply migrating the singing synthesis model and method to Gezaixi generation cannot achieve good results. There are two main reasons: On the one hand, Gezaixi synthesis requires a large amount of training corpus, and the cost of obtaining high-quality Gezaixi audio and music score information is high, which limits the development of Gezaixi synthesis technology. On the other hand, since Gezaixi involves complex acoustic characteristics and musical performance, how to achieve natural and realistic singing synthesis is still a difficult problem.

[0005] The research on the synthesis of Taiwanese opera can effectively reduce the labor cost of the arrangement and innovation of Taiwanese opera, so as to better inherit and innovate the cultural treasure of Taiwanese opera in today's information and digital era. Therefore, the opera synthesis model based on the characteristics of Taiwanese opera has high research value and practical significance. Most of the existing Taiwanese opera generation models follow the model framework of speech synthesis or singing synthesis, but the effect of generating opera is unsatisfactory. There are the following main reasons:

[0006] First, there is a lack of annotated data. The existing Gezi Opera dataset GOAT (Gezi Opera Audio-Text alignment dataset) has a total length of only 4.5 hours and contains 5 speakers. The more mature speech synthesis models, such as Fastspeech2, are built on the LJSpeech dataset, which is a single speaker dataset of about 24 hours. The singing synthesis dataset popcs contains 6 hours of single speaker data. In comparison, Gezi Opera data is relatively scarce;

[0007] Second, the speech synthesis model and the singing synthesis model do not adequately model the local features of the data. The current opera synthesis model is often modified from the speech synthesis model or the singing synthesis model. Compared with opera, the pitch changes of singing and speech are relatively gentle, and the duration of phonemes is not much different. Therefore, the speech synthesis and singing synthesis models can synthesize good audio even if they do not model the local features separately. Opera is more expressive, its pitch changes are more diverse than speech synthesis or singing synthesis, and the duration of phonemes varies greatly. For example, in the audio of Taiwanese opera, sometimes a vowel may last for a considerable time (for example, a few seconds). In this case, the vowel may not just be pronounced like itself, and some similar but different vowels may be pronounced before or after it. This is reflected in the model level that we should pay more attention to the local correlation of the data, and the implicit representation should be modeled at a finer granularity than the phoneme level in the alignment module of phonemes and spectrograms. Therefore, a method for synthesizing the singing of Taiwanese opera is proposed. Summary of the invention

[0008] The purpose of the present invention is to provide a method for synthesizing the singing style of a Taiwanese opera, which solves the problems in the existing Taiwanese opera synthesis, such as lack of labeled data, resulting in data scarcity, and insufficient modeling of local features of the data by a speech synthesis model and a singing synthesis model.

[0009] To achieve the above object, the present invention provides the following technical solution: a method for synthesizing the singing style of Taiwanese opera, comprising the following steps:

[0010] S1000: Data enhancement method based on neural audio compression model EnCodec:

[0011] (1) Adopting the neural audio compression model EnCodec with the architecture of encoder-quantizer-decoder, the encoder compresses the input audio into a latent representation, i.e., a vector;

[0012] (2) The residual vector quantization module is used to find the eight most similar vectors in the eight codebooks, and then the proposed topk method and residual topk method are used to sample vectors similar to the original audio from the codebook, add them together, and use this vector to reconstruct the audio as augmented data through the decoder;

[0013] (3) This reconstructed audio is then compared to the original audio using a discriminator component. The discriminator measures the numerical difference between the two, known as the discriminator / generator loss.

[0014] (4) The discriminator / generator loss involves comparing the reconstructed waveform and mel-spectrogram with the original waveform and mel-spectrogram, and evaluating the loss to stabilize the encoder's output, ultimately ensuring that the audio output closely reflects the initial input;

[0015] S2000: Establishment of the synthesis model of Taiwanese opera singing:

[0016] (1) The FG-GAN architecture of the opera generation model is proposed. The input of the architecture model is (T, S, D, P), where T represents the phoneme sequence, S represents the speaker identity, D represents the duration of the phoneme, and P represents the pitch sequence;

[0017] (2) The GConformer module is proposed, which modifies the depthwise separable convolution in the Conformer into a grouped depthwise separable convolution. The information of adjacent frames is taken into account. At the same time, one convolution kernel of the depthwise separable convolution is responsible for one channel, and one channel is convolved by only one convolution kernel. It is modified into a grouped convolution. In this way, the information of different channels at the same spatial position can be directly modeled within the group, that is, the local information of adjacent channels in adjacent frames, thereby expanding the receptive field of the depthwise separable convolution within the same Conformer block to better reconstruct the local details in the mel-spectrogram.

[0018] (3) The discriminator in the opera generation model FTGAN architecture is used for modeling on the discriminator side, and the modified Conformer is used as the generator side;

[0019] (4) Migrate the flow model in VITS and combine it with the grouped deep separable convolutional blocks for frame-level latent representation modeling to enhance its expressiveness.

[0020] Preferably, the encoder model E in S1000 is composed of a 1D convolution with C channels and B convolution blocks with a kernel size of 7, each of which is composed of a residual unit and a downsampling layer, and the downsampling layer is composed of a strided convolution, and its kernel size K is twice the stride s;

[0021] The residual unit contains two convolutions with a kernel size of 3 and a skip connection, and the number of channels is doubled whenever downsampling occurs;

[0022] The convolutional block is followed by a two-layer LSTM for sequence modeling and a final 1D convolutional layer with a kernel size of 7 and D output channels;

[0023] The EnCodec uses C=32, B=4 and (2,4,5,8) as the step and uses ELU as the nonlinear activation function, or layer normalization or weight normalization.

[0024] The decoder is the inverse process of the encoder, using transposed convolution instead of strided convolution and in the opposite order of striding in the encoder, outputting the final mono or stereo audio.

[0025] Preferably, the residual vector quantization in S1000 includes projecting the input vector onto the nearest entry in a codebook of a given size, and the residual vector quantization improves this process by calculating the quantized residual and further quantizing it using a second codebook, and so on.

[0026] Preferably, the EnCodec described in S1000 has been trained with a large amount of data, and its codebook stores sufficiently rich fine-grained implicit representations. Using the above representations, two sampling methods are proposed for data enhancement, including the topk method and the residual topk method. The topk method samples k vectors in the residual vector quantization that are close to the original vector in the eight codebooks and are used to reconstruct the audio. The residual topk method samples k vectors in the residual vector quantization that are close to the residual post-vector in the eight codebooks and are used to reconstruct the audio.

[0027] Preferably, the GConformer described in S2000 is improved based on Conformer, and the improvement lies in that the depthwise separable convolution in Conformer is modified to a grouped depthwise separable convolution, the information of adjacent frames is taken into account, and at the same time, one convolution kernel of the depthwise separable convolution is responsible for one channel, and one channel is convolved by only one convolution kernel, which is modified to a grouped convolution to better reconstruct the local details in the mel-spectrogram.

[0028] Preferably, the Conformer includes a forward propagation block, a multi-head self-attention module and a convolution block.

[0029] Compared with the related art, the method for synthesizing the singing style of Taiwanese opera provided by the present invention has the following beneficial effects:

[0030] First, in response to the scarcity of the Gezaixi opera dataset, this paper uses the powerful capabilities of the neural audio codec EnCodec to propose two novel data augmentation methods, namely the topk method and the residual topk method. By sampling latent representations similar to the encoder output features from the EnCodec codebook and reconstructing them into audio, we successfully expanded the Gezaixi opera dataset and provided more data support for model training. Experimental results show that both data augmentation methods can effectively improve the synthesis effect of the opera synthesis model.

[0031] Secondly, in order to address the problem that the singing synthesis model does not adequately model the local features of the data, this paper proposes a Taiwanese opera synthesis model FG-GAN. This model is improved on the basis of the Conformer architecture. By introducing a grouped deep separable convolution module, the model's receptive field of the spectrogram is expanded, thereby enhancing the model's ability to capture local features. In addition, FG-GAN also proposes a frame-level latent representation modeling module based on a flow model to better fit the changes in audio within the same phoneme or pitch. Experimental results show that FG-GAN has achieved significant improvements in the continuity and naturalness of synthesized audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the EnCodec model diagram of the present invention.

[0033] Figure 2 This is a quantizer model diagram of the present invention.

[0034] Figure 3 This is a model diagram of the residual vector quantizer of the present invention.

[0035] Figure 4 The mel-spectrograms of the present invention are compressed and reconstructed using EnCodec. The upper row is the real audio mel-spectrograms, and the lower row is the reconstructed audio mel-spectrograms.

[0036] Figure 5 This is the top-k algorithm diagram of the present invention.

[0037] Figure 6 This is the residual top-k algorithm diagram of the present invention.

[0038] Figure 7 The right side is the FG-GAN model diagram of the present invention, and the left side is the GConformer model diagram of the present invention.

[0039] Figure 8 This is the structural diagram of Conformer and ConformerBlock of the present invention.

[0040] Fig. 9 This is a diagram of the depth-separable convolution structure in the Conformer of the present invention.

[0041] Fig.10 This is the point-by-point convolution principle of the present invention.

[0042] Fig.11 This is the depth convolution principle of the present invention.

[0043] Fig.12 This is the GConformer module diagram of the present invention, in which the convolution kernel sizes of the two 1d convolutions are both 3.

[0044] Fig.13 Improved graph of grouped depthwise separable convolutions according to the present invention.

[0045] Fig.14 This is a frame-level implicit representation modeling module diagram based on the flow model of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] See also Figure 1-14 The present invention provides a technical solution: a method for synthesizing the singing style of a Taiwanese opera, comprising the following steps:

[0048] S1000: Data enhancement method based on neural audio compression model EnCodec:

[0049] (1) Adopting the neural audio compression model EnCodec with the architecture of encoder-quantizer-decoder, the encoder compresses the input audio into a latent representation, i.e., a vector;

[0050] (2) The residual vector quantization module is used to find the eight most similar vectors in the eight codebooks, and then the proposed topk method and residual topk method are used to sample vectors similar to the original audio from the codebook, add them together, and use this vector to reconstruct the audio as augmented data through the decoder;

[0051] (3) This reconstructed audio is then compared to the original audio using a discriminator component. The discriminator measures the numerical difference between the two, known as the discriminator / generator loss.

[0052] (4) The discriminator / generator loss involves comparing the reconstructed waveform and mel-spectrogram with the original waveform and mel-spectrogram, and evaluating a commitment loss to stabilize the encoder’s output, ultimately ensuring that the audio output closely reflects the initial input;

[0053] S2000: Establishment of the synthesis model of Taiwanese opera singing:

[0054] (1) The FG-GAN architecture of the opera generation model is proposed. The input of the architecture model is (T, S, D, P), where T represents the factor sequence, S represents the speaker identity, D represents the duration of the phoneme, and P represents the pitch sequence;

[0055] (2) The depthwise separable convolution in the Conformer is modified to a grouped depthwise separable convolution, which takes into account the information of adjacent frames. At the same time, one convolution kernel of the depthwise convolution is responsible for one channel, and one channel is convolved by only one convolution kernel. It is modified to a grouped convolution. In this way, the information of different channels at the same spatial position can be directly modeled within the group, that is, the local information of adjacent channels in adjacent frames, thereby expanding the receptive field of the depthwise separable convolution in the same Conformer block to better reconstruct the local details in the mel-spectrogram.

[0056] (3) The discriminator in the opera generation model FTGAN architecture is used for modeling on the discriminator side, and GConformer is used as the generator side;

[0057] (4) Migrate the flow model in VITS and combine it with the grouped deep separable convolutional blocks for frame-level latent representation modeling to enhance its expressiveness.

[0058] The encoder model E in S1000 consists of a 1D convolution with C channels and B convolution blocks with a kernel size of 7. Each convolution block consists of a residual unit and a downsampling layer. The downsampling layer consists of a strided convolution whose kernel size K is twice the stride s.

[0059] The residual unit contains two convolutions with a kernel size of 3 and a skip connection, and the number of channels is doubled whenever downsampling occurs;

[0060] The convolutional block is followed by a two-layer LSTM for sequence modeling and a final 1D convolutional layer with a kernel size of 7 and D output channels;

[0061] EnCodec uses C=32, B=4 and (2,4,5,8) as the step and ELU as the nonlinear activation function, or layer normalization or weight normalization.

[0062] The decoder is the inverse of the encoder, using transposed convolutions instead of strided convolutions and in the opposite order of strides in the encoder, outputting the final mono or stereo audio.

[0063] Residual vector quantization in S1000 involves projecting the input vector onto the nearest entry in a codebook of a given size. Residual vector quantization improves this process by computing the quantized residual and further quantizing it using a second codebook, and so on.

[0064] The EnCodec in S1000 has been trained with a large amount of data, and its codebook stores sufficiently rich and fine-grained latent representations. Using the above representations, two sampling methods are proposed for data enhancement, including the topk method and the residual topk method. The topk method samples k vectors in the residual vector quantization that are close to the original vector in the eight codebooks to reconstruct the audio, and the residual topk method samples k vectors in the residual vector quantization that are close to the residual post-vector in the eight codebooks to reconstruct the audio.

[0065] GConformer in S2000 is improved based on Conformer. The improvement lies in that the depthwise separable convolution in Conformer is modified to grouped depthwise separable convolution, which takes into account the information of adjacent frames. At the same time, one convolution kernel of the depthwise separable convolution is responsible for one channel, and one channel is convolved by only one convolution kernel, which is modified to grouped convolution to better reconstruct the local details in the Mel-spectrogram.

[0066] GConformer consists of a forward propagation block, a multi-head self-attention module, and a convolution block, where the outputs of the two forward propagation blocks are multiplied by 1 / 2.

[0067] Multi-head self-attention block: First, a multi-head self-attention module (MHSA) from Transformer-XL is applied, specifically a relative sinusoidal positional encoding, which allows the self-attention module to perform better on variable-length inputs, and the resulting encoder is more robust to changes in speech length. Convolutional block: Before the convolutional module, a gating mechanism consisting of point-by-point convolution and linear gated unit (GLU) is added, followed by a one-dimensional depth-separable convolution, and then an atchnorm is added to help train deeper models. The entire convolutional module manual is attached. Fig. 9 As shown: Forward propagation block: The Transformer model has a feedforward module (FeedForwardModule) consisting of two layers of linear transformation and a nonlinear activation function after the MHSA layer. In the entire feedforward module, a residual connection is added after normalization. ConformerBlock: As mentioned earlier, the GConformer module contains two forward propagation blocks, with a multi-head self-attention module and a convolution block in between.

[0068] Grouped Depthwise Separable Convolution: GConformer uses grouped depthwise separable convolution in its convolutional blocks. The principle description is attached. Fig.13 As shown in Figure 1. Depthwise separable convolution can be divided into two steps: point-by-point convolution and depth-wise convolution. The principle description is attached. Fig.10 , 11 shown.

[0069] The advantages and disadvantages of depthwise separable convolution and grouped depthwise separable convolution are analyzed as follows: First, the advantage of depthwise separable convolution is that it can reduce parameters. Depthwise separable convolution reduces the required parameters compared to ordinary convolution. At the same time, it realizes the separation of channels and regions. Depthwise separable convolution changes the previous ordinary convolution operation that considers channels and regions at the same time to: first consider only regions, and then consider channels. This facilitates the modeling of some specific scenarios. Its disadvantages are also obvious. Pointwise convolution only considers the information of all channels at the current point, and does not model the context information. The number of convolution kernels in the depthwise convolution layer remains the same as the number of channels in the previous layer. Convolution operations are performed independently on each channel of the input layer, and the feature information of different channels at the same spatial position is not effectively utilized. Depthwise separable convolution combines the two. Although it can realize the communication of contextual information and information between channels, its efficiency is not high. The grouped depthwise separable convolution is modified into a grouped convolution, so that the information of different channels at the same spatial position can be directly modeled within the group, that is, the local information of adjacent channels in adjacent frames, thereby expanding the receptive field of the depthwise separable convolution within the same GConformer block and better reconstructing the local details in the Mel-spectrogram.

[0070] Data augmentation experiments:

[0071] This section uses the two data augmentation methods proposed in this paper (topk method, residual topk method) to expand the data in the GOAT dataset and the Peking Opera dataset by 0 times (topl, 2...), pre-train with augmented data for 40,000 steps, and then train with real data for 110,000 steps. The experimental results of the data augmentation experiment of the Taiwanese opera are shown in Table 1. It can be seen that in terms of pronunciation scores, the performance of the model after data augmentation is basically the same as that of the augmented model, but in terms of naturalness and expressiveness scores, the performance of the model after data augmentation is further improved (the pronunciation score, naturalness and expressiveness after the topk method are enhanced are 0.01, 0.09, and 0.09 better than before the enhancement, while the pronunciation score, naturalness and expressiveness after the residual top-k method are enhanced are -0.03, 0.04, and 0.07 better than before the enhancement), which is consistent with the prediction of the experiment in this paper. By listening to the synthesized audio, it can be found that the voice tone generated by the model after data augmentation is more natural, and the connection between different phonemes is smoother. Both the residual topk and topk methods can clearly hear that the audio has become more natural, but the data enhanced by the residual topk method is not so accurate when the k value becomes higher. This may be because the data introduced into the dataset by the residual topk method deviates more from the original pronunciation compared to topk, but this method still plays a positive role in synthesis (the pronunciation accuracy is slightly reduced, while the naturalness and expressiveness are improved). The data enhanced by the topk method introduces more noise in the auditory sense compared to the residual topk method, but its final contribution to the Taiwanese opera synthesis model is better than the residual topk method. This article speculates that this may be because the enhanced data introduced by the topk method is closer to the real data distribution. Such experimental results are also in line with the expectations of this article for the two sampling strategies. The experimental results on the Peking Opera dataset are shown in Table 2, and the conclusions are similar to the data enhancement experiments on the Taiwanese opera dataset.

[0072] At the same time, this paper found that the model pre-trained using the data augmentation method proposed in this paper has higher Lulin resistance to unknown inputs. For example, when the user inputs a pitch sequence that does not conform to the rhythm rules of the opera, the synthesized audio has less noise and pronunciation errors, and sounds more natural and smooth.

[0073] In summary, through the data augmentation experiments in this section, it is shown that the data augmentation method proposed in this paper enriches the data diversity and can effectively improve the synthesis effect of the opera synthesis model.

[0074] Table 1 MOS score results with 95% confidence interval on the GOAT dataset

[0075]

[0076] Table 2 MOS score results with 95% confidence interval on the Peking Opera dataset

[0077]

[0078] Taiwanese opera synthesis experiment:

[0079] Compare models:

[0080] FastSpeech2: Although FastSpeech2 is a speech synthesis model, it can achieve good results for singing speech synthesis tasks. We use the same parameter settings as in this paper, and the encoder and decoder are 4-layer 8-head feed-forward transformers with hidden size of 256. The model is trained for 150,000 steps using L1 loss in a mini-batch size of 9.6k audio frames.

[0081] Diffsinger: Diffsinger is a singing voice synthesis model based on a diffusion model, which performs well in Chinese singing speech synthesis tasks. It mainly consists of FastSpeech2 and a noise estimation model. For the FastSpeech2 part, we use the same parameter settings as above. The noise estimation model is a 20-layer Wavenet, which is consistent with the model in the code. The model was trained for 150k steps with a mini-batch size of 9.6k audio frames.

[0082] Visinger2: Vsinger2 is an improvement on the speech synthesis model VITS. It introduces DDSPI to generate audio to improve performance. This article uses the official open source code to reproduce the model. We modified the data processing part and trained a model with a small batch size of 8 on the GOAT dataset for 150k steps.

[0083] DurIAN: DurIAN is a Peking Opera synthesis model built by Wu et al. based on the DurIAN model. In order to solve the problem that there is a large deviation between the real pitch of the audio and the pitch of the score in the data, which affects the model training, Wu et al. proposed a melody transcription method to generate a virtual score to replace the original song as the input of the model. In addition, they added a Lagrange multiplier constrained mixture density network (MDN) to the duration predictor of the model to improve the accuracy of phoneme duration estimation.

[0084] FT-GAN: FT-GAN is a model designed specifically for the synthesis of Taiwanese opera. It introduces Conformer and adversarial training based on FastSpeech2. In order to generate higher quality spectrograms, the discriminator of PatchGAN is introduced. To address the problem of large duration differences in Taiwanese opera, the multi-length GAN (ML-GAN) of HifiSinger is introduced. It uses a set of discriminators to process mel spectrograms of different lengths. Inspired by the sub-frequency GAN (SF-GAN) in HifiSinger, FT-gan designed an improved SF-GAN to simulate mel spectrograms in the frequency domain to improve the fullness of the pitch range in the generated audio and make the audio more expressive.

[0085] The experimental method is the same as the data enhancement experiment.

[0086] Table 3 MOS score results with 95% confidence interval on the GOAT dataset

[0087]

[0088] The experimental results are shown in Table 3. The pronunciation scores of the audio generated by the proposed model are basically the same as those of the strong baselines VISInger2 and FT-gan, while the naturalness and expressiveness of the synthesized speech are better than the strong baselines (0.1 and 0.07 higher than the strongest baseline FT-gan, respectively). FG-GAN and all baselines use FFT blocks as phoneme encoders, so there is almost no difference in pronunciation scores. However, due to the frame-level latent variable modeling module and grouped deep separable convolution proposed in this paper, the naturalness and expressiveness of the synthesized audio are significantly improved. In summary, through the experiments on the synthesis of the Taiwanese opera in this section, it is not only demonstrated that the existing methods are insufficient in modeling frequent pitch changes and duration changes, but also that the Taiwanese opera synthesis model proposed in this paper is more suitable for generating audio with large pitch changes and duration changes such as opera, which further demonstrates the effectiveness of FG-GAN in the task of generating Taiwanese opera.

Claims

1. A method for synthesizing Taiwanese opera singing, characterized in that: The following steps are involved: S1000: Data enhancement method based on neural audio compression model EnCodec: (1) Adopting the neural audio compression model EnCodec with the architecture of encoder-quantizer-decoder, the encoder compresses the input audio into a latent representation, i.e., a vector; (2) The residual vector quantization module is used to find the eight most similar vectors in the eight codebooks, and then the proposed topk method and residual topk method are used to sample vectors similar to the original audio from the codebook, add them together, and use this vector to reconstruct the audio as augmented data through the decoder; (3) This reconstructed audio is then compared to the original audio using a discriminator component. The discriminator measures the numerical difference between the two, known as the discriminator / generator loss. (4) The discriminator / generator loss involves comparing the reconstructed waveform and mel-spectrogram with the original waveform and mel-spectrogram, and evaluating the loss to stabilize the encoder's output, ultimately ensuring that the audio output closely reflects the initial input; S2000: Establishment of the synthesis model of Taiwanese opera singing: (1) The FG-GAN architecture of the opera generation model is proposed. The input of the architecture model is (T, S, D, P), where T represents the phoneme sequence, S represents the speaker identity, D represents the duration of the phoneme, and P represents the pitch sequence; (2) The GConformer module is proposed. By modifying the depthwise separable convolution in the Conformer into a grouped depthwise separable convolution, the information of adjacent frames is taken into account. At the same time, one convolution kernel of the depthwise separable convolution is responsible for one channel, and one channel is convolved by only one convolution kernel. It is modified into a grouped convolution to better reconstruct the local details in the mel-spectrogram. (3) The opera generation model FT-GAN architecture is used for modeling on the discriminator side, and the modified Conformer is used as the generator side; (4) Migrate the flow model in VITS and combine it with the grouped deep separable convolutional blocks for frame-level latent representation modeling to enhance its expressiveness.

2. A method for synthesizing Taiwanese opera singing according to claim 1, characterized in that: The encoder model E in S1000 is composed of a 1D convolution with C channels and B convolution blocks with a kernel size of 7, each of which is composed of a residual unit and a downsampling layer, and the downsampling layer is composed of a strided convolution, whose kernel size K is twice the stride s; The residual unit contains two convolutions with a kernel size of 3 and a skip connection, and the number of channels is doubled whenever downsampling occurs; The convolutional block is followed by a two-layer LSTM for sequence modeling and a final 1D convolutional layer with a kernel size of 7 and D output channels; The EnCodec uses C=32, B=4 and (2,4,5,8) as the step and uses ELU as the nonlinear activation function, or layer normalization or weight normalization. The decoder is the inverse process of the encoder, using transposed convolution instead of strided convolution and in the opposite order of striding in the encoder, outputting the final mono or stereo audio.

3. A method for synthesizing Taiwanese opera singing according to claim 2, characterized in that: The residual vector quantization in S1000 includes projecting the input vector onto the nearest entry in a codebook of a given size, and the residual vector quantization improves this process by calculating the quantized residual and further quantizing it using a second codebook, and so on.

4. A method for synthesizing Taiwanese opera singing according to claim 3, characterized in that: The EnCodec described in S1000 has been trained with a large amount of data, and its codebook stores sufficiently rich and fine-grained latent representations. Using the above representations, two sampling methods are proposed for data enhancement, including the topk method and the residual topk method. The topk method samples k vectors in the residual vector quantization that are close to the original vector in the eight codebooks to reconstruct the audio, and the residual topk method samples k vectors in the residual vector quantization that are close to the residual post-vector in the eight codebooks to reconstruct the audio.

5. A method for synthesizing Taiwanese opera singing according to claim 1, characterized in that: The GConformer described in S2000 is improved based on Conformer. The improvement lies in that the depthwise separable convolution in Conformer is modified to grouped depthwise separable convolution, which takes into account the information of adjacent frames. At the same time, one convolution kernel of the depthwise separable convolution is responsible for one channel, and one channel is convolved by only one convolution kernel, which is modified to grouped convolution to better reconstruct the local details in the mel-spectrogram.

6. A method for synthesizing Taiwanese opera singing according to claim 5, characterized in that: The Conformer includes a forward propagation block, a multi-head self-attention module and a convolution block.

Images