End-to-end Kunqu synthesis system and method based on deep learning

Abstract

The invention provides an end-to-end Kunqu synthesis system and method based on deep learning, and the system comprises a music score feature extraction module which is used for extracting music score features according to an input Kunqu music score; the libretto text feature extraction module is used for extracting libretto text features from the input Kungqu libretto text; the feature alignment module is used for aligning the music score features with the text features based on a self-attention mechanism model, and combining pitch and rhythm information corresponding to the music score features with a pronunciation time sequence of the libretto text features; the acoustic parameter generation module is used for constructing an acoustic parameter prediction model by using the aligned music score features and libretto text features, and predicting key acoustic features according to the generated acoustic parameter prediction model; and the Kungqu voice generation module is used for converting the Kungqu voice into Kungqu audio according to the generated key acoustic features by a vocoder based on a depth generation model. Reduction of the rhythm of the kelp is enhanced, and the naturalness of the kelp is higher.

Description

Technical Field

[0001] The present invention relates to the technical field of Kunqu opera synthesis, and in particular to an end-to-end Kunqu opera synthesis system and method based on deep learning. Background Art

[0002] As a treasure of traditional Chinese opera culture, the ancient Kunqu Opera has a history of more than 600 years and is listed as a "Masterpiece of the Oral and Intangible Cultural Heritage of Humanity" by UNESCO. Kunqu Opera occupies an important position in Chinese opera with its beautiful singing, delicate performance art and unique tune system. However, the inheritance of Kunqu Opera faces challenges, especially in modern society, where the younger generation has limited understanding of traditional art, and the dissemination of opera art urgently needs to be innovated and expanded with the help of modern technology.

[0003] With the development of computer technology, although speech synthesis and singing synthesis have achieved industrial results, opera synthesis is still a topic that has not been widely explored. There is currently no research on Kunqu synthesis, and only a few studies on Peking opera synthesis. Researchers use a modular approach to decompose and process factors such as vocalization, pitch, rhythm, and rhyme one by one. Although this multi-modular system can generate opera audio, it is prone to error accumulation because each module is processed independently, fails to coordinate effectively, and cannot be jointly optimized, resulting in insufficient rhythm coherence in the final synthesis result, deviations in pitch, and difficulty in meeting the high requirements of the human ear in sound quality. Among these studies, there is currently a lack of research on the reconstruction of falsetto audio in opera performances, which has relatively lower sound intensity, frequency drift, and a more special resonance peak structure compared to ordinary audio.

[0004] For example, the paper "Beijing Opera Synthesis Based on Straight Algorithm and DeepLearning" uses the intonation control model and melody control model based on the Straight algorithm to synthesize Peking Opera clips, such as Figure 1 As shown in the figure, the input of the system is: ① target speech, A tone and A content; ② original speech carrying Peking Opera text, B tone and B content; ③ MIDI score information. The tone control model is responsible for converting the two input speech segments into a new target speech, A tone and B content. The new target speech and MIDI score information are used together as the melody control model, as shown in the figure. Figure 2As shown in the figure, the duration and fundamental frequency F0 of Peking Opera audio are finely controlled to synthesize Peking Opera segments, and finally the obtained segments are spliced ​​to obtain a complete synthesized Peking Opera. In addition to the error accumulation of multiple modules, the Peking Opera text in this method is not input into the model in text format, but is input as a number of original voices to facilitate the model to control the tone. It relies on a huge voice database and sophisticated splicing technology, which is prone to problems such as unnatural splicing and acoustic incoherence. Although the Straight algorithm can modify voice parameters such as fundamental frequency F0 and spectral envelope to improve the style flexibility of speech synthesis, the generated speech often lacks naturalness, sounds more mechanical, and has the problem of "robotic voice". Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes an end-to-end Kunqu synthesis system and method based on deep learning.

[0006] The technical solution of the present invention is achieved in this way:

[0007] On the one hand, an embodiment of the present invention provides an end-to-end Kunqu opera synthesis system based on deep learning, comprising:

[0008] A music score feature extraction module is used to extract music score features according to the input Kunqu music score through a long short-term memory network model, wherein the music score features include but are not limited to pitch, rhythm and duration;

[0009] A lyrics text feature extraction module is used to segment, annotate and encode the input Kunqu lyrics text to extract lyrics text features, wherein the lyrics text features include but are not limited to pronunciation and tone;

[0010] The feature alignment module is used to align the score features with the text features based on the self-attention mechanism model, and combine the pitch and rhythm information corresponding to the score features with the pronunciation timing of the lyrics text features;

[0011] An acoustic parameter generation module is used to construct an acoustic parameter prediction model using the music score features and lyrics text features after the above alignment processing, and predict key acoustic features according to the generated acoustic parameter prediction model, wherein the key acoustic features include but are not limited to sound intensity, spectrum, fundamental frequency and phoneme duration;

[0012] The Kunqu speech generation module is used to convert the key acoustic features generated by the vocoder based on the deep generative model into Kunqu audio.

[0013] Preferably, the lyrics text feature extraction module is also used to perform deep context modeling based on the input Kunqu lyrics text through the BERT pre-training model to capture the contextual association and phonetic information of words.

[0014] Preferably, it also includes an adaptive weight allocation module for dynamically adjusting the loss function weights of different subtasks in the score feature extraction module, the lyrics text feature extraction module, the feature alignment module, the acoustic parameter generation module and the Kunqu speech generation module.

[0015] Preferably, the specific adjustment process of the adaptive weight allocation module is: measuring the gap between the predicted value and the true value by calculating the loss function, and then calculating the gradient of the loss function to the model parameters through the back propagation algorithm; finally, dynamically adjusting the parameters of the model according to the direction and size of this gradient to gradually reduce the error.

[0016] Preferably, the acoustic parameter generation module includes:

[0017] The sound intensity prediction unit is used to construct a convolutional neural network based on pitch, rhythm, duration, and phoneme duration, and to predict the acoustic characteristics of sound intensity in combination with the self-attention mechanism;

[0018] A fundamental frequency prediction unit is used to construct a multi-layer autoregressive neural network model according to pitch, rhythm, duration and phoneme duration to predict fundamental frequency acoustic features;

[0019] The phoneme duration prediction unit is used to construct a bidirectional long short-term memory network model based on pitch, rhythm, duration and phoneme duration, and predict the acoustic characteristics of phoneme duration in combination with positional encoding technology;

[0020] The spectrum prediction unit is used to construct a generation model of Mel spectrum or linear spectrum according to pitch, rhythm, duration, phoneme duration and predicted sound intensity, fundamental frequency and phoneme duration to predict spectrum acoustic features.

[0021] Preferably, it also includes a falsetto parameter prediction module, which is used to predict falsetto related features based on music score features, lyrics text features and key acoustic features;

[0022] The Kunqu voice generation module is used to convert the generated key acoustic features and false voice related features into Kunqu audio based on the vocoder of the deep generative model.

[0023] Preferably, before the Kunqu voice generation module performs Kunqu voice synthesis, the false voice spectrum is enhanced through a spectrum enhancement network model.

[0024] Preferably, it also includes a recitation synthesis module, which is used to synthesize the recitation audio based on the input recitation text and the preset reference Kunqu audio.

[0025] Preferably, the speech synthesis module includes:

[0026] A discrete token extraction unit, used to extract discrete tokens of a preset reference Kunqu audio through a pre-trained neural encoder;

[0027] A feature generation unit, used for inputting the discrete tokens and the spoken text processed into a phoneme sequence into a speech feature extractor and a text feature extractor respectively, to obtain reference audio features and reference text features;

[0028] The spoken audio generation unit is used to input reference audio features, reference text features and input spoken text into the language model to predict the complete discrete tokens of the spoken audio to be synthesized; finally, the discrete token sequence of the spoken audio to be synthesized is input into the decoding end of the neural encoder to generate the spoken audio.

[0029] On the other hand, an embodiment of the present invention further provides an end-to-end Kunqu opera synthesis method based on deep learning, comprising the following steps:

[0030] The long short-term memory network model extracts music score features according to the input Kunqu music score, wherein the music score features include but are not limited to pitch, rhythm and duration;

[0031] Segmenting, annotating and encoding the input Kunqu opera lyrics text to extract lyrics text features, wherein the lyrics text features include but are not limited to pronunciation and tone;

[0032] Based on the self-attention mechanism model, the score features are aligned with the text features, and the pitch and rhythm information corresponding to the score features are combined with the pronunciation timing of the lyrics text features;

[0033] Using the music score features and lyrics text features after the above alignment process to build a prediction model of acoustic parameters, and generating key acoustic features according to the generated acoustic parameters, the key acoustic features include but are not limited to high pitch, sound intensity, spectrum, fundamental frequency and phoneme duration;

[0034] The deep generative model-based vocoder is converted into Kunqu audio based on the generated key acoustic features.

[0035] Compared with the prior art, the present invention has the following advantages: the end-to-end Kunqu synthesis system based on deep learning of the present invention includes a score feature extraction module, a lyrics text feature extraction module, a feature alignment module, an acoustic parameter generation module and a Kunqu voice generation module, wherein the score features and the lyrics text features are respectively extracted by the score feature extraction module and the lyrics text feature extraction module, and then the Kunqu audio is finally generated by the Kunqu voice generation module after a series of processing, and this multimodal fusion effectively enhances the restoration of Kunqu rhythm, can better capture the complex rhythmic features of Kunqu, and ensure that the synthesized voice audio not only matches the lyrics text in pronunciation, but also is highly consistent with the score in melody and rhythm, thereby achieving a more realistic artistic expression. The present invention realizes end-to-end Kunqu audio synthesis, from score and text input to final audio output, the whole process does not require manual intervention; compared with the traditional synthesis method that requires multiple steps of manual adjustment, the present invention can automatically generate complete Kunqu audio, greatly improving the synthesis efficiency. Moreover, the present invention adopts a multi-task integrated framework, integrating music score feature learning, lyrics text analysis and acoustic parameter generation into one model for joint optimization. Compared with the traditional modular method, the present invention can uniformly process multiple tasks within a single framework, reducing the error accumulation between modules; through the joint training strategy, the naturalness and consistency of the generated audio and the ability to restore the rhythm of Kunqu Opera are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 It is a structural diagram of a multi-module Peking opera synthesis system in the prior art;

[0038] Figure 2 It is a melody control model structure in the prior art;

[0039] Figure 3 It is a structural block diagram of the end-to-end Kunqu opera synthesis system based on deep learning of the present invention;

[0040] Figure 4 This is a flow chart of the end-to-end Kunqu opera synthesis method based on deep learning of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 3 The embodiment of the present invention discloses an end-to-end Kunqu opera synthesis system based on deep learning, including:

[0043] A music score feature extraction module is used to extract music score features according to the input Kunqu music score through a long short-term memory network model, wherein the music score features include but are not limited to pitch, rhythm and duration;

[0044] A lyrics text feature extraction module is used to segment, annotate and encode the input Kunqu lyrics text to extract lyrics text features, wherein the lyrics text features include but are not limited to pronunciation and tone;

[0045] The feature alignment module is used to align the score features with the text features based on the self-attention mechanism model, and combine the pitch and rhythm information corresponding to the score features with the pronunciation timing of the lyrics text features;

[0046] An acoustic parameter generation module is used to construct an acoustic parameter prediction model using the music score features and lyrics text features after the above alignment processing, and predict key acoustic features according to the generated acoustic parameter prediction model, wherein the key acoustic features include but are not limited to sound intensity, spectrum, fundamental frequency and phoneme duration;

[0047] The Kunqu speech generation module is used to convert the key acoustic features generated by the vocoder based on the deep generative model into Kunqu audio.

[0048] In the embodiment of the present invention, under a single framework established by an end-to-end Kunqu synthesis system based on deep learning, several core tasks, namely, learning of score features and lyrics text features, acoustic feature modeling and Kunqu audio reconstruction, are integrated. Through the joint optimization training of these core tasks, the system of the present invention can perform end-to-end learning on the entire process from score, text to final Kunqu audio, significantly reducing the error accumulation problem in traditional multi-module methods, improving the quality and naturalness of synthesized speech, and achieving significant breakthroughs in the accurate restoration of Kunqu rhythm and falsetto artistic expression.

[0049] In the music score feature extraction module, the long short-term memory network model extracts music score features such as pitch, rhythm, duration and other time series information from the input Kunqu music score. The long short-term memory network (LSTM) model can extract features from these time series information and can effectively capture key elements such as pitch changes and rhythm patterns. The model further generates rhythmic features, which represent the melody and rhythm direction in Kunqu opera.

[0050] In the lyrics text feature extraction module, a preset Kunqu phoneme dictionary (such as the Kunqu phoneme dictionary based on "Yunxue Lizhu") can be used to segment, annotate and encode the input Kunqu lyrics text to extract lyrics text features such as pronunciation and intonation.

[0051] There are three types of word segmentation methods: 1. Rule-based word segmentation: This can be done by matching dictionaries. For example, word segmentation tools such as Jieba can be used to segment words based on the vocabulary of the Kunqu phoneme dictionary. 2. Statistical and machine learning methods: such as CRF (Conditional Random Fields), which combines contextual information and word frequency statistics for word segmentation. 3. Deep learning models: BERT, GPT and other Transformer-based models can also be used for word segmentation tasks.

[0052] For annotation, existing speech synthesis models (such as FastSpeech, VITS, etc.) directly model the input Kunqu opera lyrics text when processing the input phonemes, including the annotation task.

[0053] Encoding is the process of converting the input text sequence into a numerical representation that the model can process. Common methods include: 1. One-Hot Encoding: Encode characters, words or phonemes independently to generate sparse vector representations. 2. Embedding: Through word embedding models (Word2Vec, FastText, BERT Embeddings), words or phonemes are converted into low-dimensional dense vectors to retain their semantic information.

[0054] In the present invention, the music score features extracted by the music score feature extraction module and the lyrics text features extracted by the lyrics text feature extraction module can accurately reflect the music structure and artistic expression of Kunqu Opera, and also combine the language expression of the lyrics. Through the deep learning network, the present invention automatically abstracts and learns multimodal features such as pitch, rhythm, duration, pronunciation, and tone, laying a solid foundation for subsequent acoustic modeling and generation. Moreover, through the subsequent fine modeling of the rhythm of Kunqu Opera music scores, the system can better express the sense of hierarchy and artistic expression in the melody, ensuring that the final synthesis result is both faithful to the Kunqu Opera tradition and has a high degree of naturalness and rhythm.

[0055] Furthermore, the lyrics text feature extraction module is also used to perform deep context modeling based on the input Kunqu lyrics text through the BERT pre-trained model to capture the contextual association and phonological information of words. In this embodiment, in order to fully capture the semantic and pronunciation characteristics of the lyrics, the BERT pre-trained model is further used. Through its bidirectional Transformer architecture, deep context modeling can be performed on the input text to capture the contextual association and phonological information of words, providing rich semantic and pronunciation information for the subsequent acoustic parameter generation module.

[0056] After obtaining the above-mentioned multimodal features such as score features and lyrics text features, the score features are aligned with the text features based on the self-attention mechanism model (Self-Attention) through the feature alignment module, that is, the pitch and rhythm information corresponding to the score features are combined with the pronunciation timing of the lyrics text features. Among them, the self-attention mechanism model can capture long-range dependencies in a longer input sequence, and is particularly suitable for the task of aligning the text and the score. In this embodiment, this alignment enables the input multimodal information to remain consistent in the downstream acoustic model, thereby avoiding information loss and conflict, and ensuring that the synthesized speech can accurately capture the musical rhythm and pronunciation characteristics of Kunqu Opera.

[0057] Specifically, the specific method for aligning the score features with the text features is as follows:

[0058] First, the correlation between pitch, rhythm and pronunciation timing is learned simultaneously through a multi-head attention mechanism model, so that the model can capture the association between different features in a global context, and then generate an attention weight matrix used to represent the correlation between the score features and the lyrics text features; the higher the weight of the attention weight matrix, the stronger the correlation between the features, and the model will give priority to these features with strong correlation; finally, a weighted combination of the corresponding pitch, rhythm and text pronunciation features in the attention weight matrix is ​​performed to achieve synchronous alignment of different features.

[0059] In the acoustic parameter generation module, the acoustic parameter prediction model is constructed using the score features and lyrics text features after the above alignment processing, and then the key acoustic features are predicted based on the generated acoustic parameter prediction model; and as for the acoustic parameter prediction model, key acoustic features such as spectrum, fundamental frequency and phoneme duration can be generated based on the non-autoregressive model. Specifically, the DiffSinger model can be used as an embodiment. DiffSinger combines FastSpeech2 and diffusion decoding mechanism to generate high-quality key acoustic features. As a non-autoregressive model, FastSpeech2 can improve the efficiency of speech generation and reduce the inference delay problem existing in traditional autoregressive models. The diffusion decoding mechanism gradually optimizes the acoustic features, effectively restores high-frequency details, and generates more diverse and natural audio results. Compared with traditional autoregressive models, DiffSinger has stronger generation capabilities, especially in complex music contexts, and can generate high-fidelity audio.

[0060] In this embodiment, the acoustic parameter generation module not only ensures that the generated acoustic features can accurately match the performance of the lyrics and music, but also uses these acoustic features to model the unique rhythmic structure of Kunqu Opera, which can capture the ups and downs and rhythm changes in the melody. The system combines the multimodal information of the lyrics text features and the music score features to ensure that the synthesized sound not only meets the timbre and pitch, but also shows the unique beauty of Kunqu Opera rhythm.

[0061] In the Kunqu speech generation module, the generated key acoustic features are input into the vocoder based on the deep generative model, which converts these key acoustic features into high-fidelity Kunqu audio.

[0062] Among them, the contextual association and phonetic information of the words captured by the lyrics text feature extraction module are also input into the vocoder of the deep generative model, so that the vocoder can generate more high-fidelity Kunqu audio.

[0063] Furthermore, the end-to-end Kunqu synthesis system based on deep learning in an embodiment of the present invention also includes an adaptive weight allocation module, which is used to dynamically adjust the loss function weights of different subtasks in the music score feature extraction module, the lyrics text feature extraction module, the feature alignment module, the acoustic parameter generation module and the Kunqu speech generation module.

[0064] Specifically, the adaptive weight allocation module randomly and dynamically adjusts the value of the loss function weight according to the gap between the predicted synthesized audio and the corresponding real audio in the training data set; in deep learning, the dynamic adjustment of the loss function weight is mainly achieved through the gradient descent algorithm. Specifically, the model measures the gap (i.e., error) between the predicted value and the real value by calculating the loss function, and then calculates the gradient of the loss function to the model parameters through the back propagation algorithm; finally, according to the direction and size of this gradient, the model parameters are dynamically adjusted to gradually reduce the error.

[0065] In this embodiment, due to the use of multi-task learning, such as the extraction of pitch, rhythm and duration features in the score features, the extraction of pronunciation and tone features in the lyrics text features, etc., are performed synchronously as subtasks. In the training process of these models involved, the loss function weights of different subtasks can be dynamically adjusted, thereby improving the training efficiency and synthesis quality of the model. In addition, the loss function weights of the training models in the feature alignment module, the acoustic parameter generation module, and the Kunqu voice generation module can also be dynamically adjusted.

[0066] The present invention adopts this multi-task integrated framework, so that the system can simultaneously process multiple target tasks, including phoneme duration prediction, fundamental frequency prediction and spectrum generation. Each task module processes different acoustic features respectively. Since the adaptive weight allocation module can dynamically adjust the weights of the loss functions of different subtasks, the mutual interference between tasks can be reduced through this joint optimization strategy. For example, the accurate prediction of phoneme duration can assist the spectrum generation module to more accurately restore the rhythmic features in Kunqu Opera; this end-to-end training reduces the error propagation problem in the traditional multi-module method of speech synthesis, so that the whole system maintains a high degree of coherence during the generation process.

[0067] Specifically, the acoustic parameter generation module includes:

[0068] The sound intensity prediction unit is used to construct a convolutional neural network based on pitch, rhythm, duration, and phoneme duration, and combine it with the self-attention mechanism to predict the acoustic characteristics of sound intensity. This structure can effectively capture the complex changes in sound intensity in Kunqu performances, especially in the performance techniques of long-tone continuation, crescendo and decrescendo. The network can achieve fine dynamic control of sound intensity through correlation modeling on time series.

[0069] The fundamental frequency prediction unit is used to construct a multi-layer autoregressive neural network model based on pitch, rhythm, duration, and phoneme duration to predict fundamental frequency acoustic features; the constructed multi-layer autoregressive neural network model adopts a non-autoregressive structure similar to FastSpeech2 to improve the real-time and stability of the prediction. By introducing pitch dynamic range adjustment, it can adapt to the pitch requirements of different roles in Kunqu Opera.

[0070] The phoneme duration prediction unit is used to construct a bidirectional long short-term memory network model according to pitch, rhythm, duration and phoneme duration, and predict the acoustic characteristics of phoneme duration in combination with position coding technology; in this embodiment, the bidirectional long short-term memory network model combined with position coding technology can capture the unique rhythm and rhythm information of Kunqu Opera; especially for the transition between falsetto and traditional vocal styles, the network can ensure the natural transition and emotional expression of the synthesized audio through flexible control of duration.

[0071] The spectrum prediction unit is used to construct a generation model of Mel spectrum or linear spectrum according to pitch, rhythm, duration, phoneme duration and predicted sound intensity, fundamental frequency and phoneme duration to predict the spectrum acoustic features. In this embodiment, a multi-layer convolutional network (Conformer) based on a self-attention mechanism can also be used to improve the accuracy of spectrum prediction. This embodiment can effectively integrate long-term context information by adopting a generation model of Mel spectrum or linear spectrum, accurately reproduce the complex timbre changes in Kunqu performance, and reflect the unique frequency band performance of falsetto in the spectrum.

[0072] Furthermore, the end-to-end Kunqu opera synthesis system based on deep learning in the embodiment of the present invention further includes a falsetto parameter prediction module, which is used to predict falsetto related features based on music score features, lyrics text features, and key acoustic features;

[0073] The Kunqu voice generation module is used to convert the generated key acoustic features and false voice related features into Kunqu audio based on the vocoder of the deep generative model.

[0074] In this embodiment, considering that falsetto is often used in Kunqu Opera to simulate female characters or express emotional changes, and has a unique timbre and vocalization method, the Kunqu Opera synthesis system of the present invention introduces a falsetto parameter prediction module in the reconstruction stage to accurately capture and reproduce the delicate timbre and acoustic characteristics of falsetto art, ensuring that the final synthesis result can faithfully present the unique artistic style of Kunqu Opera.

[0075] After the falsetto-related features are predicted by the falsetto parameter prediction module, the Kunqu voice generation module converts the generated key acoustic features and falsetto-related features into Kunqu audio.

[0076] Specifically, in the embodiment of the present invention, the prediction of false voice is based on the comprehensive analysis of the music score, text and acoustic features by the model, and specifically includes the following steps:

[0077] Step 1, extracting music score features related to falsetto: Extracting features such as pitch, rhythm, and duration based on the input Kunqu music score, especially the high-range part related to falsetto. Falsetto usually corresponds to a higher pitch (range), so the pitch information in the music score is an important basis for predicting falsetto.

[0078] Step 2: Analyze the text features of the lyrics related to falsetto: Through word segmentation and annotation, extract features from the lyrics text, and focus on analyzing the use of falsetto in specific passages or pronunciations in Kunqu Opera. Certain specific sentence patterns or pronunciation methods (such as soft and weak timbre) often use falsetto. By analyzing the semantic information of the lyrics text, it can be predicted whether falsetto should be used.

[0079] Step 3, capturing key acoustic features related to falsetto: Falsetto is usually accompanied by changes in timbre, such as the vibration mode of the vocal cords is different from natural voice, showing weaker resonance and more high-frequency components. Therefore, the present invention learns the key acoustic features of falsetto (such as changes in formant peaks, frequency band energy distribution, etc.) from training data, so that the model can better capture the acoustic characteristics of falsetto.

[0080] After obtaining the falsetto-related features through the above steps, the adaptive weight allocation module will combine the score features and the lyrics text features, paying special attention to the high pitch part and the text or speech patterns related to the falsetto. The model will give higher weights to the falsetto-related features to ensure that the acoustic performance of the falsetto is fully considered when generating audio.

[0081] After predicting the false voice related features, the audio synthesis process further processes the false voice features and combines them with the synthesis of Kunqu audio. The specific processing flow is as follows:

[0082] Pitch and duration adjustment: Based on the pitch and rhythm information in the score, the Kunqu voice generation module will make appropriate adjustments to the pitch when generating falsetto according to the unique high-range characteristics of falsetto. At the same time, the duration of falsetto is usually prolonged and the timbre will appear weaker, so this feature will also be taken into account in the processing of rhythm.

[0083] Timbre processing: The timbre of falsetto is significantly different from normal voice. For this reason, the Kunqu voice generation module also adjusts the spectral features during synthesis, such as weakening low-frequency resonance and enhancing high-frequency parts to make the audio sound lighter and weaker. You can achieve these timbre changes by adjusting the frequency band processing in the vocoder model.

[0084] Dynamic Control: Falsetto usually requires more precise dynamic control, which is reflected in weaker sound pressure and fading volume. By adjusting the volume and dynamic range of the model output, ensure that the falsetto passages have a clear contrast with the normal passages.

[0085] It can be seen that under the above multi-task learning framework, the model can optimize the generation of pronunciation and rhythm at the same time. The autoregressive generation process (such as using diffusion decoding in DiffSinger) can further ensure the smooth generation of false voice and avoid abrupt transitions in the audio.

[0086] Therefore, the above falsetto-related features are predicted based on multimodal inputs such as score features (high pitch and duration), lyrics text features (falsetto-related text patterns), and acoustic features (timbre and formant changes), combined with the self-attention mechanism for feature fusion. During falsetto synthesis, the system will make targeted adjustments in pitch, duration, timbre, etc. to ensure that the falsetto part of the generated audio is significantly different from natural voice in timbre and acoustic features, while maintaining smooth connection.

[0087] The embodiment of the present invention does not distinguish between ordinary voice segments and false voice generation tasks, so that the model can simultaneously process multiple acoustic tasks in an end-to-end framework, which not only reduces the error propagation between modules, but also ensures that the generated false voice and natural voice have a smooth connection by adaptively adjusting the transition area between false voice and natural voice. When the Kunqu voice generation module converts the generated key acoustic features and false voice related features into Kunqu audio, it can use the self-attention mechanism to dynamically adjust the generation strategy to ensure that the model can naturally handle the conversion between false voice and real voice.

[0088] In addition, the present invention can also use multi-resolution STFT (Short-Time Fourier Transform) to capture the detailed changes of the false voice in the frequency domain, especially the enhancement processing of the high-frequency part. This process can ensure the clarity and energy of the false voice audio.

[0089] The multi-resolution spectrum analysis process is as follows:

[0090] Defining the falsetto characteristic spectrum: The falsetto segment is characterized by weaker vocal cord vibration and higher pitch. This way of voicing results in a reduction in low-frequency components and an enhancement of high-frequency components on the spectrum. Through multi-resolution spectrum analysis, the spectral characteristics of the falsetto segment can be accurately captured at different frequency resolutions, especially the details of the high-frequency component.

[0091] Capturing high-frequency details: Low-resolution spectrum cannot capture high-frequency changes in falsetto segments well, while high-resolution spectrum can refine the timbre changes in falsetto. Therefore, the embodiment of the present invention adopts multi-resolution spectrum analysis, which can not only capture the overall structure of the audio in the full frequency band, but also accurately depict the subtle timbre changes of falsetto segments, making the spectrum of falsetto more refined.

[0092] Segmented spectrum analysis: Falsetto segments are significantly different from ordinary segments in timbre. Therefore, multi-resolution spectrum analysis can be used to locally refine and model the falsetto segments according to their time boundaries to avoid sound quality deviations in falsetto segments during the audio generation process.

[0093] When synthesizing Kunqu audio, specific acoustic modeling is performed on the falsetto segment, mainly performing the following processing:

[0094] Spectrum enhancement processing: In the falsetto segment, by enhancing the details of the high frequency part and appropriately weakening the low frequency resonance, the soft and weak characteristics of the falsetto can be better restored. The high frequency part after multi-resolution analysis will be input into a specific modeling module to ensure that the frequency characteristics of the falsetto segment can match the real singing.

[0095] Time domain adjustment: Since the vocal cords vibrate less, the time domain performance of the falsetto segment is usually smoother and has less strong energy changes. Through time domain modeling, the waveform of the falsetto segment is smoothed to eliminate possible noise and distortion, making the generated falsetto more natural and smooth.

[0096] When generating Kunqu audio, the following optimizations are also performed through a specific vocoder:

[0097] Spectrum-waveform conversion for falsetto: The main function of the vocoder is to convert the spectrum into an audio waveform. For falsetto segments, the vocoder needs to focus on processing high-frequency components and maintain the natural smoothness of the audio. In the spectrum input of the falsetto segment, the vocoder uses an optimized decoding strategy to ensure that high-frequency details are preserved while reducing distortion in waveform generation.

[0098] Dynamic adjustment of falsetto segments: Falsetto usually has significant differences in volume and timbre from normal voice. Therefore, in the vocoder, specific dynamic adjustments are made to the falsetto segments (such as reducing the volume and enhancing the softness of the high-frequency part) to accurately restore the timbre and singing characteristics of the falsetto segments.

[0099] In summary, the present invention adopts a multi-task learning strategy to collaboratively optimize the pitch, timbre, and duration of falsetto segments. Through iterative training, the model can continuously improve its ability to capture and generate falsetto segments. In each round of iteration, the generation error of falsetto segments will be gradually reduced, while improving the model's similarity between falsetto segments and real singing.

[0100] The present invention uses multi-resolution spectrum analysis to finely model falsetto segments, and combines a specific vocoder to optimize the falsetto segments, making the falsetto audio generation more realistic and natural. This optimization greatly improves the synthesis quality of the falsetto singing part, making the generated audio more similar to the real singing.

[0101] In addition, before the Kunqu voice generation module performs Kunqu voice synthesis, the falsetto spectrum is enhanced by the spectrum enhancement network model, including correction of frequency components and smoothing of energy distribution, to ensure that the falsetto audio is consistent with the real singing in terms of overall auditory perception. In this embodiment, by training the spectrum enhancement network, the detail expression of the falsetto is further improved, especially the weak sound processing of the falsetto and the refinement of the resonance peak, so that the synthesized audio is close to the real falsetto singing in quality.

[0102] Furthermore, the end-to-end Kunqu opera synthesis system based on deep learning in the embodiment of the present invention also includes a recitation synthesis module, which is used to synthesize recitation audio based on the input recitation text and preset reference Kunqu opera audio.

[0103] In this embodiment, in view of the fact that the pronunciation characteristics of Kunqu opera recitation are different from daily spoken language, this embodiment can first rely on expert knowledge to construct a pronunciation vocabulary that combines the simplified pinyin and tone attributes of singing and recitation. This vocabulary is designed specifically for the synthesis of Kunqu opera recitation and will serve as a core component for converting the recitation text into a format that the system can recognize. Through this processing process, we can effectively convert the input recitation text into the basic data for synthesizing Kunqu opera recitation.

[0104] In order to carefully learn and model the pronunciation features, speaking style, timbre and other elements of the characters in the preset reference Kunqu audio, the embodiment of the present invention needs to synthesize the spoken audio in combination with the preset reference Kunqu audio and the input spoken text. The embodiment of the present invention can synthesize audio that retains the original timbre and singing style of the characters in the reference Kunqu audio and has a high sense of naturalness; and the embodiment of the present invention can realize spoken synthesis using only one sentence, which greatly reduces the application threshold, and the audio synthesized by the model has the characteristics of high naturalness.

[0105] Specifically, the speech synthesis module includes:

[0106] A discrete token extraction unit, used to extract discrete tokens of a preset reference audio through a pre-trained neural encoder;

[0107] A feature generation unit, used for inputting discrete tokens of a preset reference audio and the spoken text processed into a phoneme sequence into a speech feature extractor and a text feature extractor respectively, to obtain reference audio features and reference text features;

[0108] The spoken audio generation unit is used to input reference audio features, reference text features and input spoken text into the language model to predict the complete discrete tokens of the spoken audio to be synthesized; finally, the discrete token sequence of the spoken audio to be synthesized is input into the decoding end of the neural encoder to generate the spoken audio.

[0109] In view of the scarcity of Kunqu opera speech data and the high requirements of the Transformer-based speech synthesis system for the amount of training data, the present invention innovatively regards the synthesis of Kunqu opera speech as a multilingual speech synthesis problem. In this framework, Kunqu opera speech is regarded as a unique "language". When training the language model, we introduced the language label parameter, so that non-Kunqu opera speech data can be used in the training process, effectively alleviating the problem of insufficient Kunqu opera data, and thus improving the overall performance and expressiveness of the speech synthesis module.

[0110] The embodiment of the present invention can realize the speech synthesis of Kunqu opera speech that is highly consistent with the timbre and style of the original characters by only using the speech reference audio of a small number of characters. The synthesized speech is not only extremely natural, but also fully retains the artistic expression of Kunqu opera. This method can greatly reduce the threshold for the general public and Kunqu opera lovers to create Kunqu opera speech works, and stimulate the public's interest in learning Kunqu opera.

[0111] On the other hand, Figure 2 As shown, the embodiment of the present invention also provides an end-to-end Kunqu synthesis method based on deep learning, which can realize the specific process of the above end-to-end Kunqu synthesis system based on deep learning, including the following steps:

[0112] S1, the long short-term memory network model extracts music score features according to the input Kunqu music score, the music score features including but not limited to pitch, rhythm and duration;

[0113] S2, segmenting, annotating and encoding the input Kunqu opera lyrics text to extract lyrics text features, wherein the lyrics text features include but are not limited to pronunciation and tone;

[0114] S3, aligns the score features with the text features based on the self-attention mechanism model, and combines the pitch and rhythm information corresponding to the score features with the pronunciation timing of the lyrics text features;

[0115] S4, constructing a prediction model of acoustic parameters using the music score features and lyrics text features after the above alignment processing, and generating key acoustic features according to the generated acoustic parameters, wherein the key acoustic features include but are not limited to high pitch, sound intensity, spectrum, fundamental frequency and phoneme duration;

[0116] S5, the deep generative model-based vocoder is converted into Kunqu audio according to the generated key acoustic features.

[0117] In summary, the end-to-end Kunqu synthesis system based on deep learning of the present invention adopts multimodal fusion to effectively enhance the restoration of Kunqu rhythm, and can better capture the complex rhythmic characteristics of Kunqu, ensuring that the synthesized speech audio not only matches the lyrics text in pronunciation, but also is highly consistent with the score in melody and rhythm, thereby achieving a more realistic artistic expression.

[0118] In addition, the present invention realizes end-to-end Kunqu audio synthesis, from score and text input to final audio output, and the entire process does not require human intervention; compared with traditional synthesis methods that require multiple steps of manual adjustment, the present invention can automatically generate complete Kunqu audio, greatly improving the synthesis efficiency.

[0119] Moreover, the present invention adopts a multi-task integrated framework, integrating music score feature learning, lyrics text analysis and acoustic parameter generation into one model for joint optimization. Compared with the traditional modular method, the present invention can uniformly process multiple tasks within a single framework, reducing the error accumulation between modules; through the joint training strategy, the naturalness and consistency of the generated audio and the ability to restore the rhythm of Kunqu Opera are effectively improved.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An end-to-end Kunqu opera synthesis system based on deep learning, characterized in that: include: A music score feature extraction module is used to extract music score features according to the input Kunqu music score through a long short-term memory network model, wherein the music score features include but are not limited to pitch, rhythm and duration; A lyrics text feature extraction module is used to segment, annotate and encode the input Kunqu lyrics text to extract lyrics text features, wherein the lyrics text features include but are not limited to pronunciation and tone; The feature alignment module is used to align the score features with the text features based on the self-attention mechanism model, and combine the pitch and rhythm information corresponding to the score features with the pronunciation timing of the lyrics text features; An acoustic parameter generation module is used to construct an acoustic parameter prediction model using the music score features and lyrics text features after the above alignment processing, and predict key acoustic features according to the generated acoustic parameter prediction model, wherein the key acoustic features include but are not limited to sound intensity, spectrum, fundamental frequency and phoneme duration; The Kunqu speech generation module is used to convert the key acoustic features generated by the vocoder based on the deep generative model into Kunqu audio.

2. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 1 is characterized in that: The lyrics text feature extraction module is also used to perform deep context modeling based on the input Kunqu lyrics text through the BERT pre-training model to capture the contextual association and phonetic information of words.

3. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 1 is characterized in that: It also includes an adaptive weight allocation module for dynamically adjusting the loss function weights of different subtasks in the score feature extraction module, the lyrics text feature extraction module, the feature alignment module, the acoustic parameter generation module, and the Kunqu speech generation module.

4. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 3 is characterized in that: The specific adjustment process of the adaptive weight allocation module is as follows: the gap between the predicted value and the true value is measured by calculating the loss function, and then the gradient of the loss function to the model parameters is calculated through the back propagation algorithm; finally, according to the direction and size of this gradient, the parameters of the model are dynamically adjusted to gradually reduce the error.

5. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 1 is characterized in that: The acoustic parameter generation module comprises: The sound intensity prediction unit is used to construct a convolutional neural network based on pitch, rhythm, duration, and phoneme duration, and to predict the acoustic characteristics of sound intensity in combination with the self-attention mechanism; A fundamental frequency prediction unit is used to construct a multi-layer autoregressive neural network model according to pitch, rhythm, duration and phoneme duration to predict fundamental frequency acoustic features; The phoneme duration prediction unit is used to construct a bidirectional long short-term memory network model based on pitch, rhythm, duration and phoneme duration, and predict the acoustic characteristics of phoneme duration in combination with positional encoding technology; The spectrum prediction unit is used to construct a generation model of Mel spectrum or linear spectrum according to pitch, rhythm, duration, phoneme duration and predicted sound intensity, fundamental frequency and phoneme duration to predict spectrum acoustic features.

6. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 1 is characterized in that: It also includes a falsetto parameter prediction module, which is used to predict falsetto related features based on music score features, lyrics text features and key acoustic features; The Kunqu voice generation module is used to convert the generated key acoustic features and false voice related features into Kunqu audio based on the vocoder of the deep generative model.

7. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 6 is characterized in that: Before the Kunqu speech generation module performs Kunqu speech synthesis, the false voice spectrum is enhanced through the spectrum enhancement network model.

8. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 1 is characterized in that: It also includes a recitation synthesis module, which is used to synthesize the recitation audio based on the input recitation text and the preset reference Kunqu audio.

9. The end-to-end Kunqu opera synthesis system based on deep learning according to claim 8 is characterized in that: The speech synthesis module comprises: A discrete token extraction unit, used to extract discrete tokens of a preset reference Kunqu audio through a pre-trained neural encoder; A feature generation unit, used for inputting the discrete tokens and the spoken text processed into a phoneme sequence into a speech feature extractor and a text feature extractor respectively, to obtain reference audio features and reference text features; The spoken audio generation unit is used to input reference audio features, reference text features and input spoken text into the language model to predict the complete discrete tokens of the spoken audio to be synthesized; finally, the discrete token sequence of the spoken audio to be synthesized is input into the decoding end of the neural encoder to generate the spoken audio.

10. An end-to-end Kunqu opera synthesis method based on deep learning, characterized in that: The following steps are involved: The long short-term memory network model extracts score features according to the input Kunqu score, wherein the score features include but are not limited to pitch, rhythm and duration; Segmenting, annotating and encoding the input Kunqu opera lyrics text to extract lyrics text features, wherein the lyrics text features include but are not limited to pronunciation and tone; Based on the self-attention mechanism model, the score features are aligned with the text features, and the pitch and rhythm information corresponding to the score features are combined with the pronunciation timing of the lyrics text features; Using the music score features and lyrics text features after the above alignment process to build a prediction model of acoustic parameters, and generating key acoustic features according to the generated acoustic parameters, the key acoustic features include but are not limited to high pitch, sound intensity, spectrum, fundamental frequency and phoneme duration; The deep generative model-based vocoder is converted into Kunqu audio based on the generated key acoustic features.

Images