A model construction method and device for Song Dynasty music restoration

Abstract

The present invention discloses a model construction method and device for restoring Song Ci music. The construction method includes: constructing a Song Ci music dataset; constructing a Song Ci music restoration model, including: a lyrics-to-melody generation module for generating a melody based on an input Song Ci; a melody-to-accompaniment generation module for generating an accompaniment based on the melody; a melody-to-accompaniment generation module for generating an accompaniment based on the melody; and a merging restoration module for merging the melody, a cappella, and accompaniment to generate a complete musical work; and training the constructed Song Ci music restoration model using the Song Ci music dataset to obtain a trained Song Ci music restoration model. Compared with the existing technology, the present invention achieves high-quality restoration of Song Ci music.

Description

Technical Field

[0001] The present invention belongs to the field of music generation, and in particular relates to a model construction method and device for restoring Song Dynasty music. Background Art

[0002] The field of music generation has made significant progress in recent years, with significant research focused on generating singing voices (References 1 and 2) and melody generation (References 3 through 5). While deep learning holds promise for recreating Song Dynasty music, no open-source datasets of ancient Chinese Song Dynasty music are currently available. Existing Chinese music datasets, such as M4Singer (Reference 6) and Opencpop (Reference 7), primarily focus on pop music. Most existing models capable of generating complete music with a cappella and instrumental accompaniment are commercial products, such as Suno and SkyMusic. These models typically employ an end-to-end architecture and leverage the powerful generative capabilities of large language models (LLMs) (Reference 8) to generate complete audio based on user-provided text prompts and lyrics. Furthermore, users can input audio as style references, which the model then uses to generate music in the corresponding style. However, the generation process of these systems is often uncontrollable and highly stochastic. Because these models are primarily trained on contemporary pop songs, the music they generate often reflects mainstream styles, making it difficult to generate music that aligns with the style of Song Dynasty music. In summary, due to the lack of a dedicated Song Ci music dataset and an open-source complete music generation model, restoring Song Ci music based on deep learning still faces huge challenges.

[0003] Singing Voice Synthesis (SVS) is a technique that uses information from musical scores (such as lyrics, rhythm, and pitch) to generate natural and expressive singing voices (Reference 9). In recent years, deep learning has made significant progress in the field of SVS, achieving significant improvements over traditional methods. Early deep learning-based systems employed feedforward neural networks (FFNNs) (Reference 10) to predict acoustic features directly from musical scores, outperforming systems based on hidden Markov models (HMMs) (Reference 11). Subsequently, the introduction of long short-term memory (LSTM) networks (Reference 12) and convolutional neural networks (CNNs) (Reference 13) enhanced the ability to model long-term dependencies and acoustic features in singing voices. Generative adversarial networks (GANs) (References 14 and 15) have also been incorporated into SVS systems to alleviate the oversmoothing problem, thereby achieving more natural and expressive singing voice synthesis. In addition, state-of-the-art deep learning architectures, such as transformer-based models (Ref. 16) (e.g. XiaoiceSingv (Ref. 17) and HifiSinger), as well as diffusion-based models (Ref. 18) such as DiffSinger, further improved the quality of SVS.

[0004] Lyric-to-melody generation is a key task in automatic music composition, involving the generation of a melody that matches a given lyric. It typically uses end-to-end models to generate melodies directly from lyrics. Bao et al. (Ref. 19) and Yu et al. (Ref. 3) used sequence-to-sequence models to generate melodies from lyrics. However, these end-to-end models require a large amount of paired lyric and melody data, which is difficult and expensive to obtain. To address this issue, Sheng et al. (Ref. 20) avoided the reliance on paired data by training lyric-to-lyric and melody-to-melody models separately and interacting them in a subsequent stage. However, because unpaired data is not fully utilized when learning the correlation between lyrics and melody, the consistency of lyric and melody features cannot be ensured. Ju et al. (Ref. 21) proposed TeleMelody, a two-stage generative model: lyrics-to-template and template-to-melody. The template bridges the gap between lyrics and melody, promoting better feature alignment and improving the controllability of the generated melody.

[0005] To sum up, how to improve the existing music generation model so that it can better restore Song Ci music and generate high-quality Song Ci music that conforms to a specific style has become an urgent problem to be solved.

[0006] [Document 1] Liu J, Li C, Ren Y, et al. Diffsinger: Singing voices synthesis via shallow diffusion mechanism[C] / / Proceedings of the AAAIconference on artificial intelligence. 2022, 36(10): 11020-11028.

[0007] [Document 2] Chen J, Tan X, Luan J, et al. Hifisinger: Towards high-fidelity neural singing voice synthesis[J]. arXiv preprint arXiv:2009.01776,2020.

[0008] [Literature 3] Yu Y, Srivastava A, Canales S. Conditional LSTM-GAN formelody generation from lyrics[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2021, 17(1): 1-20.

[0009] [Literature 4] Zhang C, Chang L, Wu S, et al. Relyme: improving lyric-to-melody generation by incorporating lyric-melody relationships[C] / / Proceedings of the 30th ACM International Conference on Multimedia. 2022: 1047-1056.

[0010] [Reference 5] Lv A, Tan X, Qin T, et al. Re-creation of creations: A new paradigm for lyric-to-melody generation[J]. arXiv preprint arXiv:2208.05697, 2022。

[0011] [Reference 6] Zhang L, Li R, Wang S, et al. M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus[J]. Advances in Neural Information Processing Systems, 2022, 35: 6914-6926。

[0012] [Reference 7] Wang Y, Wang X, Zhu P, et al. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis[J]. arXiv preprint arXiv:2201.07429, 2022。

[0013] [Reference 8] Zhao W X, Zhou K, Li J, et al. A survey of large language models[J]. arXiv preprint arXiv:2303.18223, 2023。

[0014] [Reference 9] Yamamoto R, Song E, Kim J M. Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram[C] / / ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 6199-6203。

[0015] [Reference 10] Nishimura M, Hashimoto K, Oura K, et al. Singing Voice Synthesis Based on Deep Neural Networks[C] / / Interspeech. 2016: 2478-2482。

[0016] [Reference 11] Rabiner L R. A tutorial on hidden Markov models and selected applications in speech recognition[J]. Proceedings of the IEEE, 1989, 77(2): 257-286。

[0017] [Reference 12] Kim J, Choi H, Park J, et al. Korean singing voice synthesis system based on an LSTM recurrent neural network[C] / / Proc. Interspeech. 2018: 1551-1555。

[0018] [Reference 13] Nakamura K, Takaki S, Hashimoto K, et al. Fast and high-quality singing voice synthesis system based on convolutional neural networks[C] / / ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 7239-7243。

[0019] [Reference 14] Hono Y, Hashimoto K, Oura K, et al. Singing voice synthesis based on generative adversarial networks[C] / / ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019: 6955-6959。

[0020] [Reference 15] Chandna P, Blaauw M, Bonada J, et al. Wgansing: A multi-voice singing voice synthesizer based on the wasserstein-gan[C] / / 2019 27th European signal processing conference (EUSIPCO). IEEE, 2019: 1-5。

[0021] [Reference 16] Vaswani A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017。

[0022] [Reference 17] Lu P, Wu J, Luan J, et al. Xiaoicesing: A high-quality and integrated singing voice synthesis system[J]. arXiv preprint arXiv:2006.06261, 2020。

[0023] [Reference 18] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in neural information processing systems, 2020, 33: 6840-6851。

[0024] [Reference 19] Bao H, Huang S, Wei F, et al. Neural melody composition from lyrics[C] / / Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part I 8. Springer International Publishing, 2019: 499-511。

[0025] [Reference 20] Sheng Z, Song K, Tan X, et al. Songmass: Automatic songwriting with pre-training and alignment constraint[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(15): 13798-13805。

[0026] [Literature 21] Ju Z, Lu P, Tan X, et al. Telemelody: Lyric-to-melodygeneration with a template-based two-stage method[J]. arXiv preprint arXiv:2109.09617, 2021. Summary of the Invention

[0027] To overcome the shortcomings of the above-mentioned prior art, the present invention provides a model construction method and device for Song Ci music restoration, which takes Song Ci as input, first generates a melody corresponding to the Song Ci, then synthesizes a cappella based on the music score derived from the melody, and generates an accompaniment, and finally combines the a cappella, melody and accompaniment to generate a complete musical work.

[0028] According to one aspect of the present invention, a method for constructing a model for restoring Song Dynasty music is provided, comprising:

[0029] Construct a Song Dynasty music dataset;

[0030] Construct a Song Dynasty Ci music restoration model, including: a lyrics to melody generation module for generating a melody based on the input Song Dynasty Ci; a melody to a cappella generation module for generating a cappella based on the melody; a melody to accompaniment generation module for generating an accompaniment based on the melody; and a merging restoration module for merging the melody, a cappella, and accompaniment to generate a complete musical work.

[0031] The Song Ci music data set is used to train the constructed Song Ci music restoration model to obtain a trained Song Ci music restoration model.

[0032] As a further technical solution, a Song Dynasty music dataset was constructed, including the following annotations on the collected dataset:

[0033] Perform loudness matching on each song in the dataset to balance the volume of the dataset;

[0034] Slice the song based on the vocals in the song and control the length of the data strip within the preset duration;

[0035] Separate the vocals and accompaniment of each data, and extract the MIDI file;

[0036] Perform text annotation on vocal data. Based on the pre-collected lyrics of each song, a phoneme-based maximum string matching algorithm is used to extract the most suitable text for the current vocal from the lyrics.

[0037] As a further technical solution, we constructed a Song Dynasty music dataset, which included the following cleaning of the collected dataset:

[0038] Use the voice activity detection model to detect data without voice in the human voice dataset, and delete these data and their corresponding accompaniment audio and MIDI files;

[0039] Manual inspection was performed to delete data with text annotation errors and unsuccessful voice separation.

[0040] As a further technical solution, the lyrics to melody generation module includes:

[0041] Predicting the rhythm of each word in the lyrics and integrating the rhythm of each word into a template, the template including tonality, chords, rhythmic patterns and cadences;

[0042] The note corresponding to each word is determined based on the input of the template, a note sequence corresponding to the lyrics is generated, and the note sequence is converted into a melody.

[0043] As a further technical solution, the melody to a cappella generation module includes:

[0044] Generate speech acoustic features based on the input speech sequence;

[0045] Determine the duration of each phoneme based on the note sequence, the note sequence duration sequence, and the acoustic characteristics of the speech;

[0046] Determine fundamental frequency information based on machine learning;

[0047] Generate a cappella Mel-spectrogram based on phoneme, phoneme duration and fundamental frequency information.

[0048] As a further technical solution, determining the fundamental frequency information based on machine learning includes:

[0049] Determine the total number of frames of generated audio based on the phoneme duration and create a mapping matrix connecting phonemes to mel spectrograms;

[0050] Expanding the length of the acoustic feature to the total number of frames of the audio according to the mapping matrix;

[0051] According to preset conditions, multiple rounds of denoising are performed on the original noise to reconstruct the pitch sequence, and the fundamental frequency information is determined based on the reconstructed pitch sequence.

[0052] As a further technical solution, the melody to a cappella generation module also includes: creating a configuration file for defining the singing style of the Song Ci music restoration model, or for changing the melody MIDI file generated by the template to melody submodule.

[0053] As a further technical solution, the melody to accompaniment generation module includes:

[0054] Input text prompts and audio prompts, where the audio prompts are melody audio;

[0055] Use the encoder to quantize the input continuous audio features to obtain text tags and audio tags;

[0056] During inference, the text tokens and audio tokens are combined and input into an autoregressive Transformer model to generate accompanying audio tokens.

[0057] The accompaniment audio tokens are converted into continuous mel-spectrograms using a decoder.

[0058] According to one aspect of the present invention, a model building device for restoring Song Dynasty music is provided, comprising:

[0059] The first main module is used to construct a Song Dynasty music dataset;

[0060] The second main module is used to build a Song Dynasty poetry music restoration model, including: a lyrics to melody generation module, used to generate a melody based on the input Song Dynasty poetry; a melody to a cappella generation module, used to generate a cappella based on the melody; a melody to accompaniment generation module, used to generate an accompaniment based on the melody; and a combined restoration module, used to combine the melody, a cappella and accompaniment to generate a complete musical work;

[0061] The third main module is used to train the constructed Song Ci music restoration model using the Song Ci music dataset to obtain a trained Song Ci music restoration model.

[0062] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the steps of the model construction method for restoring Song Dynasty music.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) The Song Ci music restoration model proposed in this paper is trained using a large-scale Song Ci music dataset and has rich knowledge about Song Ci music.

[0065] (2) The Song Ci music restoration model proposed in this invention is structurally more suitable for generating Song Ci music. The model first uses a melody generation module based on template transition to generate a melody that is strictly aligned with the lyrics and conforms to the style of Song Ci. The model then guides subsequent music generation based on the melody to ensure the consistency of the a cappella and accompaniment. It then uses the combined singing generation of music parameters and the discrete accompaniment generation to generate high-quality Song Ci music that is sung strictly according to the input Song Ci and conforms to the style of Song Ci. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 Schematic diagram of the structure of SongSong, a Song Dynasty music restoration model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In recent years, music generation technology has made significant progress. However, existing models mainly focus on creating modern pop songs and have difficulty generating ancient music works with unique rhythms and styles, such as ancient Chinese Song Ci. To address this problem, the present invention proposes a Song Ci music restoration model. The model first predicts the melody from the input Song Ci, then generates a cappella and accompaniment based on the melody, and finally combines the a cappella, melody and accompaniment to generate a complete musical work. In addition, in order to solve the problem of scarcity of ancient music datasets, the present invention constructs a large-scale Chinese ancient Song Ci music dataset with a duration of 29.9 hours. By comparing SongSong with other music generation platforms Suno and SkyMusic, subjective and objective results show that the model SongSong proposed in the present invention has achieved the best level in generating high-quality Song Ci music.

[0069] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0070] This paper proposes SongSong, a music generation model that can sing Chinese Song poetry. The model uses autoregressive Transformer and diffusion model. Figure 1As shown in Figure 1, it takes Song Dynasty poetry as input and first generates a melody corresponding to the poem. Next, the model synthesizes a cappella singing based on the musical score derived from the melody and generates an accompaniment. Finally, the a cappella singing, melody, and accompaniment are combined to generate a complete musical work. To address the scarcity of Song Dynasty poetry music datasets and facilitate the restoration of Song Dynasty poetry music, this paper proposes OpenSongSong, a comprehensive dataset of ancient Chinese Song Dynasty poetry music, totaling 29.9 hours of Song Dynasty poetry music.

[0071] In order to generate high-quality Song Ci music that conforms to the traditional Song Ci style, the Song Ci music restoration model designed in the present invention consists of four modules, including: a lyrics-to-melody generation module, which is used to generate a melody based on the input Song Ci, convert the Song Ci into a template containing rhythm information, and generate the corresponding melody through melody mapping; a melody-to-accompaniment generation module, which is used to generate an a cappella based on the melody; a melody-to-accompaniment generation module, which is used to generate an accompaniment based on the melody; and a merged restoration module, which is used to merge the melody, a cappella and accompaniment to generate a complete Song Ci music work.

[0072] This embodiment of the present invention proposes the first open-source large-scale Song Dynasty music dataset, which mainly includes the following steps:

[0073] Step 1: Collect datasets. The datasets come from traditional guqin songs, ancient rhyme arrangements, restored guqin songs, and expert-restored Song Dynasty music.

[0074] Step 2, data set annotation and cleaning. In order to construct an accurate and complete data set, the present invention has designed multiple precise processes to annotate the data set. First, the loudness of each song is matched to balance the volume of the data set; then the song is sliced ​​based on the vocals in the song, and the length of the data is controlled within 30 seconds to ensure that multiple batches of training can be performed and the video memory occupancy remains stable; then the vocals and accompaniment of each data are separated, and the MIDI file is extracted; finally, the text annotation of the vocal data is performed. Based on the lyrics of each song collected in advance, the phoneme-based maximum string matching algorithm is used to extract the most suitable text for the current vocal from the lyrics. When cleaning the data set, the voice activity detection model is first used to detect the data without vocals in the vocal data set, and these data and their corresponding accompaniment audio and MIDI files are deleted. Then, a manual check is performed to delete the data with text annotation errors and unsuccessful vocal separation. After careful manual screening, data set annotation, cleaning and sorting, a total of 1,902 works were collected.

[0075] This embodiment of the present invention proposes SongSong, the first AI model capable of restoring high-quality Song Dynasty music. The model mainly includes the following steps:

[0076] Step 1: Lyrics to Melody Generation. This step uses a two-stage approach consisting of a lyrics to template submodule and a template to melody submodule to convert lyrics into melody.

[0077] The template consists of tonality, chords, rhythmic patterns and cadences. The tonality consists of a scale and a tonic, and each piece of music has only one tonic, which means that each template also contains only one tonality. The chords are a set of sounds composed of specific interval relationships, which are determined by the selected chord progression. The cadence indicates the end of a melodic paragraph and can be divided into "no cadence", "standard cadence" and "half cadence". The rhythmic pattern comes from the lyrics to template submodule, which consists of stacked Transformers and mainly represents the beats aligned with the lyrics. Let X represent a lyrics sequence of length n and Y represent a beat sequence of the same length.

[0078]

[0079] The template-to-melody submodule shares the same structure as the lyrics-to-template submodule. It takes a template as input and generates a sequence of notes corresponding to the lyrics. Each note is represented by a four-tuple consisting of measure, position, pitch, and duration. The measure and position determine the note's onset time, the pitch represents the note's pitch, and the duration specifies the note's duration. Let Z denote the note sequence, whose length is n.

[0080]

[0081] The training objectives of these two submodules are to minimize the number of lyrics to template data. and template to melody data The negative log-likelihood value of is used as the loss function. For example, the optimization objective of the lyrics-to-template submodule is as follows.

[0082]

[0083] Step 2, melody to a cappella generation. The a cappella generation module of the present invention takes speech units as input, which means that the input consists of a speech sequence P, not a lyric sequence X. The length of P corresponds to the total number of speech units m. However, P alone is not enough, and the duration of each phoneme needs to be determined, expressed as Q, and its length is also m. In addition, the a cappella generation module requires melody information to ensure that the lyrics are sung at the correct pitch. In the previous step, the present invention obtains a note quadruple sequence Y, which includes a note sequence U and a note duration sequence V. They correspond one-to-one to each lyric, so the length of U and V are both n. In order to generate a complete song, the present invention also requires fundamental frequency information F0, whose length is equal to the number of frames T of the generated audio. This F0 must also be inferred by a machine learning model. The variance encoder is a speech acoustic encoder based on FastSpeech2 (Ren Y, Hu C, TanX, et al. Fastspeech 2: Fast and high-quality end-to-end text to speech[J].arXiv preprint arXiv:2006.04558, 2020), which consists of Transformer layers. It takes P as input and generates speech acoustic features from the last hidden layer. .

[0084]

[0085] The present invention uses FastSpeech2's duration predictor to estimate the duration of each phoneme. The core component of this predictor is a convolutional network consisting of two layers of one-dimensional convolution (LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.). It uses the note information U, V and the speech acoustic features. as input.

[0086]

[0087] During training, the present invention optimizes the predictor by computing the L2 norm of the phoneme duration.

[0088]

[0089] The F0 predictor is based on a diffusion model. It first determines the total number of frames T of generated audio by adding the phoneme duration Q and creates a mapping matrix that connects the phonemes to the Mel spectrograms. It then uses this mapping matrix to extend the length of the acoustic features from m to T. Finally, it uses the speech acoustic features from 1 to T. As a condition, from the original noise conduct A round of denoising is performed to reconstruct the pitch sequence. The F0 predictor is implemented using Wavenet (Van Den Oord A, Dieleman S, Zen H, et al. Wavenet: A generative model for raw audio[J]. arXiv preprint arXiv:1609.03499, 2016, 12), a fully probabilistic autoregressive model consisting of causal convolutional layers. During training, the F0 predictor uses the L2 norm as its loss function.

[0090]

[0091] After obtaining the phoneme duration Q and F0, the present invention will generate a cappella Mel spectrum M consisting of T frames. First, the present invention uses an acoustic encoder to obtain an acoustic representation The input of the acoustic encoder includes phoneme P, phoneme duration Q and F0, which means It captures the key information of singing skills. The basic structure of the acoustic encoder is the same as that of the variance encoder.

[0092]

[0093] Then, the present invention converts the human voice characteristics The mel spectrum of the singing voice is reconstructed by inputting it into the diffusion model as a condition. The standard diffusion model inference process is summarized as follows, where represents the initial Gaussian noise, and R represents the number of denoising iterations.

[0094]

[0095] To improve the quality of the final audio output and speed up the inference process of the diffusion model, this paper implements the shallow diffusion mechanism introduced by DiffSinger. This mechanism uses a ConvNeXt-based decoder (Liu Z, Mao H, Wu CY, et al. A convnet for the 2020s[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition. 2022: 11976-11986), which consists of deep convolutional blocks and derives the spectrum at the Kth denoising step. Therefore, the denoiser does not perform R-round denoising on the original noise, but Start K rounds of denoising.

[0096]

[0097] The basic model of the denoiser is the same as that of the F0 predictor. During the training phase, the present invention enhances the auxiliary decoder by calculating the L1 norm of the Mel spectrum, while the denoising uses the L2 norm.

[0098]

[0099] The configuration file created in this step defines the singing style of the music generation model and can also be used to modify the melody MIDI file generated by the Template to Melody submodule. Because the music generated by the model may not suit everyone's taste, a music generation model that allows flexible control over the output content is more suitable for everyone. Therefore, users can access the configuration file generated by SongSong. If the user is not satisfied with the generated melody, they can modify the configuration file independently, adjusting the generated melody audio and a cappella audio simultaneously.

[0100] Step 3: Melody to Accompaniment Generation. After completing the first two steps, we obtain the a cappella audio and melody audio, which can be combined to generate a complete music track. To enrich the final music output, additional generative models can be used to create accompaniment played by other instruments.

[0101] This paper implements an architecture that discretizes acoustic feature information and generates discrete acoustic tokens for accompaniment generation using an autoregressive language model. The input consists of a text prompt and an audio prompt, where the audio prompt is a melody. The continuous audio features are quantized using Encodec's encoder (Défossez A, Copet J, Synnaeve G, et al. Highfidelity neural audio compression[J]. arXiv preprint arXiv:2210.13438, 2022). During inference, the text and audio tokens are combined and input into an autoregressive Transformer model to generate the accompaniment audio tokens. Finally, Encodec's decoder converts the tokens into continuous mel-spectrograms.

[0102] As a preferred embodiment, the deep learning framework used in this embodiment of the present invention is Pytorch, version 2.1.0, and CUDA version 12.1. The experimental hardware environment is an RTX 3090 graphics card and an Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz. The specific implementation process of the model device for restoring Song Dynasty music described in this embodiment of the present invention is as follows:

[0103] Step 1: Build a large-scale, multi-purpose Song Dynasty music dataset OpenSongSong.

[0104] In step 2, the lyrics-to-template module predicts the rhythm of each word in the poem and then integrates this rhythm into a template containing specific key and chord details.

[0105] Step 3: The template is processed by the template-to-melody module to determine the note corresponding to each word.

[0106] Step 4: Convert the note sequence into a melody MIDI file and generate a configuration file. This configuration file contains information about the phonemes, notes, and fundamental frequency (F0) of the audio to be created, which is used to generate a cappella audio.

[0107] Step 5: The accompaniment generation module uses the melody audio to create accompaniment for other instruments.

[0108] Step 6: Generate complete Song Ci music audio by merging the a cappella, melody and accompaniment audio.

[0109] The specific implementation plan is as follows:

[0110] 1. Experimental Setup

[0111] Currently, there are limited open-source systems capable of generating audio that combines a cappella and instrumental accompaniment. Therefore, this paper compares two leading commercial music generation software programs: Suno and SkyMusic. These systems leverage the powerful generative capabilities of large language models to generate high-quality music, both a cappella and instrumental, based on simple text prompts. Furthermore, they allow users to input audio references to generate music of similar styles.

[0112] The method proposed in this embodiment of the present invention uses both objective and subjective metrics. The objective metric used is the Frechet audio distance (Kilgour K, Zuluaga M, Roblek D, et al. Frechet audio distance: Ametric for evaluating music enhancement algorithms[J]. arXiv preprint arXiv:1812.08466, 2018), which measures the Frechet distance between the embedding value distributions of two audio groups, thereby providing a generated audio quality assessment that is closer to human perception. This embodiment of the present invention utilizes the VGGish and PANN models (Kong Q, Cao Y, Iqbal T, et al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition[J]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 2880-2894) to extract audio embeddings. In terms of subjective evaluation, the present invention evaluates musical structure, musical richness, continuity of motivation, timbre, accuracy of pronunciation, naturalness of sound, conformity with the style of Song Ci, and conformity with accompaniment. The first four indicators are evaluated from the perspective of ordinary music, while the last four indicators evaluate the generated music from the perspective of Song Ci music.

[0113] The present invention randomly selected 50 songs from the dataset, which were not part of the training data used to build OpenSongSong. After basic segmentation to eliminate long periods of silence, the present invention finally obtained 85 audio segments, totaling 1.8 hours. The singers in the test set are male. The testing process consists of two stages. The first stage is a zero-sample test, in which only text prompts and Song lyrics are provided to Suno and SkyMusic, and the prompt content is "Chinese classical style, Song Dynasty, Guzheng, male voice". The second stage is a few-sample test, in which Suno and SkyMusic receive an additional Song Dynasty work as a reference. The present invention invited 9 experts from music schools to participate in the subjective evaluation.

[0114] 2. Training details

[0115] The lyrics to template and template to melody modules share the same Transformer architecture, both consisting of a 4-layer encoder block and a 4-layer decoder block. Each block contains 4 attention heads and 256 linear units. In the a cappella generation module, the acoustic encoder and variance encoder of the present invention are based on FastSpeech2 and include 4 feed-forward Transformers. The duration predictor consists of a 5-layer one-dimensional convolutional network with a convolution kernel size of 3, and the input and output dimensions are both 512. The auxiliary decoder consists of a 6-layer ConvNeXt decoder, with a convolution kernel size of 7 per layer and the input and output dimensions are both 512. The denoiser and F0 predictor are both based on Wavenet, containing 20 layers of convolution and a convolution kernel size of 3. The input and output dimensions of each layer of the denoiser are 512 and 1024, respectively, while the dimension of the F0 predictor is half that of the denoiser. For the accompaniment generation module, the present invention uses 1.5B-Musicgen-melody as the initial parameters.

[0116] This paper uses data from the largest speaker in the OpenSongSong corpus (approximately 3.5 hours) to train SongSong's lyrics-to-template, template-to-melody, and a cappella generation modules. All training processes use the Adam optimizer with a learning rate of 5e-4. Training is performed on a GeForce RTX-3090, with a maximum training step of 160,000 for each module.

[0117] 3. Experimental Results

[0118] Table 1 shows the comparison results for SongSong, Suno, and SkyMusic. Zero-shot and Few-shot indicate whether Song Ci audio was provided as a reference for Suno and SkyMusic. Regarding subjective metrics, MS represents musical structure, ER represents musical richness, MC represents motivic continuity, PA represents pronunciation accuracy, VN represents vocal naturalness, SQ represents sound quality, SCS represents conformity to the Song Ci style, and AC represents conformity to the accompaniment.

[0119] Table 1 Comparison results of SongSong, Suno and SkyMusic

[0120]

[0121] In a subjective evaluation of Song Ci music, the present invention found that Suno and SkyMusic performed poorly in cue acceptance, failing to fully grasp the Chinese classical style. The music they generated almost always exhibited a pop music style, with a stylistic consistency significantly different from that of Song Ci music. Furthermore, the generated musical accompaniment did not utilize the guzheng instrument, as designed by the present invention, but instead employed instruments commonly found in pop music, such as guitar and electronic keyboards. Even when Song Ci music was input as a reference, this problem remained unresolved. Consequently, Suno and SkyMusic scored significantly lower than SongSong on the SCS (Song Ci Style Fit) and AC (Accompaniment Fit) metrics. After training on the OpenSongSong Song Ci music corpus, SongSong was able to fully grasp the pitch and rhythm of Song Ci music, generating singing styles and accompaniments that better align with the characteristics of Song Ci music. Furthermore, due to the presence of many uncommon characters in Song Ci, these two models, trained on pop music, were unable to clearly and accurately sing each character, sometimes even missing characters or randomly generating lyrics, resulting in low scores on the PA (Accuracy of Pronunciation) metric. After training with a small number of samples, SkyMusic's pronunciation accuracy improved, while Suno's showed a downward trend. However, this issue did not significantly affect Suno and SkyMusic's VN (Voice Naturalness) scores. Despite the pre-training scale and large parameter model size, Suno and SkyMusic were still able to maintain a certain degree of naturalness from the original performance, resulting in VN scores comparable to SongSong. SongSong undergoes rigorous lyric and melody alignment, and its training corpus already includes rare characters that may appear in Song Ci music. This allows it to not only strictly follow the input Song Ci but also ensure that every word is pronounced correctly. As a result, it significantly outperforms the other two models in the PA metric and is comparable to them in the VN metric. However, when subjectively evaluated from the perspective of general music, SongSong underperforms Suno and SkyMusic. This phenomenon is due to multiple factors, in addition to limited model size and training data capacity. Song Ci has relatively fixed sentence structure and length, and the performance style is relatively monotonous. However, Suno and SkyMusic treated the performance as a pop song and employed structural optimization techniques such as repetition, contrast, and pitch variation to generate more complex and coherent pop music, resulting in better performance in terms of MS (Musical Structure) and MC (Motivational Continuity). Suno and SkyMusic exhibited poor controllability. Despite receiving prompts requiring them to use only the guzheng for accompaniment, they still incorporated other instruments, resulting in a more complex arrangement and, consequently, a higher ER (Richness of Musical Arrangement). After training with a small number of samples, Suno and SkyMusic's MS (Musical Structure) and ER (Richness of Musical Arrangement) scores decreased, while their MC (Motivational Continuity) scores remained unchanged.

[0122] This observation supports the analysis of the present invention, that is, the closer the generated music is to Song Dynasty poetry, the more monotonous and consistent its style is. In terms of sound quality, due to the lack of high-quality Song Dynasty music, the model trained by the present invention is difficult to generate music with extremely high sound quality. In terms of objective evaluation indicators, SongSong's Relatively high, but In this respect, it is comparable to Suno and SkyMusic. Since the audio used for comparison is Song Dynasty poetry, objectively speaking, SongSong also generates high-quality Song Dynasty poetry audio that conforms to human auditory perception. The standard deviation statistics of the subjective evaluation are shown in Table 2.

[0123] Table 2 Standard deviation statistics of subjective evaluation

[0124]

[0125] The implementation of each embodiment of the present invention is based on programmed processing performed by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of each embodiment of the present invention are packaged into various modules. Based on this reality, and in addition to the aforementioned embodiments, an embodiment of the present invention provides a model-building device for restoring Song Ci music. This device is used to execute a model-building method for restoring Song Ci music described in the aforementioned method embodiment.

[0126] The device includes: a first main module for constructing a Song Ci music dataset; a second main module for constructing a Song Ci music restoration model, including: a lyrics to melody generation module for generating a melody according to an input Song Ci; a melody to a cappella generation module for generating a cappella according to the melody; a melody to accompaniment generation module for generating an accompaniment according to the melody; a merging and restoration module for merging the melody, a cappella and accompaniment to generate a complete musical work; and a third main module for training the constructed Song Ci music restoration model using the Song Ci music dataset to obtain a trained Song Ci music restoration model.

[0127] An embodiment of the present invention provides a model construction device for restoring Song Dynasty poetry music, which adopts a melody generation module based on template transition to generate a melody that is strictly aligned with the lyrics and conforms to the style of Song Dynasty poetry, and guides subsequent music generation based on the melody to ensure the consistency of a cappella and accompaniment. It then uses singing generation combined with music parameters and discrete accompaniment generation respectively to generate high-quality Song Dynasty poetry music that is sung strictly according to the input Song Dynasty poetry and conforms to the style of Song Dynasty poetry.

[0128] It should be noted that the device embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set, and the principles are basically the same as the principles of the above-mentioned device embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiments, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned device embodiments to obtain corresponding device class embodiments for implementing the methods in other method class embodiments.

[0129] Based on the same inventive concept as the above-mentioned method embodiment, an embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the steps of the model construction method for restoring Song Dynasty music.

[0130] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A model construction method for Song Ci music restoration, characterized in that: include: Construct a Song Dynasty music dataset; Constructing a Song Dynasty Ci music restoration model, including: The lyric-to-melody generation module is used to generate a melody based on the input Song Dynasty lyrics. The module predicts the rhythm of each word in the lyrics and integrates the rhythm of each word into a template. The template includes tonality, chords, rhythmic patterns, and cadences. The module determines the musical note corresponding to each word based on the input template, generates a musical note sequence corresponding to the lyrics, and converts the musical note sequence into a melody. A melody to a cappella generation module is configured to generate a cappella based on the melody, comprising: generating speech acoustic features based on an input speech sequence; determining the duration of each phoneme based on a note sequence, a note sequence duration sequence, and the speech acoustic features; determining fundamental frequency information based on machine learning; and generating a cappella Mel-spectrogram based on the phoneme, phoneme duration, and fundamental frequency information. A melody to accompaniment generation module is used to generate an accompaniment based on the melody; a merging and restoration module is used to merge the melody, a cappella and accompaniment to generate a complete musical work; The Song Ci music data set is used to train the constructed Song Ci music restoration model to obtain a trained Song Ci music restoration model.

2. The model construction method for restoring Song Ci music according to claim 1 is characterized in that: Constructing the Song Dynasty music dataset includes annotating the collected dataset as follows: Perform loudness matching on each song in the dataset to balance the volume of the dataset; Slice the song based on the vocals in the song and control the length of the data strip within the preset duration; Separate the vocals and accompaniment of each data, and extract the MIDI file; Perform text annotation on vocal data. Based on the pre-collected lyrics of each song, a phoneme-based maximum string matching algorithm is used to extract the most suitable text for the current vocal from the lyrics.

3. The model construction method for restoring Song Ci music according to claim 2 is characterized in that: Constructing the Song Dynasty music dataset involves cleaning the collected dataset as follows: Use the voice activity detection model to detect data without voice in the human voice dataset, and delete these data and their corresponding accompaniment audio and MIDI files; Manual inspection was performed to delete data with text annotation errors and unsuccessful voice separation.

4. The model construction method for restoring Song Ci music according to claim 1, characterized in that: Determine fundamental frequency information based on machine learning, including: Determine the total number of frames of generated audio based on the phoneme duration and create a mapping matrix connecting phonemes to mel spectrograms; Expanding the length of the acoustic feature to the total number of frames of the audio according to the mapping matrix; According to preset conditions, multiple rounds of denoising are performed on the original noise to reconstruct the pitch sequence, and the fundamental frequency information is determined based on the reconstructed pitch sequence.

5. The model construction method for restoring Song Ci music according to claim 4 is characterized in that: The melody to a cappella generation module also includes: creating a configuration file for defining the singing style of the Song Ci music restoration model, or for changing the melody MIDI file generated by the template to melody submodule.

6. The model construction method for restoring Song Ci music according to claim 1, characterized in that: The melody to accompaniment generation module includes: Input text prompts and audio prompts, where the audio prompts are melody audio; Use the encoder to quantize the input continuous audio features to obtain text tags and audio tags; During inference, the text tokens and audio tokens are combined and input into an autoregressive Transformer model to generate accompanying audio tokens. The accompaniment audio tokens are converted into continuous mel-spectrograms using a decoder.

7. A model building device for restoring Song Dynasty music, characterized in that: include: The first main module is used to construct a Song Dynasty music dataset; The second main module is used to build a Song Dynasty music restoration model, including: The lyric-to-melody generation module is used to generate a melody based on the input Song Dynasty lyrics. The module predicts the rhythm of each word in the lyrics and integrates the rhythm of each word into a template. The template includes tonality, chords, rhythmic patterns, and cadences. The module determines the musical note corresponding to each word based on the input template, generates a musical note sequence corresponding to the lyrics, and converts the musical note sequence into a melody. A melody to a cappella generation module is configured to generate a cappella based on the melody, comprising: generating speech acoustic features based on an input speech sequence; determining the duration of each phoneme based on a note sequence, a note sequence duration sequence, and the speech acoustic features; determining fundamental frequency information based on machine learning; and generating a cappella Mel-spectrogram based on the phoneme, phoneme duration, and fundamental frequency information. A melody to accompaniment generation module is used to generate an accompaniment based on the melody; a merging and restoration module is used to merge the melody, a cappella and accompaniment to generate a complete musical work; The third main module is used to train the constructed Song Ci music restoration model using the Song Ci music dataset to obtain a trained Song Ci music restoration model.

8. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the steps of the model construction method for restoring Song Dynasty music as described in any one of claims 1 to 6.

Images