Song poem music restoration model construction method and device

Abstract

The invention discloses a Song poem music restoration model construction method and device. The construction method comprises the following steps: constructing a Song poem music data set; the Song poem music restoration model comprises a lyric-to-melody generation module which is used for generating melody according to input Song poems; the melody-to-aclasas generation module is used for generating an aclasas according to the melody; the melody-to-accompaniment generation module is used for generating an accompaniment according to the melody; the merging and restoring module is used for merging the melody, the unsold singing and the accompaniment to generate a complete music work; and training the constructed Song poem music restoration model by using the Song poem music data set to obtain a trained Song poem music restoration model. Compared with the prior art, high-quality restoration of the Song poem music is realized.

Description

Technical Field

[0001] The present invention belongs to the field of music generation, and in particular relates to a model building method and device for restoring Song Dynasty poetry music. Background Art

[0002] In recent years, the field of music generation has made significant progress, and a large number of studies have focused on the generation of singing voices. [1,2] and melody generation [3-5] Although deep learning has the potential to restore Song Dynasty music, there is currently no open-source dataset of ancient Chinese Song Dynasty music available. Existing Chinese music datasets, such as M4Singer [6] and Opencpop [7] , mainly for pop music. Most of the models that can generate complete music with a cappella and accompaniment are commercial products, such as Suno and SkyMusic. These models usually adopt an end-to-end architecture and use the powerful generation capabilities of large language models (LLMs). [8] , generating complete audio based on the text prompts and lyrics provided by the user. In addition, users can also input audio as a style reference for the model to generate music of the corresponding style. However, the generation process of these systems is often uncontrollable and highly random. Since the training data of these models mainly comes from contemporary pop songs, the music they generate often reflects the mainstream style, making it difficult to generate music that conforms to the style of Song Ci. In short, due to the lack of a dedicated Song Ci music dataset and an open source complete music generation model, restoring Song Ci music based on deep learning still faces huge challenges.

[0003] Singing Voice Synthesis (SVS) is a technology that uses musical notation information (such as lyrics, rhythm, pitch, etc.) to generate natural and expressive singing voices. [9] In recent years, deep learning has made revolutionary progress in the field of SVS, and has made significant progress compared to traditional methods. Early deep learning-based systems used feedforward neural networks (FFNN).

[10] , by predicting acoustic features directly from musical scores, outperforming systems based on Hidden Markov Models (HMMs)

[11] . Later, Long Short-Term Memory (LSTM)

[12] and Convolutional Neural Networks (CNNs)

[13] The introduction of GAN has enhanced the ability to model the long-term dependencies and acoustic features of singing. [14,15] It is also integrated into the SVS system to alleviate the over-smoothing problem, thereby achieving more natural and expressive singing voice synthesis. In addition, state-of-the-art deep learning architectures, such as transformer-based models

[16] (such as XiaoiceSingv

[17] and HifiSinger), as well as diffusion-based models

[18] (such as DiffSinger), further improving the quality of SVS.

[0004] Lyrics-to-melody generation is a key task in automatic composition, which involves generating a melody that matches a given lyric. It usually uses an end-to-end model to generate the melody directly from the lyrics. Bao et al.

[19] and Yu et al. [3] Use sequence-to-sequence models to generate melodies from lyrics. However, these end-to-end models require a large amount of paired lyrics and melody data, and obtaining sufficient amounts of paired data is difficult and expensive. To address this problem, Sheng et al.

[20] By training lyrics-to-lyrics and melody-to-melody models separately and interacting them in the subsequent stage, the reliance on paired data is avoided. However, since the unpaired data is not fully utilized when learning the correlation between lyrics and melody, the consistency of lyrics and melody features cannot be ensured. Ju et al.

[21] We propose TeleMelody, a two-stage generative model: lyrics-to-template and template-to-melody. The template bridges the gap between lyrics and melody, promotes better feature alignment, and improves the controllability of the generated melody.

[0005] In summary, how to improve the existing music generation model so that it can better restore Song Ci music and generate high-quality Song Ci music that conforms to a specific style has become an urgent problem to be solved.

[0006] [Document 1] Liu J, Li C, Ren Y, et al. Diffsinger: Singing voices synthesis via shallow diffusion mechanism[C] / / Proceedings of the AAAIconference on artificial intelligence. 2022, 36(10): 11020-11028.

[0007] [Reference 2] Chen J, Tan X, Luan J, et al. Hifisinger: Towards high-fidelity neural singing voice synthesis[J]. arXiv preprint arXiv:2009.01776, 2020。

[0008] [Reference 3] Yu Y, Srivastava A, Canales S. Conditional LSTM-GAN for melody generation from lyrics[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2021, 17(1): 1-20。

[0009] [Reference 4] Zhang C, Chang L, Wu S, et al. Relyme: improving lyric-to-melody generation by incorporating lyric-melody relationships[C] / / Proceedings of the 30th ACM International Conference on Multimedia. 2022: 1047-1056。

[0010] [Reference 5] Lv A, Tan X, Qin T, et al. Re-creation of creations: A new paradigm for lyric-to-melody generation[J]. arXiv preprint arXiv:2208.05697, 2022。

[0011] [Reference 6] Zhang L, Li R, Wang S, et al. M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus[J]. Advances in Neural Information Processing Systems, 2022, 35: 6914-6926。

[0012] [Reference 7] Wang Y, Wang X, Zhu P, et al. Opencpop: A high-quality opensource chinese popular song corpus for singing voice synthesis[J]. arXivpreprint arXiv:2201.07429, 2022。

[0013] [Reference 8] Zhao W X, Zhou K, Li J, et al. A survey of large languagemodels[J]. arXiv preprint arXiv:2303.18223, 2023。

[0014] [Reference 9] Yamamoto R, Song E, Kim J M. Parallel WaveGAN: A fastwaveform generation model based on generative adversarial networks withmulti-resolution spectrogram[C] / / ICASSP 2020-2020 IEEE InternationalConference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020:6199-6203。

[0015] [Reference 10] Nishimura M, Hashimoto K, Oura K, et al. Singing VoiceSynthesis Based on Deep Neural Networks[C] / / Interspeech. 2016: 2478-2482。

[0016] [Reference 11] Rabiner L R. A tutorial on hidden Markov models and selectedapplications in speech recognition[J]. Proceedings of the IEEE, 1989, 77(2):257-286。

[0017] [Reference 12] Kim J, Choi H, Park J, et al. Korean singing voice synthesissystem based on an LSTM recurrent neural network[C] / / Proc. Interspeech. 2018:1551-1555。

[0018] [Reference 13] Nakamura K, Takaki S, Hashimoto K, et al. Fast and high-quality singing voice synthesis system based on convolutional neural networks[C] / / ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP). IEEE, 2020: 7239-7243。

[0019] [Reference 14] Hono Y, Hashimoto K, Oura K, et al. Singing voice synthesisbased on generative adversarial networks[C] / / ICASSP 2019-2019 IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP).IEEE, 2019: 6955-6959。

[0020] [Reference 15] Chandna P, Blaauw M, Bonada J, et al. Wgansing: A multi-voice singing voice synthesizer based on the wasserstein-gan[C] / / 2019 27thEuropean signal processing conference (EUSIPCO). IEEE, 2019: 1-5。

[0021] [Reference 16] Vaswani A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017。

[0022] [Reference 17] Lu P, Wu J, Luan J, et al. Xiaoicesing: A high-quality and integrated singing voice synthesis system[J]. arXiv preprint arXiv:2006.06261, 2020。

[0023] [Reference 18] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in neural information processing systems, 2020, 33: 6840-6851。

[0024] [Reference 19] Bao H, Huang S, Wei F, et al. Neural melody composition from lyrics[C] / / Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part I 8. Springer International Publishing, 2019: 499-511。

[0025] [Reference 20] Sheng Z, Song K, Tan X, et al. Songmass: Automatic songwriting with pre-training and alignment constraint[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(15): 13798-13805。

[0026] [Literature 21] Ju Z, Lu P, Tan X, et al. Telemelody: Lyric-to-melodygeneration with a template-based two-stage method[J]. arXiv preprint arXiv:2109.09617, 2021. Summary of the invention

[0027] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a model construction method and device for Song Ci music restoration, which takes Song Ci as input, first generates a melody corresponding to the Song Ci, then synthesizes a cappella based on the music score derived from the melody, and generates an accompaniment, and finally combines the a cappella, melody and accompaniment to generate a complete musical work.

[0028] According to one aspect of the present invention, a model construction method for restoring Song Ci music is provided, comprising: Construct a Song Dynasty music dataset; Constructing a Song Ci music restoration model, including: a lyrics to melody generation module, used to generate a melody according to the input Song Ci; a melody to a cappella generation module, used to generate a cappella according to the melody; a melody to accompaniment generation module, used to generate an accompaniment according to the melody; a combined restoration module, used to combine the melody, a cappella and accompaniment to generate a complete musical work; The Song Ci music data set is used to train the constructed Song Ci music restoration model to obtain a trained Song Ci music restoration model.

[0029] As a further technical solution, a Song Ci music dataset is constructed, including the following annotations on the collected dataset: Perform loudness matching on each song in the dataset to balance the volume of the dataset; Slice the song based on the vocals in the song and control the length of the data strip within the preset duration; Separate the vocals and accompaniment of each data, and extract the MIDI file; Text annotation of vocal data is performed based on the lyrics of each song collected in advance, and a phoneme-based maximum string matching algorithm is used to extract the most suitable text for the current vocal from the lyrics.

[0030] As a further technical solution, a Song Ci music dataset is constructed, including the following cleaning of the collected dataset: Use the voice activity detection model to detect the data without voice in the human voice dataset, and delete these data and their corresponding accompaniment audio and MIDI files; Manual inspection was performed to delete data with text annotation errors and unsuccessful voice separation.

[0031] As a further technical solution, the lyrics to melody generation module includes: Predicting the rhythm of each word in the lyrics and integrating the rhythm of each word into a template, the template including tonality, chords, rhythmic patterns and cadences; The note corresponding to each word is determined based on the input of the template, a note sequence corresponding to the lyrics is generated, and the note sequence is converted into a melody.

[0032] As a further technical solution, the melody to a cappella generation module includes: Generate speech acoustic features according to the input speech sequence; Determine the duration of each phoneme based on the note sequence, the note sequence duration sequence and the acoustic features of the speech; Determine fundamental frequency information based on machine learning; Generate a cappella Mel-spectrogram based on phonemes, phoneme duration and fundamental frequency information.

[0033] As a further technical solution, the baseband information is determined based on machine learning, including: Determine the total number of frames of generated audio based on the phoneme duration and create a mapping matrix connecting the phonemes to the Mel spectrograms; Expanding the length of the acoustic feature to the total number of audio frames according to the mapping matrix; According to preset conditions, multiple rounds of denoising are performed from the original noise to reconstruct the pitch sequence, and the fundamental frequency information is determined based on the reconstructed pitch sequence.

[0034] As a further technical solution, the melody to a cappella generation module also includes: creating a configuration file for defining the singing style of the Song Ci music restoration model, or for changing the melody MIDI file generated by the template to melody submodule.

[0035] As a further technical solution, the melody to accompaniment generation module includes: Input text prompts and audio prompts, wherein the audio prompts are melody audio; Use an encoder to quantize the input continuous audio features to obtain text tags and audio tags; During inference, the text tokens and audio tokens are combined and input into an autoregressive Transformer model to generate the accompaniment audio tokens. The accompaniment audio tokens are converted into continuous mel-spectrograms using a decoder.

[0036] According to one aspect of the present invention, a model building device for restoring Song Ci music is provided, comprising: The first main module is used to construct a Song Dynasty music dataset; The second main module is used to construct a Song Ci music restoration model, including: a lyrics to melody generation module, used to generate a melody according to the input Song Ci; a melody to a cappella generation module, used to generate a cappella according to the melody; a melody to accompaniment generation module, used to generate an accompaniment according to the melody; a combined restoration module, used to combine the melody, a cappella and accompaniment to generate a complete musical work; The third main module is used to train the constructed Song Ci music restoration model using the Song Ci music data set to obtain a trained Song Ci music restoration model.

[0037] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the steps of the method for constructing a model for restoring Song Ci music.

[0038] Compared with the prior art, the present invention has the following beneficial effects: (1) The Song Ci music restoration model proposed in this invention is trained using a large-scale Song Ci music dataset and has rich knowledge about Song Ci music.

[0039] (2) The Song Ci music restoration model proposed in the present invention is more suitable for generating Song Ci music in terms of structure. The model first adopts a melody generation module based on template transition to generate a melody that is strictly aligned with the lyrics and conforms to the style of Song Ci, and guides subsequent music generation based on the melody to ensure the consistency of the a cappella and the accompaniment. Then, the singing generation based on the joint use of music parameters and the discrete accompaniment generation are respectively adopted to generate high-quality Song Ci music that is sung strictly according to the input Song Ci and conforms to the style of Song Ci. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of the structure of SongSong, a Song Dynasty music restoration model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In recent years, music generation technology has made significant progress. However, existing models mainly focus on creating modern pop songs, and it is difficult to generate ancient music works with unique rhythms and styles, such as the ancient Chinese Song Ci. To address this problem, the present invention proposes a Song Ci music restoration model, which first predicts the melody from the input Song Ci, then generates a cappella and accompaniment based on the melody, and finally merges the a cappella, melody and accompaniment to generate a complete music work. In addition, in order to solve the problem of scarcity of ancient music datasets, the present invention constructs a large-scale Chinese ancient Song Ci music dataset with a duration of 29.9 hours. By comparing SongSong with other music generation platforms Suno and SkyMusic, the subjective and objective results show that the model SongSong proposed in the present invention has reached the best level in generating high-quality Song Ci music.

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0044] The present invention proposes SongSong, a music generation model that can sing Chinese Song lyrics. The model uses autoregressive Transformer and diffusion model. Figure 1 As shown in Figure 1, it takes Song Ci as input and first generates a melody corresponding to the Song Ci. Then, the model synthesizes a cappella based on the score derived from the melody and generates an accompaniment. Finally, the a cappella, melody, and accompaniment are combined to generate a complete musical work. In order to solve the scarcity problem of Song Ci music datasets and help restore Song Ci music, the present invention proposes OpenSongSong, a comprehensive dataset of ancient Chinese Song Ci music, totaling 29.9 hours of Song Ci music.

[0045] In order to generate high-quality Song Ci music that conforms to the traditional Song Ci style, the Song Ci music restoration model designed by the present invention consists of four modules, including: a lyrics-to-melody generation module, which is used to generate a melody based on the input Song Ci, convert the Song Ci into a template containing rhythm information, and generate a corresponding melody through melody mapping; a melody-to-capella generation module, which is used to generate a cappella based on the melody; a melody-to-accompaniment generation module, which is used to generate an accompaniment based on the melody; and a combined restoration module, which is used to combine the melody, cappella and accompaniment to generate a complete Song Ci music work.

[0046] The embodiment of the present invention proposes the first open-source large-scale Song Dynasty music dataset, which mainly includes the following steps: Step 1: Collect data sets. Data sets include traditional guqin songs, ancient rhyme arrangements, restored guqin songs, and expert restored Song Dynasty music.

[0047] Step 2, data set annotation and cleaning. In order to construct an accurate and complete data set, the present invention designs multiple precise processes for data set annotation. First, loudness matching is performed on each song to balance the volume of the data set; then the song is sliced ​​based on the vocals in the song, and the length of the data is controlled within 30 seconds to ensure that multiple batches of training can be performed during training and the video memory occupancy remains stable; then the vocals and accompaniment are separated for each data, and the MIDI file is extracted; finally, the text annotation of the vocal data is performed, based on the lyrics of each song collected in advance, and the phoneme-based maximum string matching algorithm is used to extract the most suitable text under the current vocal from the lyrics. When cleaning the data set, the voice activity detection model is first used to detect the data without vocals in the vocal data set, and these data and their corresponding accompaniment audio and MIDI files are deleted, and then manual inspection is performed to delete the data with text annotation errors and unsuccessful vocal separation. After careful manual screening, data set annotation, cleaning and sorting, a total of 1,902 works were collected.

[0048] The embodiment of the present invention proposes the first AI model SongSong that can restore high-quality Song Dynasty music, which mainly includes the following steps: Step 1, lyrics to melody generation. This step adopts a two-stage method including lyrics to template submodule and template to melody submodule to convert lyrics into melody.

[0049] The template consists of tonality, chords, rhythmic patterns and cadences. The tonality consists of a scale and a tonic, and each piece of music has only one tonic, which means that each template also contains only one tonality. The chord is a set of sounds composed of specific interval relationships, which is determined by the selected chord progression. The cadence indicates the end of a melodic paragraph and can be divided into "no cadence", "proper cadence" and "half cadence". The rhythmic pattern comes from the lyrics to template submodule, which consists of stacked Transformers and mainly represents the beats aligned with the lyrics. Let X represent a lyrics sequence of length n, and Y represent a beat sequence of the same length.

[0050]

[0051] The template-to-melody submodule has the same structure as the lyrics-to-template submodule. It takes a template as input and generates a sequence of notes corresponding to the lyrics. Each note is represented by a 4-tuple consisting of measure, position, pitch, and duration. The measure and position are used to determine the start time of the note, the pitch represents the pitch of the note, and the duration specifies the duration of the note. Let Z represent the sequence of notes, with length n.

[0052]

[0053] The training objectives of these two submodules are to minimize the number of lyrics to template data. and template to melody data The negative log-likelihood value of is used as the loss function. For example, the optimization objective of the lyrics-to-template submodule is as follows.

[0054]

[0055] Step 2, melody to a cappella generation. The a cappella generation module of the present invention uses speech units as input, which means that the input consists of a speech sequence P, not a lyric sequence X. The length of P corresponds to the total number of speech units m. However, P alone is not enough, and the duration of each phoneme needs to be determined, expressed as Q, and its length is also m. In addition, the a cappella generation module requires melody information to ensure that the lyrics are sung at the correct pitch. In the previous step, the present invention obtains a note quadruple sequence Y, which includes a note sequence U and a note duration sequence V. They correspond one-to-one to each lyric, so the length of U and V are both n. In order to generate a complete song, the present invention also requires fundamental frequency information F0, whose length is equal to the number of frames T of the generated audio. This F0 must also be inferred by a machine learning model. The variance encoder is a speech acoustic encoder based on FastSpeech2 (Ren Y, Hu C, TanX, et al. Fastspeech 2: Fast and high-quality end-to-end text to speech[J].arXiv preprint arXiv:2006.04558, 2020), which is composed of Transformer layers. It takes P as input and generates speech acoustic features from the last hidden layer. .

[0056]

[0057] The present invention uses the duration predictor of FastSpeech2 to estimate the duration of each phoneme. The core component of the predictor is a convolutional network composed of two layers of one-dimensional convolution (LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition [J]. Proceedings of the IEEE, 1998, 86 (11): 2278-2324.). It uses note information U, V and speech acoustic features as input.

[0058]

[0059] During the training process, the present invention optimizes the predictor by computing the L2 norm of the phoneme duration.

[0060]

[0061] The F0 predictor is based on a diffusion model. It first determines the total number of frames T of generated audio by adding the phoneme duration Q and creates a mapping matrix that connects the phonemes with the Mel spectrograms. It then uses this mapping matrix to extend the length of the acoustic features from m to T. Finally, it uses the speech acoustic features from 1 to T. As the condition, from the original noise conduct The F0 predictor is implemented by Wavenet (Van Den Oord A, Dieleman S, Zen H, et al. Wavenet: A generative model for raw audio[J]. arXiv preprint arXiv:1609.03499, 2016, 12), which is a fully probabilistic autoregressive model composed of causal convolutional layers. During training, the F0 predictor uses the L2 norm as its loss function.

[0062]

[0063] After obtaining the phoneme duration Q and F0, the present invention will generate a cappella Mel spectrum M consisting of T frames. First, the present invention uses an acoustic encoder to obtain an acoustic representation The input of the acoustic encoder includes phoneme P, phoneme duration Q and F0, which indicates that It captures the key information of singing skills. The basic structure of the acoustic encoder is the same as that of the variance encoder.

[0064]

[0065] Then, the present invention converts the human voice feature The mel spectrum of the singing voice is reconstructed as a conditional input to the diffusion model. The standard diffusion model inference process is summarized as follows, where represents the initial Gaussian noise, and R represents the number of denoising iterations.

[0066]

[0067] In order to improve the quality of the final audio output and speed up the inference process of the diffusion model, the present invention implements the shallow diffusion mechanism introduced by DiffSinger. This mechanism uses a ConvNeXt-based decoder (Liu Z, Mao H, Wu CY, et al. A convnet for the 2020s[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition. 2022: 11976-11986), which consists of deep convolutional blocks and derives the spectrum at the Kth denoising step. Therefore, the denoiser does not perform R rounds of denoising on the original noise, but rather Start K rounds of denoising.

[0068]

[0069] The basic model of the denoiser is the same as the F0 predictor. During the training phase, the invention enhances the auxiliary decoder by calculating the L1 norm of the Mel spectrum, while the denoising uses the L2 norm.

[0070]

[0071] The configuration file created in this step defines the singing style of the music generation model and can also be used to change the melody MIDI file generated by the template to melody submodule. Since the music generated by the model may not suit everyone's taste, a music generation model that allows flexible control of the output content is more suitable for everyone. Therefore, users can access the configuration file generated by SongSong. If the user is not satisfied with the generated melody, the configuration file can be modified independently to adjust the generated melody audio and a cappella audio at the same time.

[0072] Step 3, melody to accompaniment generation. After completing the first two steps, the a cappella audio and melody audio are obtained, which can be merged to generate a complete music track. In order to enrich the final music output, additional generative models can be used to create accompaniments played by other instruments.

[0073] According to the method proposed by Copet et al. (Copet J, Kreuk F, Gat I, et al. Simple and controllable music generation[J]. Advances in Neural Information Processing Systems, 2024, 36.), the present invention implements an architecture that discretizes acoustic feature information and generates discrete acoustic tags for accompaniment generation through an autoregressive language model. The input includes text prompts and audio prompts, where the audio prompts are melody audio. The continuous audio features are quantized using the encoder of Encodec (Défossez A, Copet J, Synnaeve G, etal. High fidelity neural audio compression[J]. arXiv preprint arXiv:2210.13438, 2022). During inference, the text tags and audio tags are combined and input into the autoregressive Transformer model to generate accompaniment audio tags. Finally, the decoder of Encodec is used to convert the tag into a continuous Mel spectrogram.

[0074] As a preferred embodiment, the deep learning framework used in the embodiment of the present invention is Pytorch, version 2.1.0, and CUDA version 12.1. The hardware environment of the experiment is RTX 3090 graphics card, and the processor is Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz. The specific implementation process of the model device for restoring Song Ci music described in the embodiment of the present invention is as follows: Step 1: Build a large-scale, multi-purpose Song Dynasty music dataset OpenSongSong.

[0075] In step 2, the lyrics-to-template module predicts the rhythm of each word in the poem and then integrates this rhythm into a template containing specific key and chord details.

[0076] Step 3: The template is processed by the template-to-melody module to determine the note corresponding to each word.

[0077] Step 4, convert the note sequence into a melody MIDI file and generate a configuration file. The configuration file contains information about the phonemes, notes, and fundamental frequency (F0) of the audio to be created, which is used to generate a cappella audio.

[0078] Step 5, the accompaniment generation module uses the melody audio to create the accompaniment played by other instruments.

[0079] Step 6, generate complete Song Ci music audio by combining the a cappella, melody and accompaniment audio.

[0080] The specific implementation plan is as follows: 1. Experimental Setup Currently, there are limited open source systems that can generate audio that combines a cappella and instrumental. Therefore, the present invention selected two leading commercial music generation software, Suno and SkyMusic, for comparison. These systems use the powerful generation capabilities of large language models to generate high-quality music, including a cappella and instrumental, based on simple text prompts. In addition, they allow users to input audio references to generate music of similar styles.

[0081] The method proposed in the embodiment of the present invention adopts objective and subjective indicators. The objective indicator adopted is Frechet audio distance (Kilgour K, Zuluaga M, Roblek D, et al. Frechet audio distance: Ametric for evaluating music enhancement algorithms[J]. arXiv preprint arXiv:1812.08466, 2018), which measures the Frechet distance between the embedding value distributions of two audio groups, thereby providing a generated audio quality assessment that is closer to human perception. The embodiment of the present invention uses VGGish and PANN models (Kong Q, CaoY, Iqbal T, et al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition[J]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2020, 28: 2880-2894) to extract audio embedding. In terms of subjective evaluation, the present invention evaluates music structure, richness of the soundtrack, continuity of motivation, sound quality, accuracy of pronunciation, naturalness of sound, fit with the style of Song Ci, and fit with accompaniment. The first four indicators are evaluated from the perspective of ordinary music, while the last four indicators evaluate the generated music from the perspective of Song Ci music.

[0082] The present invention randomly selected 50 songs from the data set, which were not part of the training data used to build OpenSongSong. After basic segmentation to eliminate long periods of silence, the present invention finally obtained 85 audio segments, totaling 1.8 hours. The singers in the test set are male. The testing process includes two stages. The first stage is a zero-sample test, in which only text prompts and Song lyrics are provided to Suno and SkyMusic, and the prompt content is "Chinese classical style, Song Dynasty, Guzheng, male voice". The second stage is a few-sample test, in which Suno and SkyMusic receive an additional Song work as a reference. The present invention invited 9 experts from music schools to participate in the subjective evaluation.

[0083] 2. Training details The lyrics to template and template to melody modules share the same Transformer architecture, both consisting of 4 layers of encoder blocks and 4 layers of decoder blocks. Each block contains 4 attention heads and 256 linear units. In the a cappella generation module, the acoustic encoder and variance encoder of the present invention are based on FastSpeech2 and include 4 feedforward Transformers. The duration predictor consists of a 5-layer one-dimensional convolutional network with a convolution kernel size of 3, and the input and output dimensions are both 512. The auxiliary decoder consists of a 6-layer ConvNeXt decoder, each with a convolution kernel size of 7, and the input and output dimensions are both 512. The denoiser and F0 predictor are both based on Wavenet, containing 20 layers of convolutions with a convolution kernel size of 3. The input and output dimensions of each layer of the denoiser are 512 and 1024, respectively, while the dimension of the F0 predictor is half of that of the denoiser. For the accompaniment generation module, the present invention uses 1.5B-Musicgen-melody as the initial parameter.

[0084] The present invention uses the data of the speaker with the largest corpus in OpenSongSong (about 3.5 hours) to train the lyrics to template, template to melody and a cappella generation modules of SongSong. All training processes use the Adam optimizer with a learning rate set to 5e-4. Training is performed on a GeForce RTX-3090, and the maximum number of training steps for each module is 160,000.

[0085] 3. Experimental results The comparison results of SongSong, Suno, and SkyMusic are shown in Table 1. Zero-shot and Few-shot indicate whether the Song Ci audio is provided as a reference for Suno and SkyMusic. In terms of subjective indicators, MS represents the music structure, ER represents the richness of the soundtrack, MC represents the continuity of motivation, PA represents the accuracy of pronunciation, VN represents the naturalness of the sound, SQ represents the sound quality, SCS represents the fit with the Song Ci style, and AC represents the fit with the accompaniment.

[0086] Table 1 Comparison results of SongSong, Suno and SkyMusic

[0087] When conducting a subjective evaluation of Song Ci music, the present invention found that Suno and SkyMusic performed poorly in terms of prompt acceptance and could not understand the Chinese classical style well. The generated music almost all presents the style of pop music, and the style consistency is far from that of Song Ci music. In addition, the generated music accompaniment does not use the guzheng set by the present invention, but uses instruments commonly used in pop music such as guitars and electronic keyboards. Even if Song Ci music is input as a reference, the problem is still not improved. Therefore, Suno and SkyMusic scored much lower than SongSong in the SCS (fitness of Song Ci style) and AC (fitness of accompaniment) indicators. After training on the Song Ci music corpus of OpenSongSong, SongSong can fully grasp the pitch and rhythm of Song Ci music and generate singing and accompaniment that are more in line with the characteristics of Song Ci music. In addition, due to the presence of many uncommon characters in Song Ci, these two models based on pop music training cannot sing each word clearly and accurately, and sometimes even miss words or randomly generate lyrics, resulting in low scores on the PA (accuracy of pronunciation) indicator. After training with a small number of samples, SkyMusic's pronunciation accuracy increased, while Suno's pronunciation accuracy showed a downward trend. However, this issue did not significantly affect the VN (voice naturalness) scores of Suno and SkyMusic. Under the pre-training scale and large parameter model, Suno and SkyMusic can still maintain the naturalness of the original performance to a certain extent, so their VN scores are not lower than SongSong. SongSong has undergone strict lyrics and melody alignment, and the training corpus already contains rare characters that may appear in Song Ci music, so it can not only sing strictly according to the input Song Ci, but also ensure that each word is pronounced correctly. Therefore, it far exceeds the other two models in PA indicators and is comparable to them in VN indicators. However, if subjectively evaluated from the perspective of ordinary music, SongSong's performance is not as good as Suno and SkyMusic. In addition to the lack of model scale and training data capacity, there are multiple reasons for this phenomenon. The sentence structure and length of Song Ci are relatively fixed, and the performance style is relatively single. However, Suno and SkyMusic regarded it as a pop song, and used structural optimization methods such as repetition, contrast, and pitch change to generate more complex and coherent pop music, thus performing better in MS (music structure) and MC (motivational continuity); Suno and SkyMusic had poor controllability. Although the prompts input by the present invention required them to use only the guzheng accompaniment, they still used other instruments for accompaniment, resulting in a higher complexity of the arrangement, so their ER (richness of the soundtrack) was higher. After training with a small number of samples, the MS (music structure) and ER (richness of the soundtrack) scores of Suno and SkyMusic decreased, while MC (motivational continuity) remained unchanged.

[0088] This observation supports our analysis that the closer the generated music is to Song poetry, the more monotonous and consistent its style is. In terms of sound quality, due to the lack of high-quality Song Dynasty music, it is difficult for the model trained by our invention to generate music with extremely high sound quality. In terms of objective evaluation indicators, SongSong’s Relatively high, but In this respect, it is comparable to Suno and SkyMusic. Since the audio for comparison is the audio of Song poetry, objectively speaking, SongSong can also generate high-quality audio of Song poetry that conforms to human auditory perception. The statistical results of the standard deviation of subjective evaluation are shown in Table 2.

[0089] Table 2 Standard deviation statistics of subjective evaluation

[0090] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a model construction device for restoring Song Ci music, which is used to execute a model construction method for restoring Song Ci music in the above method embodiment.

[0091] The device comprises: a first main module, which is used to construct a Song Ci music data set; a second main module, which is used to construct a Song Ci music restoration model, including: a lyrics to melody generation module, which is used to generate a melody according to an input Song Ci; a melody to a cappella generation module, which is used to generate a cappella according to the melody; a melody to accompaniment generation module, which is used to generate an accompaniment according to the melody; a combined restoration module, which is used to combine the melody, the a cappella and the accompaniment to generate a complete musical work; and a third main module, which is used to train the constructed Song Ci music restoration model using the Song Ci music data set to obtain a trained Song Ci music restoration model.

[0092] An embodiment of the present invention provides a model building device for restoring Song Ci music, which adopts a melody generation module based on template transition to generate a melody that is strictly aligned with the lyrics and conforms to the style of Song Ci, and guides subsequent music generation based on the melody to ensure the consistency of a cappella and accompaniment, and then uses singing generation combined with music parameters and discrete accompaniment generation respectively to generate high-quality Song Ci music that is sung strictly according to the input Song Ci and conforms to the style of Song Ci.

[0093] It should be noted that the device embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned device embodiments provided by the present invention. As long as technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned device embodiments to obtain corresponding device class embodiments for implementing the methods in other method class embodiments.

[0094] Based on the same inventive concept as the above-mentioned method embodiment, an embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the steps of the model construction method for restoring Song Ci music.

[0095] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A model construction method for Song Ci music restoration, characterized in that: include: Construct a Song Dynasty music dataset; Constructing a Song Ci music restoration model, including: a lyrics to melody generation module, used to generate a melody according to the input Song Ci; a melody to a cappella generation module, used to generate a cappella according to the melody; a melody to accompaniment generation module, used to generate an accompaniment according to the melody; a combined restoration module, used to combine the melody, a cappella and accompaniment to generate a complete musical work; The Song Ci music data set is used to train the constructed Song Ci music restoration model to obtain a trained Song Ci music restoration model.

2. A model construction method for restoring Song Ci music according to claim 1, characterized in that: Constructing the Song Ci music dataset includes annotating the collected dataset as follows: Perform loudness matching on each song in the dataset to balance the volume of the dataset; Slice the song based on the vocals in the song and control the length of the data strip within the preset duration; Separate the vocals and accompaniment of each data, and extract the MIDI file; Text annotation of vocal data is performed based on the lyrics of each song collected in advance, and a phoneme-based maximum string matching algorithm is used to extract the most suitable text for the current vocal from the lyrics.

3. A model construction method for restoring Song Ci music according to claim 2, characterized in that: Constructing the Song Ci music dataset includes cleaning the collected dataset as follows: Use the voice activity detection model to detect the data without voice in the human voice dataset, and delete these data and their corresponding accompaniment audio and MIDI files; Manual inspection was performed to delete data with text annotation errors and unsuccessful voice separation.

4. The model building method for restoring Song Ci music according to claim 1 is characterized in that: The lyrics to melody generation module includes: Predicting the rhythm of each word in the lyrics and integrating the rhythm of each word into a template, the template including tonality, chords, rhythmic patterns and cadences; The note corresponding to each word is determined based on the input of the template, a note sequence corresponding to the lyrics is generated, and the note sequence is converted into a melody.

5. The model building method for restoring Song Ci music according to claim 1 is characterized in that: The melody to a cappella generation module includes: Generate speech acoustic features according to the input speech sequence; Determine the duration of each phoneme based on the note sequence, the note sequence duration sequence and the acoustic features of the speech; Determine fundamental frequency information based on machine learning; Generate a cappella Mel-spectrogram based on phonemes, phoneme duration and fundamental frequency information.

6. A model building method for restoring Song Ci music according to claim 5, characterized in that: Determine fundamental frequency information based on machine learning, including: Determine the total number of frames of generated audio based on the phoneme duration and create a mapping matrix connecting the phonemes to the Mel spectrograms; Expanding the length of the acoustic feature to the total number of audio frames according to the mapping matrix; According to preset conditions, multiple rounds of denoising are performed from the original noise to reconstruct the pitch sequence, and the fundamental frequency information is determined based on the reconstructed pitch sequence.

7. A model building method for restoring Song Ci music according to claim 5, characterized in that: The melody to a cappella generation module also includes: creating a configuration file for defining the singing style of the Song Ci music restoration model, or for changing the melody MIDI file generated by the template to melody submodule.

8. The model building method for restoring Song Ci music according to claim 1 is characterized in that: The melody to accompaniment generation module includes: Input text prompts and audio prompts, wherein the audio prompts are melody audio; Use an encoder to quantize the input continuous audio features to obtain text tags and audio tags; During inference, the text tokens and audio tokens are combined and input into an autoregressive Transformer model to generate the accompaniment audio tokens. The accompaniment audio tokens are converted into continuous mel-spectrograms using a decoder.

9. A model building device for restoring Song Ci music, characterized in that: include: The first main module is used to construct a Song Dynasty music dataset; The second main module is used to construct a Song Ci music restoration model, including: a lyrics to melody generation module, used to generate a melody according to the input Song Ci; a melody to a cappella generation module, used to generate a cappella according to the melody; a melody to accompaniment generation module, used to generate an accompaniment according to the melody; a combined restoration module, used to combine the melody, a cappella and accompaniment to generate a complete musical work; The third main module is used to train the constructed Song Ci music restoration model using the Song Ci music data set to obtain a trained Song Ci music restoration model.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the steps of a model building method for restoring Song Ci music as described in any one of claims 1 to 8.

Images