A music generation method based on a large pre-training language model
By fine-tuning and preprocessing GPT-3, and combining a left-to-right sequence method with a deep hierarchical strategy, melodies with long-term structure and musicality are generated. This addresses the shortcomings of large pre-trained language models in melody generation and achieves efficient melody generation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)
- Filing Date
- 2023-11-14
- Publication Date
- 2026-06-02
AI Technical Summary
Existing large-scale pre-trained language models have not been effectively utilized in melody generation, making it difficult to generate melodies with long-term structure and musicality.
By fine-tuning GPT-3 and using thousands of melody MIDI files, melodies are encoded into symbol sequences. A left-to-right sequence approach and a deep hierarchical strategy are employed, combined with data preprocessing and similarity error assessment, to generate music with realistic structure.
It enables the generation of melodies with long-term structure and musicality, can simulate different melody generation methods, and can generate melodies of similar style with less data.
Smart Images
Figure CN117496926B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of language processing technology, and in particular to a method for music generation based on a large pre-trained language model. Background Technology
[0002] Research on melody generation based on deep learning has yielded many noteworthy results. Currently, an increasing number of studies employ language models from Natural Language Processing (NLP) to generate melodies. Dedicated architectures for generating long-term coherent melodies have emerged, including efficient attention mechanisms, enhanced memory networks, and large-scale deep neural networks. Simultaneously, various symbolic musical representations of MIDI help models better learn long-range dependencies in melody generation. Furthermore, recent research has utilized hierarchical generation strategies based on deep learning. These strategies first generate hierarchical musical structure representations of the melody, including measure-level musical structure relationship graphs. However, state-of-the-art large-scale pre-trained language models in NLP have been largely neglected in this context. Summary of the Invention
[0003] In view of the defects or deficiencies in the existing technology, the technical problem to be solved by the present invention is to provide a music generation method based on a large pre-trained language model.
[0004] To achieve the above objectives, the technical solution adopted by the present invention is to provide a music generation method for large-scale pre-trained language models, comprising the following specific steps:
[0005] The GPT-3 was fine-tuned using thousands of melody MIDI files, and then the fine-tuned model was used for melody generation.
[0006] Encode the melody into a sequence of symbols;
[0007] Represented using strings of GPT-3 fine-tuning data;
[0008] A sequence method and a deep hierarchical strategy method are used to generate sequences from left to right by adjusting the fine-tuning data format.
[0009] The ability of a large pre-trained language model to generate melodies was verified using the publicly available dataset Wikifonia and the aforementioned generation methods.
[0010] The ability of the model to generate music with realistic structure is evaluated by calculating the error between the similarity distribution of the training data and the generated music.
[0011] As a further improvement of the present invention, the symbols include five types: measure, position, note, chord, and rhythm. For measure and position, a measure symbol is used to represent the bar line, and a position symbol is used to represent the start of a note or chord event. Mixed-precision time quantization is used, integrating the time resolution of straight notes, triplets, and MIDI files. Specifically, the time quantization is a set of {0, 30, 60, ..., 1890} ∪ {0, 40, 80, ..., 1880}, and uses...<Pos_Value> Symbols are used to represent various starting positions of 96;
[0012] For notes, the pitch ranges from 48 (C3) to 83 (C5), the dynamic range ranges from 0 to 127, and the note value ranges from {30, 60, 90, ..., 1920} to {40, 80, 160, 320, 640}.
[0013] Use compound words<Pitch_Value、Velocity_Value、Duration_Value> Compress these three attributes of a note into a single tag;
[0014] As a further improvement of the present invention, the GPT-3 fine-tuning data string is a multi-line string, in which each line consists of three space-separated numeric characters, representing the event symbol, velocity, and duration from left to right respectively; when the event symbol is not a note, the velocity and duration are represented by the character "0", and the multi-line string is arranged in the order of bar event line, position event line, and note event line; a newline character "SEP" is added to the end of each measurement, and another newline character "END" is added to the end of each measurement.
[0015] As a further improvement of the present invention, the sequence method: For the sequence method, during the training phase, we set the first 4 measures as prompts and the remaining measures as completions. During the testing phase, we provide the model with the first 4 measures of the melody in the test set as input and generate the rest. Similar to text generation in NLP, the model can stop at any number of measures, depending on its understanding of the melody symbols.
[0016] The deep hierarchical strategy involves: first, generating a melody skeleton; then, repairing the melody skeleton to obtain the complete melody. For melody skeleton generation, during the training phase, the first four measures of the melody skeleton are used as input, and the remaining measures are used as completion. During the testing phase, the model is provided with the first four measures of the melody skeleton from the test set as input, and the remaining parts are generated. For melody skeleton repair, during the training phase, the original rhythm skeleton is used as a cue, and the corresponding melody is used as completion. During the testing phase, the original rhythm skeleton and the rhythm skeleton generated by the fine-tuned model are used for evaluation, with the first 24 measures of the rhythm skeleton provided as a cue.
[0017] As a further improvement of the present invention, the data preprocessing method cleans up noisy data and corrects errors in manual annotation, retaining only melodies in 4 / 4 time. A mixed-precision time quantization network is used to force the MIDI files to be converted to the keys of "C major" and "A minor". Octave transposition is applied to all melodies, shifting the pitch to the range from C3 to C5. After preprocessing, the training set consists of 2888 melodies, and the test set consists of 50 randomly selected melodies.
[0018] As a further improvement to the present invention, a sequence generation method was used, and the effects of fine-tuning using only 500 melodic data and fine-tuning using all melodic data were compared. Then, we used a deep layering method to first fine-tune GPT3 to generate a melodic skeleton, and then fine-tune GPT3 to repair the melodic skeleton. During the repair stage, the effects of using the original melodic skeleton and using the generated melodic skeleton were tested.
[0019] As a further improvement of the present invention, for each song in the training set, the similarity between each pair of measures is calculated, and is defined as:
[0020]
[0021] Where N(i) is the sequence of notes in the i-th measure. Two notes are considered equal if their pitch, duration, and starting position within the measure are all the same. i,j The value ranges from 0.0 to 1.0. If two subsections are completely identical, the value is equal to 1.0. Then, the average similarity of subsection pairs with an interval of t on the training set D is calculated using the following formula:
[0022]
[0023] The similarity error between the generated melody set and the test set is defined as:
[0024]
[0025] In our experiment, we set T=32. It is calculated based on 50 music clips generated by each model.
[0026] The beneficial effects of this invention are as follows: This invention is based on the large-scale language model GPT-3 pre-trained on a massive text corpus. It fine-tunes GPT-3 using thousands of melody MIDI files, and then uses the fine-tuned model to generate melodies. The main advantages of this method are as follows:
[0027] (1) The algorithm can learn the long-term dependency structure of melodies and generate music with long-term structure and musicality;
[0028] (2) The algorithm can simulate different melody generation methods by adjusting the fine-tuning data format;
[0029] (3) The algorithm allows for the generation of melodies with similar styles using only a small amount of data. Attached Figure Description
[0030] Figure 1 This is the musical score of the melody generated by the finely tuned GPT-3 according to the present invention;
[0031] Figure 2 This is a schematic diagram of the representation of the melody sample used for fine-tuning GPT-3 according to the present invention;
[0032] Figure 3 This invention is a box plot that displays all indicators;
[0033] Figure 4 This is a similarity distribution diagram of the structural correlation strips between the four evaluation sets and the training and test sets generated using four different methods according to the present invention. Detailed Implementation
[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0035] like Figure 1 As shown, this invention provides a method for fine-tuning a large pre-trained language model, GPT-3, using thousands of melody MIDI files, and then using the fine-tuned model for melody generation. Depending on the fine-tuning data format, the algorithm can simulate both a left-to-right sequence generation method and a melody generation method based on a deep hierarchical strategy. Experimental results show that GPT-3 can generate melodies with long-term structure and musicality using both methods.
[0036] The algorithm uses MeMIDI to encode the melody as a sequence of symbols, including five symbol representations: measure, position, note, chord, and rhythm. For measures and positions, measure symbols are used to represent bar lines, and position symbols are used to indicate the start of note or chord events. Mixed-precision time quantization is used, combining the time resolution of straight notes, triplets, and the MIDI file. Specifically, the time quantization is a set of {0, 30, 60, ..., 1890} ∪ {0, 40, 80, ..., 1880}, and uses...<Pos_Value> Symbols are used to represent various starting positions of 96, for example<Pos_30> Assign a position symbol to each musical note. For notes, the pitch ranges from 48 (C3) to 83 (C5), the dynamic range is 0 to 127, and the note value is the set of {30, 60, 90, ..., 1920} ∪ {40, 80, 160, 320, 640}. Use compound words.<Pitch_Value、Velocity_Value、Duration_Value> Compress these three attributes of a note into a single tag to shorten the length of the melody event sequence.
[0037] A string representation of GPT-3 fine-tuning data is proposed. Essentially, it is a multi-line string, where each line consists of three space-separated numeric characters. These characters, from left to right, represent the event symbol, tempo, and duration, respectively. When the event symbol is not a note, the tempo and duration are represented by the character "0". For simplicity, chord and tempo events are omitted. The multi-line string is arranged in the order of bar event lines, position event lines, and note event lines. Please refer to [reference needed]. Figure 2 Examples are provided. To help GPT-3 better identify and understand metric boundaries, a newline character "SEP" is added to the end of each metric, and another newline character "END" is added to the end of each metric. End of last section.
[0038] It is the string representation of MeMIDI. '-' replaces spaces, and a space is added before '\n' for easier reading.
[0039] The algorithm can achieve both a left-to-right sequential generation method similar to traditional generative models and a generation method based on a deep hierarchical strategy by adjusting and fine-tuning the data format.
[0040] For the sequence method, during the training phase, we set the first 4 measures as cues and the remaining measures as completions. During the testing phase, we provide the model with the first 4 measures of a melody from the test set as input and generate the rest. Similar to text generation in NLP, the model can stop at any number of measures, depending on its understanding of the melody notation. For the hierarchical strategy method, a melody skeleton is first generated, and then the melody skeleton is repaired to obtain the complete melody. For melody skeleton generation, during the training phase, we set the first 4 measures of the melody skeleton as input and the remaining measures as completions. During the testing phase, we provide the model with the first 4 measures of the melody skeleton from the test set as input and generate the rest. For melody skeleton repair, during the training phase, the original rhythm skeleton is used as a cue, and the corresponding melody is used as a completion. During the testing phase, the model is evaluated using the original rhythm skeleton and the rhythm skeleton generated by the fine-tuned model. Due to the maximum token limit of 2048 (the sum of cues and completions), we only provide the first 24 measures of the rhythm skeleton as cues. Note that GPT-3 may generate new measures beyond the 24 cue measures.
[0041] To verify the capabilities of large pre-trained language models in melody generation, we used the publicly available dataset Wikifonia and employed both of the aforementioned different generation methods.
[0042] We used data preprocessing methods to clean up noisy data and correct errors in manual annotations. Only melodies in 4 / 4 time were retained. A mixed-precision time quantization network was used to force the MIDI files to be aligned. For simplicity, the pitch of these MIDI files was converted to "C major" and "A minor," and octave transposition was applied to all melodies, shifting the pitch to the range from C3 to C5. After preprocessing, the training set consisted of 2888 melodies, and the test set consisted of 50 randomly selected melodies.
[0043] We used a sequence generation method and compared the effects of fine-tuning using only 500 melody data points versus fine-tuning using all melody data. Then, we used a deep layering method. First, we fine-tuned GPT3 to generate the melody skeleton, then fine-tuned GPT3 again to repair the melody skeleton. During the repair phase, we tested the effects of using the original melody skeleton versus using the generated melody skeleton.
[0044] To further validate the quality of music generated by GPT-3 in different tasks, we conducted a subjective evaluation test. We randomly selected ten melodies from evaluation sets generated by four different tasks as listening materials, combining them with corresponding real music, for a total of fifty tracks. We invited ten listeners, five of whom were music professionals, to conduct the evaluation. In the blind listening test, participants were asked to rate each melody from very poor to excellent using a five-point Likert scale across four dimensions: 1) Structure: Does the melody have a clear structure? 2) Richness: Is the melodic content rich? 3) Pleasantness: Does the melody sound pleasant? 4) Overall: Comprehensive quality. Table 1 shows the detailed experimental results.
[0045] Table 1 Subjective evaluation scores (mean ± standard deviation) of melodies generated based on different tasks.
[0046]
[0047] We used similarity error as an objective metric, calculating the error between the similarity distribution of the training data and the generated music to evaluate the model's ability to generate music with realistic structure. Specifically, for each song in the training set, the similarity between each pair of measures was calculated, defined as:
[0048]
[0049] Where N(i) is the sequence of notes in the i-th measure. Two notes are considered equal if their pitch, duration, and starting position in the measure are the same. i,j The value ranges from 0.0 to 1.0. If two sections are completely identical, the value is equal to 1.0. Then, the average similarity of section pairs with an interval of t on the training set D is calculated using the following formula:
[0050]
[0051] The similarity error between the generated melody set and the test set is defined as:
[0052]
[0053] In our experiment, we set T=32. It is calculated based on 50 music clips generated by each model. The smaller the value, the more similar the structure of the generated music is to artificial music. Figure 4 The similarity distribution of structure-related subsections between four evaluation sets generated using four different methods and the training and test sets is shown.
[0054] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A music generation method based on a large-scale pre-trained language model, characterized in that, The specific steps include the following: The GPT-3 was fine-tuned using thousands of melody MIDI files, and then the fine-tuned model was used for melody generation. Encode the melody into a sequence of symbols; Represent the data using strings from the GPT-3 fine-tuning data; By adjusting and fine-tuning the data format, a sequence generation method from left to right and a generation method based on a deep hierarchical strategy were constructed respectively. The ability of a large pre-trained language model to generate melodies was validated using the publicly available dataset Wikifonia, sequence generation methods, and a generation method based on a deep hierarchical strategy. The ability of the model to generate music with realistic structure is evaluated by calculating the error between the similarity distribution of the training data and the generated music. The symbols include five types: measure, position, note, chord, and rhythm. The GPT-3 fine-tuning data string is a multi-line string, where each line consists of three space-separated numeric characters, representing the event symbol, intensity, and duration from left to right; When the event symbol is not a note, the dynamics and duration are represented by the character "0". The multi-line string is arranged in the order of the measure event line, the position event line, and the note event line; a newline character "SEP" is added at the end of each measure, and another newline character "END" is added at the end of each measure. Sequence method: For the sequence method, during the training phase, we set the first 4 measures as cue and the remaining measures as completion. During the testing phase, we provide the model with the first 4 measures of the melody in the test set as input and generate the rest. Similar to text generation in NLP, the model can stop at any number of measures, depending on its understanding of the melody symbols. The deep hierarchical strategy approach first generates a melody skeleton, then repairs the melody skeleton to obtain a complete melody. For melody skeleton generation, during the training phase, the first four measures of the melody skeleton are set as input, and the remaining measures are set as completion. During the testing phase, the model is provided with the first four measures of the melody skeleton in the test set as input, and the remaining parts are generated. For melody skeleton repair, during the training phase, the original melody skeleton is used as a cue, and the corresponding melody is used as completion. During the testing phase, the original melody skeleton and the melody skeleton generated by the fine-tuned model are used for evaluation, and the first 24 measures of the melody skeleton are provided as cue.