Multi-track music generation method and apparatus
By modifying the structure of compound words and using an improved Transformer neural network model, multi-track music was generated, solving the problem of poor multi-track music generation in existing technologies and achieving efficient multi-track music generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-04
- Publication Date
- 2026-04-07
AI Technical Summary
Existing automatic music composition techniques struggle to effectively generate multi-track music, especially given the limited variety of instruments and poor performance in generating long sequences. Furthermore, existing models lack consideration for music theory.
By modifying the structure of compound words, including uneven segmentation and preset encoding methods, a compound word sequence is generated. The compound word structure is generated using MIDI file data and the modified compound word sequence. The compound word sequence and compound word structure are then generated to create a compound word model and obtain multi-track music.
The generation of multi-track music has been achieved, solving the problem of multi-track music generation in existing technologies and improving the effect and efficiency of music generation.
Smart Images
Figure CN115331647B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating multi-track music. Background Technology
[0002] The field of music artificial intelligence has entered the era of deep learning, and one of the most eye-catching topics is automatic music composition. With the rise of deep learning, automatic music composition technology has made great progress, but problems still exist.
[0003] Sequence models such as RNNs (Recurrent Neural Networks), LSTMs (Long Short-Term Memory), GRUs (Gate Recurrent Units), and Transformers can only process single tracks when used for NLP (Natural Language Processing) tasks. However, music (especially multi-instrument music) is usually a sequence of multiple tracks. Many previous works have ignored this problem, using only one instrument to create music.
[0004] There are two solutions for generating multitrack music. The first is to design a new network architecture that allows the model to process relevant information between multiple sequences, such as the MuseGan (Multi-track Sequential Generative Adversarial Networks) model. However, the MuseGan model generates works with a limited and fixed variety of instruments, and its performance in generating long sequences is poor. The second is to design merging strategies to combine the sequences into a single sequence, such as MusicBert (Music Bidirectional Encoder Representations from Transformers) and PopMAG (Pop Music Accompaniment Generation). However, the current merging methods chosen by MusicBert or PopMAG are not suitable for music generation tasks and mostly lack consideration for music theory. Summary of the Invention
[0005] To address the problems existing in the prior art, the present invention provides a method and apparatus for generating multi-track music.
[0006] This invention provides a method for generating multi-track music, comprising:
[0007] Modify the structure of compound words;
[0008] Using MIDI file data and the modified compound word structure, a compound word sequence is generated; the compound word sequence includes a sequence of lexical elements representing instrument attributes;
[0009] The compound word sequence is input into a trained improved Transformer neural network model to obtain multi-track music.
[0010] Optionally, the modification of the compound word structure includes:
[0011] The positional information of the beat in the compound word structure is modified by using an uneven segmentation method and a preset encoding method.
[0012] Optionally, the modification of the compound word structure includes:
[0013] The pitch representation in the compound word structure is modified using the twelve-tone equal temperament.
[0014] Optionally, after generating the compound word sequence, the method further includes:
[0015] The single-instrument sequences within each measure are merged to obtain the multi-instrument representation of each measure.
[0016] Optionally, the trained improved Transformer neural network model is obtained through the following training steps:
[0017] The improved Transformer neural network model is pre-trained to obtain a pre-trained improved Transformer neural network model;
[0018] The pre-trained improved Transformer neural network model is fine-tuned using the selected MIDI file data to obtain a trained improved Transformer neural network model.
[0019] Optionally, the improved Transformer neural network model is obtained by introducing a time offset mechanism, rotation position encoding, and R-Drop strategy on the basis of the Transformer neural network model.
[0020] The present invention also provides a multi-track music generation device, comprising:
[0021] The modification module is used to modify the structure of compound words;
[0022] A generation module is used to generate a compound word sequence using MIDI file data and a modified compound word structure; the compound word sequence includes a sequence of lexical elements representing instrument attributes;
[0023] The acquisition module is used to input the compound word sequence into a trained improved Transformer neural network model to acquire multi-track music.
[0024] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-track music generation method as described in any of the above.
[0025] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-track music generation method as described in any of the above.
[0026] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-track music generation method as described in any of the above.
[0027] The multi-track music generation method and apparatus provided by this invention modify the structure of compound words and add instrument attributes to the compound word sequence, which is beneficial to the generation of multi-track music. It also utilizes an improved Transformer neural network model to generate multi-track music with better performance. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0029] Figure 1 This is one of the flowcharts illustrating the multi-track music generation method provided by the present invention;
[0030] Figure 2 This is the second flowchart of the multi-track music generation method provided by the present invention;
[0031] Figure 3 This is a schematic diagram of the structure of the multi-track music generation device provided by the present invention;
[0032] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0034] Figure 1 This is one of the flowcharts illustrating the multi-track music generation method provided by the present invention, such as... Figure 1 As shown, the present invention provides a method for generating multi-track music, the method comprising:
[0035] Step 101: Modify the structure of the compound word.
[0036] Specifically, because existing compound word structures have problems such as not supporting triplets, weak mathematical relationships between generated notes, and only being able to generate single-track music, it is necessary to modify the compound word structure to better facilitate the generation of multi-track music.
[0037] To support the representation of triplets and to alleviate insufficient data and uneven distribution, the position information of the beat needs to be modified.
[0038] Optionally, the modification of the compound word structure includes:
[0039] The positional information of the beat in the compound word structure is modified by using an uneven segmentation method and a preset encoding method.
[0040] Specifically, uneven segmentation methods, such as 1 / 2, 1 / 3, 2 / 3, 1 / 4, etc., are used to divide each section into beats, thereby obtaining multiple uneven beats within each section. The uneven segmentation method can be selected based on the occurrence position of commonly used words.
[0041] The preset encoding method can be sequential numbering. For example, the first beat in a measure can be encoded as 1, the first beat in a measure can be encoded as 2, and so on, to complete the encoding of each beat in a measure.
[0042] Based on the segmentation and encoding, obtain the position information of the beat in each measure.
[0043] Uneven segmentation can achieve beat lengths of 32nd and 24th notes instead of 16th notes, thereby improving the generalizability and robustness of compound word structures.
[0044] To prevent long-tailed pitch distribution and enhance the mathematical relationships between notes, the pitch representation method needs to be modified.
[0045] Optionally, the modification of the compound word structure includes:
[0046] The pitch representation in the compound word structure is modified using the twelve-tone equal temperament.
[0047] Specifically, the twelve-tone equal temperament is a tuning system that divides a group of notes into twelve semitones, and uses the twelve-tone equal temperament to represent pitch.
[0048] Introducing the twelve equal temperaments into the representation of pitch helps to better understand the relationships between pitches, rather than simply embedding them.
[0049] Step 102: Using MIDI file data and the modified compound word structure, generate a compound word sequence; the compound word sequence includes a sequence of lexical elements representing instrument attributes.
[0050] Specifically, MIDI (Musical Instrument Digital Interface) is a unified communication protocol between electronic musical instruments and between electronic musical instruments and computers. A MIDI file is a descriptive musical language that uses bytes to describe the musical information to be played.
[0051] The MIDI file data is processed using the Python third-party library miditooklit, and mapped to several lists: ticks_per_beat (number of microbeats per beat), max_tick (number of microbeats in the whole song), tempo_changes (tempo changes), time_signature_changes (time signature changes), key_signature_changes (key signature changes), lyrics, and instruments, and then converted into two-dimensional tensor form.
[0052] Different tokens are obtained from the two-dimensional tensor in the list above. Different tokens belong to different types, and different types of tokens have different attributes. After obtaining different types of tokens, a compound word sequence is generated based on the modified compound word structure. The compound word sequence includes tokens for compound word type, chord, bar-beat, pitch, duration, and instrument.
[0053] Considering practicalities, tempo variations have little impact on the music; a global tempo is sufficient, and arbitrary tempos would interfere with the model. Furthermore, from a musical composition perspective, tempo control is the performer's responsibility. Therefore, a globally averaged token only needs to be added at the beginning of the compound word sequence. The compound word sequence itself does not learn to generate tempo tokens, but the global tempo will still help in generating the overall composition.
[0054] Compared to traditional compound word sequences, this method incorporates the "Instrument" attribute to indicate the generation of multi-track music, i.e., multi-instrument music. Categorizing instruments into different types contributes to the stability of the generated music.
[0055] To accommodate the instrument attributes added to the compound word sequence and the generation of multi-track music, a measure-level merging operation is required.
[0056] Optionally, after generating the compound word sequence, the method further includes:
[0057] The single-instrument sequences within each measure are merged to obtain the multi-instrument representation of each measure.
[0058] Specifically, let X represent the musical piece, and S represent the symbol sequence corresponding to the musical piece X.
[0059] For the i-th section Xi of X, we can obtain a sequence of symbols for a single instrument, namely {S(1,i), S(2,i), ..., S(I,i)}, where I represents the number of instruments.
[0060] Add an "Instrument Type" marker before each single-instrument sequence. The "Instrument Type" marker contains only the instrument information, while other information elements are filled with "[Mask]". For other Notes in the single-instrument sequence, the instrument type will be filled with "[Continue]".
[0061] After merging the single instrument sequences, there are multiple merged single instrument sequences within a measure. By splicing together all the single instrument sequences belonging to this measure, a multi-instrument representation of the i-th measure is obtained.
[0062] Inheriting the work of Compound Word, <bar>A marker is placed at the beginning of each measure sequence to indicate the start of that measure. Then, all the single-instrument sequences within a measure are simply concatenated in chronological order.
[0063] By merging the single-instrument sequences within each measure, the instrument attributes added to the compound word sequences are adapted, which is beneficial for the generation of multi-track music.
[0064] Step 103: Input the compound word sequence into the trained improved Transformer neural network model to obtain multi-track music.
[0065] Specifically, Figure 2 This is the second flowchart illustrating the multi-track music generation method provided by the present invention, as shown below. Figure 2 As shown, the improved Transformer neural network model is trained by inputting a compound word sequence generated using MIDI file data into the model. Based on the type of lexical units in the compound word sequence, the model iteratively generates the compound word type, chord, beat, pitch, duration, and instrument in the next compound word sequence.
[0066] A linear transformation is used to map the vectors corresponding to the compound word sequence onto the input dimension of the improved Transformer neural network model. This vector, along with the contextual information from previous iterations, is then input into the improved Transformer neural network, resulting in a feature vector with the same dimensions as the input. This feature vector is then input into the classification part.
[0067] The classification part can be understood as retrieving the next optimal generated result and using it as the output. This consists of multiple linear layers (Linear & Softmax). First, it determines the compound word type of the next note based on the feature vector. Then, it maps the compound word type information into a corresponding vector, concatenates it with the feature vector from the input classification part, and uses a linear transformation to restore the concatenated vector to the input dimension. Next, using this feature vector that integrates the compound word type information, five linear transformations are applied to predict information such as instrument, duration, pitch, beat, and chords, thereby generating the next note.
[0068] The predicted next note's type, chord, beat, pitch, duration, and instrument are mapped to corresponding vectors. These vectors are then concatenated to generate a compound word sequence. This compound word sequence, generated using the predicted next note, is then input into a trained, improved Transformer neural network model. This process is repeated to generate multi-track music.
[0069] Optionally, the trained improved Transformer neural network model is obtained through the following training steps:
[0070] The improved Transformer neural network model is pre-trained to obtain a pre-trained improved Transformer neural network model;
[0071] The pre-trained improved Transformer neural network model is fine-tuned using the selected MIDI file data to obtain a trained improved Transformer neural network model.
[0072] Specifically, firstly, a large-scale dataset of mixed-style MIDI files is input into the improved Transformer neural network model for pre-training. The large-scale MIDI file dataset helps improve the overall performance of the model.
[0073] The pre-training process can involve using a GPU (Graphics Processing Unit) to continuously compute for 10 hours, performing 20 iterations to obtain a pre-trained improved Transformer neural network model, and saving the best model parameters from each iteration to a model parameter file.
[0074] Then, by manually selecting small-scale MIDI file data of a specific style, and based on the pre-trained model, fine-tuning was performed using the selected small-scale MIDI file data. The small-scale MIDI file data was then input into the model for further training until the loss function converged to a low constant value, indicating successful training, thus obtaining a well-trained improved Transformer neural network model.
[0075] Pre-training the model before fine-tuning it improves its stability and facilitates the acquisition of a large number of diverse network models.
[0076] Optionally, the improved Transformer neural network model is obtained by introducing a time offset mechanism, rotation position encoding, and R-Drop strategy on the basis of the Transformer neural network model.
[0077] Specifically, the original basic positional encoding is replaced by Rotary Position Embedding (RoPE). The relationships between words are no longer simple basic trigonometric functions, but rather more complex elementary function relationships obtained through calculation.
[0078] R-Drop (Regularized Dropout) is used to compute the data twice to obtain the corresponding data results. Then, KL divergence (Kullback-Leibler divergence) is calculated on the data results of the two passes. KL divergence is used as an auxiliary loss function to participate in backpropagation, thereby playing a regularization role in the training process.
[0079] The time shift mechanism is used to fuse the contextual information of neighbors and adjust the calculation method of the self-attention matrix.
[0080] In addition, a Mixture of Experts (MoE) system is incorporated to improve training speed and inference efficiency. Each data point is assigned to multiple experts, and the outputs are then summed to obtain the actual output. This significantly increases the number of model parameters without a substantial increase in computational cost.
[0081] exist Figure 2 In this model, the improved Transformer neural network used in the middle consists of N Transformer Blocks that maintain consistent input and output shapes. Within each Transformer Block, a Time Shift mechanism is first used to fuse nearest neighbor contextual information. Then, a linear attention mechanism with rotational position encoding is used to integrate global contextual information from the preceding text needed for the current position. Finally, a Feedforward Neural Network (FNN) is used for nonlinear transformation to obtain the output of the Transformer Block. Furthermore, residual connections and regularization mechanisms are used within the Transformer Block to improve the stability of gradient propagation and the stability of feature vector values.
[0082] An improved Transformer neural network model was constructed by introducing time offset mechanism, rotation position encoding and R-Drop strategy, which improved the model's performance and is beneficial for the generation of multi-track music.
[0083] The multi-track music generation method provided by this invention modifies the structure of compound words and adds instrument attributes to the compound word sequence, which is beneficial for the generation of multi-track music. It also utilizes an improved Transformer neural network model to generate multi-track music with better performance.
[0084] The music generation results of this invention are interpretable. The model can analyze large sections of music features such as structure and repetition, and can also extract and utilize small sections of music with similar characteristics, such as material and motifs. Traditional models take 30-60 seconds to generate a piece of music of about 180 seconds on an RTX3080, with a failure rate of over 90%. Using the method of this invention, a piece of the same length can be generated in just 3 seconds on a typical home computer, with a failure rate of less than 10%.
[0085] Table 1 is the data table of the ablation experiment. In Table 1, the effectiveness of all improvements is verified by two indicators: perplexity and accuracy. The lower the perplexity, the higher the music fluency, and the higher the accuracy, the richer the semantics. The ablation experiment also verified the effectiveness of each design in the method of this invention.
[0086] Table 1 Data from the ablation experiment
[0087]
[0088] The multi-track music generation method and apparatus provided by the present invention are described below. The multi-track music generation method and apparatus described below can be referred to in correspondence with the multi-track music generation method described above.
[0089] Figure 3 This is a schematic diagram of the multi-track music generation device provided by the present invention, as shown below. Figure 3 As shown, the present invention also provides a multi-track music generation device, comprising: a modification module 301, a generation module 302, and an acquisition module 303, wherein:
[0090] Modification module 301 is used to modify the structure of compound words;
[0091] The generation module 302 is used to generate a compound word sequence using MIDI file data and the modified compound word structure; the compound word sequence includes a sequence of lexical elements representing musical instruments;
[0092] The acquisition module 303 is used to input the compound word sequence into the trained improved Transformer neural network model to acquire multi-track music.
[0093] Optionally, the modification module is further configured to modify the position information of the beat in the compound word structure using an uneven segmentation method and a preset encoding method.
[0094] Optionally, the modification module is further configured to modify the pitch representation in the compound word structure using the twelve-tone equal temperament.
[0095] Optionally, the device further includes a merging module, which is used to merge the single instrument sequences within each measure to obtain a multi-instrument representation of each measure.
[0096] Optionally, the device further includes a training module for pre-training the improved Transformer neural network model to obtain a pre-trained improved Transformer neural network model.
[0097] The pre-trained improved Transformer neural network model is fine-tuned using the selected MIDI file data to obtain a trained improved Transformer neural network model.
[0098] Optionally, the improved Transformer neural network model is obtained by introducing a time offset mechanism, rotation position encoding, and R-Drop strategy on the basis of the Transformer neural network model.
[0099] Specifically, the multi-track music generation device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.
[0100] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a multi-track music generation method, which includes: modifying the structure of compound words; generating a compound word sequence using MIDI file data and the modified compound word structure; the compound word sequence including a sequence of lexical elements representing instrument attributes; and inputting the compound word sequence into a trained improved Transformer neural network model to obtain multi-track music.
[0101] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multi-track music generation method provided by the above methods. The method includes: modifying the structure of compound words; generating a compound word sequence using MIDI file data and the modified compound word structure; the compound word sequence including a sequence of lexical elements representing instrument attributes; and inputting the compound word sequence into a trained improved Transformer neural network model to obtain multi-track music.
[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a multi-track music generation method provided by the above methods. The method includes: modifying the structure of a compound word; generating a compound word sequence using MIDI file data and the modified compound word structure; the compound word sequence including a sequence of lexical elements representing instrument attributes; and inputting the compound word sequence into a trained improved Transformer neural network model to obtain multi-track music.
[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0106] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish similar objects, and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited; for example, the first object can be one or more.
[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / bar>
Claims
1. A method for generating multi-track music, characterized in that, include: Modify the structure of compound words; Using MIDI file data and the modified compound word structure, generate compound word sequences; The compound word sequence includes a sequence of lexical terms related to musical instruments; The compound word sequence is input into a trained improved Transformer neural network model to obtain multi-track music. The modification of the compound word structure includes: The position information of the beat in the compound word structure is modified by using an uneven segmentation method and a preset encoding method; The modification of the compound word structure includes: The pitch representation in the compound word structure is modified using the twelve-tone equal temperament.
2. The multi-track music generation method according to claim 1, characterized in that, After generating the compound word sequence, the method further includes: The single-instrument sequences within each measure are merged to obtain the multi-instrument representation of each measure.
3. The multi-track music generation method according to claim 1, characterized in that, The trained improved Transformer neural network model was obtained through the following training steps: The improved Transformer neural network model is pre-trained to obtain a pre-trained improved Transformer neural network model; The pre-trained improved Transformer neural network model is fine-tuned using the selected MIDI file data to obtain a trained improved Transformer neural network model.
4. The multi-track music generation method according to claim 1, characterized in that, The improved Transformer neural network model is obtained by introducing a time offset mechanism, rotation position encoding, and R-Drop strategy on the basis of the Transformer neural network model.
5. A multi-track music generation device, characterized in that, include: The modification module is used to modify the structure of compound words; The generation module is used to generate compound word sequences using MIDI file data and the modified compound word structure; The compound word sequence includes a sequence of lexical terms related to musical instruments; The acquisition module is used to input the compound word sequence into a trained improved Transformer neural network model to acquire multi-track music; The modification of the compound word structure includes: The position information of the beat in the compound word structure is modified by using an uneven segmentation method and a preset encoding method; The modification of the compound word structure includes: The pitch representation in the compound word structure is modified using the twelve-tone equal temperament.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-track music generation method as described in any one of claims 1 to 4.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-track music generation method as described in any one of claims 1 to 4.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multi-track music generation method as described in any one of claims 1 to 4.