Automatic generation method and system of conditional music theme melody based on Transformer

Abstract

The present invention discloses a method and system for automatically generating conditional music theme melody based on Transformer. The method comprises the following steps: converting data in an original data set into a unified data format and screening to obtain a theme music data set; extracting music theme fragments from theme music in the theme music data set; inputting a music theme fragment sequence of each sample in the theme music data set into a Transformer encoder to obtain an encoder output; inputting a complete music sequence of each sample in the theme music data set into a Transformer decoder, wherein the decoder input consists of two parts, namely, the complete music sequence of the sample and the output of the encoder; performing a decoding operation to obtain a music sequence containing the music theme fragments, performing error calculation to update parameters; repeatedly training the model until termination, inputting a user-input music theme melody as a conditional music theme fragment sequence into the model, obtaining a complete music sequence containing the music theme fragment sequence, and saving the file.

Description

Technical Field

[0001] The present invention relates to the field of intelligent music generation, and in particular to a method and system for automatically generating conditional music theme melody based on Transformer. Background Art

[0002] Music is an art form that transcends time, and a beautiful piece of music is composed of many components. Among these many musical expression factors, melody is the most important, arguably the soul of music. A beautiful melody is repetitive, with certain fragments recurring over time. These repetitive fragments often embody the emotions the composer wishes to express, and are therefore often referred to as the musical theme. Traditional automatic music generation methods rarely consider musical themes, resulting in chaotic, lacking harmony and leaving a poor impression. For example, a melody generation method and system proposed by Huang Songguo et al. uses music files in MIDI format as a data set, then extracts features from the MIDI files to obtain melody features, and uses an algorithm generator to process these melody features to obtain the final melody. This method only considers melody features such as note series, note intensity, motive position, and interval, and these features are only at the note level, without considering the arrangement pattern of these notes in the entire melody, resulting in poor regularity and harmony of the final generated music, which cannot reflect the theme style of the music; another example is a method and system for automatic generation of music accompaniment based on an encoding-decoding network proposed by Zhao Zhou et al. The music generation method adopted still only considers the music features at the note level, and does not take the overall structure of the music and the music theme style as considerations in the generation stage. It only relies on the model to extract these potential features through deep learning, but this generation model is often inefficient, and the quality of the music generated is difficult to compare with real music.

[0003] Among the current patent literature, an authorized patent (CN109727590B) provides a music generation method and device based on a recurrent neural network, involving the field of deep learning technology. The method includes the following steps: establishing a recurrent neural network; preprocessing a first note sequence in MIDI format to obtain a training data set; training the recurrent neural network using the training data set to obtain a neural network model; calculating and sorting the individual note events of the input second note sequence using the neural network model and a sampling strategy to output a third note sequence; decoding the third note sequence into a note sequence file in MIDI format; and converting the note sequence file into an output file in audio format. This invention generates music through a neural network model, leveraging the powerful learning and representation capabilities of deep neural networks and a sampling strategy to quickly and effectively obtain high-quality melodies, making it easier for users to generate original melodies and effectively improving the efficiency of music creation.

[0004] However, the invention's drawback is that it uses recurrent neural networks for music generation. These networks suffer from the vanishing gradient problem when processing longer sequences, making them ineffective at learning the corresponding features of longer sequences. Consequently, the music generated by this invention often lacks long-term structural connections, which impacts its pleasing quality and lacks practical value.

[0005] It can be seen that there is an urgent need for corresponding technical solutions in this field. Summary of the Invention

[0006] The technical solution adopted by the present invention overcomes the shortcomings of the existing technology and provides a Transformer-based solution for automatically generating conditional music theme melody.

[0007] The present invention provides a method for automatically generating conditional music theme melody based on Transformer, comprising the following steps:

[0008] Step 1: Convert the data in the original dataset into a unified data format and convert it into a music dataset in MIDI format. Then, filter the MIDI format dataset to select data samples with theme music and filter out non-theme music to obtain the theme music dataset.

[0009] Step 2, extracting music theme fragments from the theme music in the theme music dataset;

[0010] Step 3: Input the music theme fragment sequence of each sample in the theme music dataset into the Transformer encoder, and encode the music theme fragment sequence to obtain the encoder output; the music theme fragment sequence is the initial music representation sequence obtained by processing the original MIDI format music file, and is converted into an integer array through the corresponding vocabulary;

[0011] Step 4: Input the complete music sequence of each sample in the theme music dataset into the Transformer decoder, where the complete music sequence of the sample is the initial music representation sequence obtained after processing the original MIDI format music file and the integer array obtained after the corresponding vocabulary conversion;

[0012] The decoder input consists of two parts: a complete music sample sequence and the encoder output. The decoder input sequence is decoded to obtain a music sequence containing the music theme fragment. Finally, the error between the decoder output sequence and the original sample sequence is calculated, and the back-propagation algorithm is used to synchronously update the encoder and decoder parameters.

[0013] Step 5, return to repeat steps 3 and 4, and repeatedly train the model until the model reaches the training termination condition; input the user-input music theme melody as the conditional music theme fragment sequence into the model, obtain a complete music sequence containing the music theme fragment sequence, and save it as a specified music format file.

[0014] Moreover, the implementation method of step 1 is as follows: for the MIDI format music dataset after format conversion, each piece of MIDI format music is taken as a sample, the sample is encoded to obtain an initial music representation in the REMI format, divided according to bars, with two bars as a segment, each sample is divided into several segments, and then the number of repetitions of each segment in the sample is calculated, and the segment with the most repetitions in the sample is taken as the representative segment. If the number of repetitions of the representative segment reaches a preset threshold, it is determined that the sample has a music theme; all samples in the dataset are divided to obtain a theme music dataset.

[0015] Moreover, the implementation method of step 2 is to take each theme music in MIDI format as a sample, encode the sample, obtain the initial music representation in the format of REMI, divide it according to bars, take two bars as a segment, divide each sample into several segments, embed the divided samples into words, obtain the corresponding word vector, and then use a clustering algorithm for each sample word vector to cluster the same or similar segments in the word vector to obtain multiple clusters, select the cluster with the largest number of segments as the theme cluster, and then take the segment closest to the cluster center in the theme cluster as the music theme segment.

[0016] Moreover, step 3 is implemented in such a way that the Transformer encoder consists of 6 identical encoding layers, each of which consists of a multi-head self-attention mechanism and a feedforward layer, and uses superposition and regularization operations.

[0017] Moreover, the decoder consists of 6 identical decoding layers, each of which consists of a multi-head self-attention mechanism, a cross-attention mechanism and a feed-forward layer, and uses stacking and regularization operations.

[0018] On the other hand, the present invention provides a Transformer-based automatic generation system for conditional music theme melody, which is used to implement the above-mentioned Transformer-based automatic generation method for conditional music theme melody.

[0019] Moreover, it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a Transformer-based conditional music theme melody automatic generation method as described in any of the above items.

[0020] Furthermore, the invention comprises a readable storage medium on which a computer program is stored. When the computer program is executed, the method for automatically generating a conditional music theme melody based on Transformer as described above is implemented.

[0021] Advantages of the present invention include:

[0022] 1) A music generation method that incorporates musical themes and styles is proposed, taking the theme of the music as the generation condition. The resulting music is superior to other methods in terms of both the subtlety of note-level features and the overall integrity of the melody, thus improving the quality of the generated music.

[0023] 2) Under the condition that the data set has sufficient data, the generated music with theme styles is more diverse and can meet the user's personal requirements for music theme styles. It can not only bring creative inspiration to professional musicians, but also provide music theme style customization services for ordinary users.

[0024] 3) The REMI music event representation, Transformer model, and MIDI format music dataset mentioned in the specific implementation scheme of the present invention are all open source resources and are easy to obtain, which reduces the difficulty of implementing the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of filtering the original music data set to obtain the theme music data set according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of extracting music theme segments from a theme music dataset according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of encoding an original music file to obtain a music sequence according to an embodiment of the present invention;

[0028] Figure 4 1 is a model structure diagram of a Transformer-based theme music generation method according to an embodiment of the present invention;

[0029] Figure 5 2 is a schematic diagram of the structure of a theme music generation system based on Transformer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0031] The embodiment of the present invention provides a method for automatically generating conditional music theme melody based on Transformer, comprising the following steps:

[0032] Step 1: Convert the data in the original dataset into a unified data format and convert it into a music dataset in MIDI format. Then, filter the MIDI format dataset to select data samples with theme music and filter out non-theme music to obtain the theme music dataset.

[0033] Furthermore, the present invention proposes a preferred solution: for the MIDI format music data set after format conversion, each piece of MIDI format music is taken as a sample, the sample is encoded to obtain an initial music representation in the REMI format, divided according to bars, with two bars as a segment, each sample is divided into several segments, and then the number of repetitions of each segment in the sample is calculated, and the segment with the most repetitions in the sample is taken as the representative segment. If the number of repetitions of the representative segment reaches a preset threshold (for example, 4 times) or above, it is determined that the sample has a music theme; all samples in the data set are divided to obtain a theme music data set.

[0034] like Figure 1 As shown in the embodiment, the original music data set is preprocessed to obtain the theme music data set. Specifically, the preprocessing includes the following steps:

[0035] 1) Obtain an original music dataset and convert the music files in the original music dataset into music files in MIDI format to obtain a music dataset in MIDI format.

[0036] 2) Filter the music in the MIDI format music dataset to obtain the theme music dataset. The specific filtering process is as follows: first, each music file in the MIDI format music dataset is converted into a REMI event sequence, and the REMI event sequence is divided into two-bar units to obtain a number of REMI event sequence segments of equal length. The number of repetitions of these segments is then calculated, and the segment with the largest number of repetitions is selected as the representative segment. If the number of repetitions of the representative segment is greater than or equal to 4, the music file is determined to belong to the theme music. All the music in the dataset is filtered to obtain the theme music dataset.

[0037] Step 2, extracting music theme fragments from the theme music in the theme music dataset;

[0038] Furthermore, the present invention proposes a preferred solution: taking each theme music in MIDI format as a sample, encoding the sample to obtain an initial music representation in REMI format, dividing the music into several segments according to measures, with two measures as a segment, performing word embedding on the divided samples to obtain corresponding word vectors, and then using a clustering algorithm on each sample word vector to cluster the same or similar segments in the word vectors to obtain multiple clusters, selecting the cluster with the largest number of segments as the theme cluster, and then taking the segment closest to the cluster centroid in the theme cluster as the music theme segment;

[0039] like Figure 2 As shown, in the embodiment, for a theme music dataset, it is necessary to obtain the music theme segments of all the music in the dataset. Specifically, the acquisition process includes the following steps:

[0040] 1) Each theme music sample in the theme music dataset is encoded to obtain a REMI event sequence. At the same time, the obtained REMI event sequence is divided into two sections to obtain several REMI event sequence fragments of equal length. The REMI vocabulary is used to find the corresponding event number to obtain the corresponding integer array of REMI event number. Then, these integer arrays are word embedded to obtain several word vector fragments of equal length.

[0041] 2) Use a clustering algorithm on the obtained theme music word vector fragments to divide the same or similar word vector fragments into the same cluster, obtain several different clusters, and calculate the number of word vector fragments in each cluster. Then, take the cluster with the largest number of word vector fragments as the theme cluster, and take the fragment closest to the cluster centroid in the theme cluster as the theme fragment of the music.

[0042] Step 3: Input the music theme fragment sequence of each sample in the theme music dataset into the encoder of the Transformer, where the music theme fragment sequence is the initial music representation sequence in REMI format obtained after the original MIDI format music file is processed by the Python toolkit, and is converted into an integer array through the corresponding vocabulary.

[0043] Furthermore, the present invention proposes a preferred solution: the encoder consists of 6 identical encoding layers, each encoding layer is composed of a multi-head self-attention mechanism and a feedforward layer, and uses superposition and regularization operations.

[0044] like Figure 3 As shown in FIG. 1 , a schematic diagram of encoding an original music file to obtain a music sequence in an embodiment is provided, which is used to encode a music file in a music data set to obtain a music sequence in a corresponding format. The specific encoding process includes the following steps:

[0045] 1) Obtain the original music clip 301 from the music file.

[0046] 2) Use the Python toolkit to parse the original music clip and obtain the REMI event sequence 302.

[0047] 3) For each event in the REMI event sequence 302, the REMI event vocabulary 305 is searched to obtain an integer array 303 corresponding to the REMI event number.

[0048] Specifically, the REMI event vocabulary described in 305 is composed of: Note-On_1, ......., Note-On_127 represents 127 different pitch events, Note-Duration_1,..., Note-Duration_64 represents 64 different note duration events, Note-Velocity_1,..., Note-Velocity_126 represents 126 different note velocity events, Tempo_17,..., Tempo_197 represents 60 different velocity events consisting of velocities from 17 to 197 divided into intervals of 3, Position_0,..., Position_15 represents 16 position events that divide a musical measure into 16 different positions, and Bar represents the beginning of a musical measure, represented by a measure event.

[0049] 4) Perform word embedding on the integer array 303 of the REMI event number to obtain a word vector with a dimension of d_model, where the specific value of d_model can be set manually and is usually set to 512.

[0050] Further, Figure 1 and Figure 2 The REMI event sequence described in may be represented by 302; Figure 2 The REMI vocabulary, integer array of REMI event numbers and word vector segments described in can be represented by 305, 303, and 304, respectively.

[0051] Step 4: Input the complete music sequence of each sample in the theme music dataset into the Transformer decoder, where the complete music sequence of the sample is the initial music representation sequence obtained after processing the original MIDI format music file, and is converted into an integer array after the corresponding vocabulary. The input of the decoder consists of two parts: the complete music sequence of the sample and the output of the encoder. The decoder input sequence is decoded to obtain a music sequence containing the music theme fragment. Finally, the error between the decoder output sequence and the original sample sequence is calculated, and the parameters of the encoder and decoder are synchronously updated using the backpropagation algorithm.

[0052] Furthermore, the present invention proposes a preferred solution: the decoder consists of 6 identical decoding layers, each decoding layer consists of a multi-head self-attention mechanism, a cross-attention mechanism and a feedforward layer, and uses superposition and regularization operations.

[0053] like Figure 4 As shown in the figure, the model structure of the Transformer-based theme music generation method is described, which mainly consists of two parts: the encoder part is described on the left and the decoder part is described on the right.

[0054] Furthermore, the original input of the encoder is a sequence of music theme fragments, and the final input of the encoder is x e The acquisition process is as follows: the music theme fragment sequence of each music sample in the theme music dataset is used as the original input of the encoder, where the sample music theme fragment sequence is an integer array obtained by converting the initial music representation sequence in REMI format through the corresponding REMI vocabulary, denoted as s e , the final input x of the encoder e =TE(s e )+PS(d_model), where TE() represents the token embedding function, PS() represents the sinusoidal positional encoding function, and s e An integer array representing the initial music representation corresponding to the sequence of music theme fragments, and d_model represents the embedding dimension of the word embedding function.

[0055] The encoder processes data as follows: First, get the encoder's final input x e , then x ePerform self-attention calculation, and then send it to the feedforward layer after superposition and regularization. The output of the feedforward layer is superimposed and regularized again to obtain the final output y of the encoder. e .

[0056] Furthermore, the input of the decoder consists of two parts, namely the input x at the bottom of the decoder d and the encoder output y e .x d The acquisition process is as follows: the complete music sequence of each music sample in the theme music dataset is input into the decoder of Transformer, where the complete music sequence is an integer array obtained by converting the initial music representation sequence in REMI format through the corresponding REMI vocabulary, denoted as s d , the bottom input x of the decoder d =TE(s d )+PS(d_model), where TE() represents the token embedding function, PS() represents the sinusoidal positional encoding function, and s d An integer array representing the initial music representation corresponding to the complete music sequence, d_model represents the embedding dimension of the word embedding function, and the output y of the encoder e As input to the decoder when computing cross attention.

[0057] The decoder processes data as follows: First, get the final input x at the bottom of the decoder d , then x d Perform masked multi-head attention calculations, and then superimpose and regularize the calculation results with the encoder output y e Together they serve as the input of the cross-attention calculation module, and the output of the cross-attention calculation module is output to the feedforward layer after superposition and regularization. The output result of the feedforward layer is then output to the linear layer after another superposition and regularization. Finally, the output result is obtained through the Softmax function, which is a complete music sequence containing a sequence of music theme fragments.

[0058] Step 5, return to repeat steps 3 and 4, and repeatedly train the model until the model reaches the training termination condition; input the user-input music theme melody as the conditional music theme fragment sequence into the model, obtain a complete music sequence containing the music theme fragment sequence, and save it as a specified music format file.

[0059] During each iteration, the model parameters will undergo random changes, and the training stop condition can be set when the model loss drops to a certain value. In specific implementation, those skilled in the art can set the training stop condition.

[0060] The above process focuses on the generation of theme music, that is, the automatically generated music has better regularity and repetitiveness, conforms to the composer's composition habits, and can bring better auditory enjoyment to the audience through intelligent solutions.

[0061] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0062] like Figure 5 As shown, in some possible embodiments, a conditional music theme melody automatic generation system based on Transformer is provided, including the following modules:

[0063] 1) Conversion module: For the format conversion of the original data set, the music file format of the original data set can be WAV, MP3, WMA and other audio formats. The conversion module is used to convert the music file format of the original data set into MIDI (Musical Instrument Digital Interface) format to facilitate unified processing by the model and computer.

[0064] 2) Filtering module: Filter the original MIDI format data set obtained from the conversion module, select the theme music data, discard the non-theme music data, and obtain the theme music data set.

[0065] 3) Extraction module: For each music sample of the theme music data set obtained from the filtering module, the extraction module is used to extract the music theme segment corresponding to each theme music sample.

[0066] 4) Building Module: Use the building module to build a Transformer-based music theme melody automatic generation model, including the construction of the encoder and decoder, as well as the encoding module corresponding to the input sequence.

[0067] 5) Training module: The music theme fragment sequence and the complete music sequence are input into the model constructed by the construction module for training, and the final trained model is obtained through the training module.

[0068] 6) Generation module: The user inputs a music theme melody, which is then sent to the extraction module to extract the music theme fragments and obtain a music theme fragment sequence. The music theme fragment sequence is input into the trained model to generate a complete music sequence containing the music theme fragment sequence.

[0069] 7) Output module: converts the music sequence output of the generation module into a music format to obtain the final playable music file output.

[0070] In some possible embodiments, a Transformer-based automatic generation system for conditional music theme melody is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, the Transformer-based automatic generation method for conditional music theme melody is implemented.

[0071] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A method for automatically generating conditional music theme melody based on Transformer, characterized by: The following steps are included: Step 1: Convert the data in the original dataset into a unified data format and convert it into a music dataset in MIDI format. Then, filter the MIDI format dataset to select data samples with theme music and filter out non-theme music to obtain the theme music dataset. Step 2, extracting music theme fragments from the theme music in the theme music dataset; Step 3: Input the music theme fragment sequence of each sample in the theme music dataset into the Transformer encoder, and encode the music theme fragment sequence to obtain the encoder output; the music theme fragment sequence is the initial music representation sequence obtained by processing the original MIDI format music file, and is converted into an integer array through the corresponding vocabulary; Step 4: Input the complete music sequence of each sample in the theme music dataset into the Transformer decoder, where the complete music sequence of the sample is the initial music representation sequence obtained after processing the original MIDI format music file and the integer array obtained after the corresponding vocabulary conversion; The decoder input consists of two parts: a complete music sample sequence and the encoder output. The decoder input sequence is decoded to obtain a music sequence containing the music theme fragment. Finally, the error between the decoder output sequence and the original sample sequence is calculated, and the back-propagation algorithm is used to synchronously update the encoder and decoder parameters. Step 5, return to repeat steps 3 and 4, and repeatedly train the model until the model reaches the training termination condition; input the user-input music theme melody as the conditional music theme fragment sequence into the model, obtain a complete music sequence containing the music theme fragment sequence, and save it as a specified music format file.

2. The method for automatically generating conditional music theme melody based on Transformer according to claim 1, characterized in that: The implementation method of step 1 is as follows: for the MIDI format music dataset after format conversion, each piece of MIDI format music is taken as a sample, the sample is encoded to obtain an initial music representation in the REMI format, divided according to bars, with two bars as a segment, each sample is divided into several segments, and then the number of repetitions of each segment in the sample is calculated, and the segment with the most repetitions in the sample is taken as the representative segment. If the number of repetitions of the representative segment reaches a preset threshold, it is determined that the sample has a music theme; all samples in the dataset are divided to obtain a theme music dataset.

3. The method for automatically generating conditional music theme melody based on Transformer according to claim 1, characterized in that: Step 2 is implemented by taking each MIDI format theme music as a sample, encoding the sample to obtain an initial music representation in the REMI format, dividing it according to bars, with two bars as a segment, dividing each sample into several segments, embedding the divided samples into words, and obtaining the corresponding word vectors. Then, a clustering algorithm is used for each sample word vector to cluster the same or similar segments in the word vectors to obtain multiple clusters. The cluster with the largest number of segments is selected as the theme cluster, and then the segment closest to the cluster centroid in the theme cluster is taken as the music theme segment.

4. The method for automatically generating conditional music theme melody based on Transformer according to claim 1, characterized in that: Step 3 is implemented as follows: the Transformer encoder consists of 6 identical encoding layers, each of which consists of a multi-head self-attention mechanism and a feedforward layer, and uses superposition and regularization operations.

5. The method for automatically generating conditional music theme melody based on Transformer according to claim 4 is characterized in that: The decoder consists of 6 identical decoding layers, each of which consists of a multi-head self-attention mechanism, a cross-attention mechanism and a feed-forward layer, and uses stacking and regularization operations.

6. A Transformer-based automatic generation system for conditional music theme melody, characterized by: Used to implement a Transformer-based automatic generation method for conditional music theme melody as described in any one of claims 1-5.

7. The Transformer-based conditional music theme melody automatic generation system according to claim 6, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for automatically generating conditional music theme melody based on Transformer as described in any one of claims 1 to 5.

8. The Transformer-based conditional music theme melody automatic generation system according to claim 6, characterized in that: The invention comprises a readable storage medium on which a computer program is stored. When the computer program is executed, the method for automatically generating a conditional music theme melody based on Transformer is implemented.

Images