Music melody generation method and device, equipment and medium

Abstract

Embodiments of the invention disclose a music melody generation method and apparatus, a device and a medium. The method comprises the steps of obtaining an original audio file in an MIDI format; performing event representation on the original audio file through a self-defined music representation form, and obtaining a first melody sequence through preprocessing; wherein the music representation forms comprise the number of bars, the position, the speed, the pitch, the note rate, the duration, the musical instrument type and the chord; inputting the first melody sequence into a trained music melody generation model for prediction to obtain a second melody sequence; wherein the music melody generation model is constructed based on an Informer network; the Informer network comprises an encoder and a decoder; each layer of the encoder uses a multi-head ProbSparse self-attention mechanism; and converting the second melody sequence into an MIDI format file to obtain the music work. According to the method, music information of a higher level can be captured to better represent music, and meanwhile, richness and diversity of generated music are improved by utilizing a model constructed by an Informer network.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and includes, but is not limited to, methods, devices, and media for generating music melodies. Background Technology

[0002] Music melody generation, as an emerging research field, has broad application prospects and value. Besides traditional tasks such as prediction, classification, and translation, deep learning is receiving increasing attention as a method for music generation. Currently, the mainstream generative models in music generation tasks are mainly based on RNN, LSTM, GAN, and Transformer architectures.

[0003] RNNs can capture long-term dependencies in musical sequences, generate rhythmic music, and handle multiple tracks; however, they suffer from vanishing and exploding gradients, lack global attention, and have complex training data requirements and models. LSTMs can also capture long-term dependencies, handle variable-length sequences, and learn musical structure and patterns; however, they lack global attention, and the generated musical segments may lack consistency and coherence. GANs have a relatively complex training process, including a game between the generator and discriminator. This can lead to instability during training, making them prone to pattern collapse and other problems, affecting the quality and diversity of the generated music.

[0004] The Transformer model has a more complex structure, requiring more computational resources and training time. Lacking explicit sequentiality, it considers all inputs simultaneously, potentially leading to a lack of clear temporal order in the generated music. This can result in relatively weak overall structure and rhythm in the generated musical segments. Although a self-attention mechanism is introduced to capture long-distance relationships within the sequence, issues of delayed information delivery or information loss may still exist for music generation tasks, especially when dealing with longer time series. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, device, and medium for generating music melody.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a method for generating a musical melody, the method comprising:

[0008] A first melody sequence is obtained by acquiring a raw audio file in MIDI format; the raw audio file is represented by an event representation using a custom music representation format and preprocessed to obtain a first melody sequence; wherein the music representation format includes measure number, position, tempo, pitch, note rate, duration, instrument type, and chords; the first melody sequence is input into a trained music melody generation model for prediction to obtain a second melody sequence; wherein the music melody generation model is built based on an Informer network; the Informer network includes an encoder and a decoder; each layer of the encoder uses a multi-head ProbSparse self-attention mechanism; the second melody sequence is converted into a MIDI format file to obtain a music melody.

[0009] Secondly, embodiments of this application provide a music melody generation apparatus, comprising: a data acquisition module, an event representation module, a melody prediction module, and a format conversion module, wherein:

[0010] The data acquisition module is used to acquire raw audio files in MIDI format;

[0011] The event representation module is used to represent events in the original audio file using a custom musical representation format and obtain a first melody sequence through preprocessing; wherein, the musical representation format includes the number of measures, position, tempo, pitch, note rate, duration, instrument type, and chords;

[0012] The melody prediction module is used to input the first melody sequence into a trained music melody generation model for prediction to obtain a second melody sequence; wherein, the music melody generation model is built based on an Informer network; the Informer network includes an encoder and a decoder; each layer of the encoder uses a multi-head ProbSparse self-attention mechanism;

[0013] The format conversion module is used to convert the second melody sequence into a MIDI format file to obtain a musical work.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps in the above-described music melody generation method.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described music melody generation method.

[0016] The beneficial effects of the technical solutions provided in this application include at least the following:

[0017] In this embodiment, firstly, an original audio file in MIDI format is acquired; then, the original audio file is represented by an event representation using a custom music representation method and preprocessed to obtain a first melody sequence; next, the first melody sequence is input into a trained music melody generation model for prediction to obtain a second melody sequence; finally, the second melody sequence is converted into a MIDI format file to obtain a musical work. Thus, based on the MIDI-Like format, an innovative music event representation method is proposed by adding supporting music tags to the original audio file, which can capture higher-level music information to better represent music. Simultaneously, an Informer-based music melody generation model is constructed to flexibly handle multimodal features such as notes, rhythms, and chords. By embedding different types of features into the model input, the model can comprehensively utilize multiple information when generating music. This improves the richness and diversity of the generated music. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0019] Figure 1 A flowchart illustrating the music melody generation method provided in this application embodiment;

[0020] Figure 2 This is a schematic diagram of a MIDI-like audio file provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a custom format file provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram of the data preprocessing process provided in the embodiments of this application;

[0023] Figure 5 This is a block diagram of the overall architecture of the Informer model provided in the embodiments of this application;

[0024] Figure 6 This is a schematic diagram of the encoder structure provided in an embodiment of this application;

[0025] Figure 7 A schematic block diagram of a melody generation system provided in an embodiment of this application;

[0026] Figure 8 This application provides a diagram illustrating the research and system implementation architecture for music melody generation in an embodiment.

[0027] Figure 9 This application provides a technical roadmap for the research and system implementation of music melody generation in its embodiments.

[0028] Figure 10 This is a schematic diagram of the composition structure of a music melody generation device provided in an embodiment of this application;

[0029] Figure 11 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] It should be noted that the terms "first, second, and third" used in the embodiments of this application are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0034] First, this application summarizes and analyzes the current status of music melody generation from two aspects: the research status of traditional music melody generation methods and the research status of music melody generation using deep learning models.

[0035] The current state of research on traditional music melody generation methods is as follows:

[0036] Traditional music melody generation methods refer to those that do not employ neural networks, but rather rely more on data-driven statistical methods, such as N-gram models and Markov models. A Markov chain is a model used to describe a sequence of possible events. This sequence should ensure that the probability of the next state occurring depends only on the previous state. Generally speaking, Markov chains are a good way to enforce rules: for example, it can be assumed that after a chord, the probability of some other chords appearing is zero, while some other chords have a certain probability of appearing. However, the state of the entire measure should be considered, not just a single chord.

[0037] Evolutionary algorithms are based on Darwin's theory of evolution and are self-organizing, adaptive artificial intelligence techniques that solve problems by simulating the processes and mechanisms of biological evolution. In music generation, it typically starts with a musical fragment (composed of notes, chords, or measures), applies a mutation function to it, and generates a new melody based on the initial melody without breaking the rules. The generated melody is then continuously optimized using a user-defined adaptive function.

[0038] There are also rule-based music generation methods, which primarily utilize specific rules or grammar for composition. Generally, a grammatical system includes start symbols, end symbols, non-end symbols, and a series of rules governing the generation process. This grammar-based generation approach is very popular in music generation research because music also has its own grammar, such as the compositional rules of harmony, counterpoint, form, and orchestration. Those who study this generation method are often multidisciplinary professionals with strong musical backgrounds and expertise in both music and computer science.

[0039] The current state of research on music generation using deep learning models is as follows:

[0040] Besides traditional tasks such as prediction, classification, and translation, deep learning is receiving increasing attention as a method for music generation. Currently, the mainstream generative models in music generation tasks are primarily based on RNN, LSTM, GAN, Transformer, and Informer architectures.

[0041] Music generation based on RNNs. Unlike basic neural networks, recurrent neural networks can process sequential data and extract temporal and semantic information from the data. They are well-suited for text and audio classification problems and have already achieved significant results in speech recognition, music generation, sentiment classification, DNA sequence analysis, and machine translation. Many scholars are also exploring the potential of RNNs in the field of music.

[0042] Music generation based on LSTM. While traditional RNNs have a simple internal structure, low computational resource requirements, and perform well on short sequence tasks, they are prone to gradient vanishing and exploding phenomena when handling long sequence tasks. Therefore, the improved LSTM (Long Short-Term Memory) network was proposed. LSTM mainly consists of four parts: a forget gate, an input gate, a cell state, and an output gate. The output gate determines whether information is still usable. Information that is still usable after algorithmic judgment is retained; otherwise, it is discarded through the forget gate.

[0043] Music generation based on GANs. Generative Adversarial Networks consist of a generator and a discriminator. The generator generates new data based on existing real data, while the discriminator is responsible for determining whether the data is real or generated. Both need to continuously optimize their generation and discrimination capabilities through repeated adversarial training to optimize the model.

[0044] Music generation is based on the Transformer architecture. The Transformer abandons traditional CNN and RNN structures, relying solely on self-attention to compute the implicit representations of input and output. It employs an encoder-decoder architecture, where the encoding and decoding modules are each composed of an equal number of stacked encoders and decoders. The encoder unit consists of a multi-head attention mechanism and a feedforward neural network. Similar to the encoder unit, the decoder unit also includes a multi-head attention mechanism and a feedforward neural network, but with an additional multi-head attention layer to focus on relevant parts of the input sequence. Residual connections are added to both the encoder and decoder units to prevent degradation issues during deep neural network training. Layer normalization is then applied to accelerate convergence.

[0045] Hidden Markov Models (HMMs) may oversimplify data distribution assumptions, failing to generate music with greater innovation and diversity, and are limited in handling complex musical structures and long-term dependencies. Evolutionary algorithms offer advantages in music generation such as creativity, diversity, and adaptability, but also suffer from subjectivity, computational complexity, and a lack of prior knowledge. Rule-based methods typically require manually designed and defined rules, potentially failing to capture the complexity and innovation of music, and are limited in handling long-term dependencies.

[0046] RNNs can capture long-term dependencies in musical sequences, generate rhythmic music, and handle multiple tracks; however, they suffer from vanishing and exploding gradients, lack global attention, and have complex training data requirements and models. LSTMs can also capture long-term dependencies, handle variable-length sequences, and learn musical structure and patterns; however, they lack global attention, and the generated musical segments may lack consistency and coherence. GANs have a relatively complex training process, including a game between the generator and discriminator. This can lead to instability during training, making them prone to pattern collapse and other problems, affecting the quality and diversity of the generated music.

[0047] The Transformer model has a more complex structure, requiring more computational resources and training time. Lacking explicit sequentiality, it considers all inputs simultaneously, potentially leading to a lack of clear temporal order in the generated music. This can result in relatively weak overall structure and rhythm in the generated musical segments. Although a self-attention mechanism is introduced to capture long-distance relationships within the sequence, issues of delayed information delivery or information loss may still exist for music generation tasks, especially when dealing with longer time series.

[0048] Research on Sequence Prediction Based on Informer Models. While Transformer models are widely used for music melody generation, they still face challenges such as quadratic spatiotemporal complexity, high memory consumption, and inherent limitations of the encoder-decoder architecture. To address these issues, researchers have proposed an efficient algorithm model called Informer for long-sequence time prediction. This model, based on the Transformer model, employs a ProbSparse self-attention mechanism, achieving O(L / logL) time complexity and enhancing sequence dependency alignment for more reliable performance. Informer models are currently widely used in time series forecasting, weather forecasting, energy load forecasting, and traffic flow forecasting, demonstrating excellent performance in handling long-sequence data and capturing global dependencies.

[0049] Music can be viewed as a temporal sequence, and the Informer model, as a sequence prediction model based on a self-attention mechanism, can be used to assist in music melody generation and composition. It can provide musicians with creative inspiration and novel melodic ideas, promoting innovation and experimentation in music composition. Therefore, this application studies and analyzes music melody generation models based on Informer networks, exploring the richness and diversity of generated music, as well as generation efficiency.

[0050] This application provides a method for generating music melody, applied to an electronic device. The electronic device includes, but is not limited to, mobile phones, laptops, tablets, handheld internet devices, multimedia devices, streaming media devices, mobile internet devices, wearable devices, or other types of electronic devices. The function implemented by this method can be achieved by a processor in the electronic device calling program code. The program code can be stored in a computer storage medium; therefore, the electronic device includes at least a processor and a storage medium. The processor can be used to process the music melody generation process, and the memory can be used to store the data needed and generated during the music melody generation process.

[0051] Figure 1 This is a flowchart illustrating a method for generating music melody provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes at least the following steps:

[0052] Step S110: Obtain the original audio file in MIDI format.

[0053] Here, Musical Instrument Digital Interface (MIDI) is a communication standard used to allow computer music programs, synthesizers, and other electronic audio devices to exchange information and control signals. A MIDI file is a universal music file format used to record information about musical performance and composition. It does not contain the audio data itself, but rather information such as notes, tracks, instrument types, volume, and pitch. MIDI-like audio files include... Figure 2 As shown.

[0054] A MIDI file has a relatively simple structure, consisting of a series of MIDI events. Each event contains a timestamp and a command. The timestamp indicates when the event occurred during the performance, while the command contains relevant musical information. A MIDI file can contain multiple tracks simultaneously, and each track can contain multiple MIDI events.

[0055] In some implementations, melody datasets of different styles and time periods are downloaded from publicly available music databases such as MIDI databases. The types of datasets include, but are not limited to, piano or guitar types. In some implementations, relevant audio is downloaded from the network, edited using a DAW, and processed into MIDI format. This application does not limit this approach.

[0056] Step S120: The original audio file is represented by a custom music representation format and the first melody sequence is obtained through preprocessing.

[0057] Here, the musical representation forms include measure number, position, tempo, pitch, note rate, duration, instrument type, and chord. In implementation, an event can be established for each musical marker corresponding to the above representation forms, such as a note tempo setting event and a time signature setting event.

[0058] MIDI-Like notation is most widely used in deep learning-based symbolic music generation tasks. It uses Note on and Note off events to represent the start and end of a note, respectively, and integers between 0 and 127 to represent the pitch of the note. However, it only records the process of musical performance and cannot provide an intuitive representation of the musical content. This application's embodiments, based on the MIDI-Like format, add the aforementioned supporting musical markers to capture higher-level musical information for a better representation of the music.

[0059] The preprocessing process may include, but is not limited to, feature extraction, data serialization, standardization, data augmentation, and storage, and then the data is divided into training set, test set, and validation set according to a certain ratio.

[0060] Step S130: Input the first melody sequence into the trained music melody generation model for prediction to obtain the second melody sequence.

[0061] Here, the music melody generation model is built upon an Informer network; the Informer network includes an encoder and a decoder; each layer of the encoder uses a multi-head ProbSparse self-attention mechanism. The Informer model is designed to handle long sequences, thus offering advantages in generating longer melodies. This is beneficial for music generation tasks, as music melodies typically involve long time spans.

[0062] The Informer architecture is used as the main body of the model, with both the encoder and decoder having 6 layers, such as... Figure 5 As shown. The left side is the encoder, which receives a large number of long sequence inputs (vectors X obtained from music sequences). en The first melody sequence mentioned above is used, and a multi-head ProbSparse self-attention mechanism is employed. The trapezoidal representation between adjacent ProbSparse self-attention mechanisms represents the self-attention distillation operation, significantly reducing the network size. Stacking multiple layers increases the model's robustness. The decoder on the right receives a long sequence input (vector) from the encoder and a vector X composed of a cue sequence (to improve prediction accuracy) and a prediction sequence (padded with zeros). de ={X token,X0}, and generate the Outputs vector, i.e., the predicted second melody sequence, all at once.

[0063] The model encoder uses an attention distillation mechanism, which halves the features of a single layer in the time dimension, thus allowing for longer input sequences and capturing long-range dependencies between them. See the encoder's structure diagram below. Figure 6 The given structure is the main stack that receives the entire input sequence. The second stack takes half of the input, and this process is repeated for subsequent stacks. The red layers are dot product matrices, decreasing progressively by applying a self-attention distillation mechanism at each layer. The feature maps from all stacks are concatenated as the encoder output. A standard decoder structure is used, which involves stacking two identical multi-head attention layers.

[0064] Step S140: Convert the second melody sequence into a MIDI format file to obtain a musical work.

[0065] Here, the decode_midi function in Miditok is used to convert the second melody sequence output by the music melody generation model into a MIDI format file and store it as a musical work.

[0066] It should be noted that because a MIDI file is a series of instructions rather than a waveform, it requires very little disk space. Editing and modifying MIDI data is very flexible, allowing you to easily add or delete notes or change their attributes.

[0067] This application, based on the MIDI-Like format, proposes an innovative method for representing music events by adding supporting music tags to the original audio files. This method can capture higher-level music information for better music representation. Simultaneously, it constructs an Informer-based music melody generation model that flexibly handles multimodal features such as notes, rhythms, and chords. By embedding different types of features into the model input, the model can comprehensively utilize various information when generating music. This improves the richness and diversity of the generated music.

[0068] In some implementations, the above step S120, "representing the original audio file through a custom music representation and obtaining a first melody sequence through preprocessing," is implemented through the following process: using the open-source MidiTok library, the original audio file is represented by adding supporting music tags, and combined into a feature representation using byte-pair encoding; the feature representation is vectorized, segmented, padded, and standardized to obtain the first melody sequence.

[0069] Here, the custom format file is as follows: Figure 3As shown, the melody consists of events of type bar, pitch, velocity, and duration. The data preprocessing flow is as follows: Figure 4 As shown, the extracted note features include note duration, pitch, and volume; rhythm features include rhythmic patterns and beat structure; and emotional features can be modeled through rhythm, timbre, and chords. Byte Pair Encoding (BPE) is used to represent the melody as a combination of these basic features, thereby symbolizing the original audio file. Then, it is vectorized through time steps, context, and sequence vectors, and then segmented or padded. Finally, the normalization set normalizes and equalizes the note sequence.

[0070] Informer networks can flexibly handle multimodal features, such as notes, rhythms, and chords. By embedding different types of features into the first melody sequence as input to the music generation model, the model can comprehensively utilize multiple types of information when generating music. This can improve the richness and diversity of generated music.

[0071] In some implementations, after standardization, the method further includes: randomly transforming the first melody sequence and adding noise to obtain a new first melody sequence.

[0072] Here, the random transformation includes at least one of the following: pitch transformation, volume transformation, and speed transformation. Adding noise can increase the model's adaptability to noisy environments.

[0073] In some implementations, the music melody generation model is trained through the following process:

[0074] Step S131: Obtain the public dataset and the custom dataset to get the sample dataset;

[0075] Here, the custom dataset is a dataset of melodies of different styles and time periods downloaded from the network; the public dataset, such as the Maestro dataset or the Piano-E dataset, divides the sample dataset into a training set, a test set, and a validation set in a ratio of 7:2:1.

[0076] Step S132: For each sample in the sample dataset, perform event representation and obtain the corresponding third melody sequence through preprocessing.

[0077] Here, the method for preprocessing the event representation set of the samples in the sample dataset is similar to the method for processing the original audio file. For the specific implementation process, please refer to the description of step S120 above.

[0078] Step S133: Input the third melody sequence into the Informer network to obtain the prediction sequence.

[0079] Here, the encoder first learns features from the third training sequence to obtain the encoded representation, and then the decoder decodes the encoded representation to obtain the predicted sequence.

[0080] Step S134: Calculate the error between the predicted sequence and the third melody sequence, and optimize the parameters of the encoder and decoder until convergence, to obtain the music melody generation model.

[0081] Here, the error is obtained by comparing the predicted sequence with the third melody sequence, and the loss function, such as the cross entropy loss function, is determined. Then, the loss function is used to update the parameters in reverse through the gradient update method until the iteration condition is reached, thus obtaining the music melody generation model.

[0082] The following are examples of relevant settings during training, but this application is not limited to them and can be adjusted adaptively according to actual conditions: batch size is set to 4, learning-rate is set to 1e-4, epoch is set to 200, max_sequence length is set to 2048, and the Adam optimizer is used for optimization. The model is trained / tested on a single Nvidia A100 GPU.

[0083] In some implementations, the method further includes: in the ProbSpare self-attention mechanism, setting different random sampling factor numbers for input elements in the input first melody sequence to randomly update the attention weight allocation between elements; adding convolutional layers and max pooling after each attention block of the encoder, and adjusting the parameter weights of the convolutional layers and the max pooling layers respectively, so that the features of a single layer are halved in the time dimension.

[0084] Here, different input elements represent different time steps or features within the sequence. Each attention head can focus on different time steps or features within the sequence. By adjusting the number of random sampling factors (specifically, the `factor` parameter set in the code), different attention weights are assigned to different input elements, thereby capturing details and forms at different levels. For example, in the original Informer model, the value of the `factor` parameter is 5, while in this model, for long musical melodic sequences, the value is set to 15. This allows the Informer model to model musical sequences at different levels. Furthermore, the model can simultaneously process global and local information of the musical sequence, improving the diversity and creativity of generated music.

[0085] At the same time, the convolution module, including the downConv layer and the maxpool layer, was adjusted. The parameters of the downConv layer were set to kernel_size=3 and padding=1, and the parameters of the maxpool layer were set to kernel_size=3, stride=2 and padding=1, thereby optimizing the distillation mechanism and improving the performance of the model.

[0086] The Probable Sparse Self-attention mechanism introduces probabilistic sparsity, selectively focusing on a subset of elements in the input sequence to improve computational efficiency and model performance. The process is as follows:

[0087] 1. For each input element, calculate its similarity score with other elements (usually calculated by dot product or similar methods).

[0088] 2. Determine which elements are selected as participants based on a pre-set probability threshold or probability distribution.

[0089] 3. For the selected participant, calculate the attention weight between it and other elements, and normalize it using the Softmax function.

[0090] 4. Multiply the attention weights by their corresponding value vectors and perform a weighted sum to obtain the final self-attention output. The calculation formula is as follows:

[0091]

[0092] Where A(q) i, (K,V) represents a given query vector q i, Attention weights for K and V It is a sparse matrix of the same size as q, containing only the top-u queries under the sparsity measure M(q,K), where K represents the key vector and V represents the value vector. It is the scaling factor. Under the control of a constant sampling factor c, set u = c.lnL Q This allows ProbSparse self-attention to only require finding and computing O(lnL) for each query key. Q The dot product is calculated, and the layer memory usage is O(L). K lnL Q With multiple heads working together, this attention generates different sparse query-key pairs for each head, thus avoiding severe information loss.

[0093] In this way, the melody generation model uses the ProbSpare self-attention mechanism and the Self-attention distillation mechanism, effectively reducing the complexity of the algorithm, saving computing resources, and improving generation efficiency. The ProbSpare self-attention mechanism of the Informer model is optimized by changing the attention weight allocation method and using different probability distribution functions to improve model performance. For the distillation mechanism, the weights of the convolutional layer are adjusted to optimize it. The model also mines the contextual information of the musical melody, learns musical features, and weakens the interference of irrelevant information.

[0094] In some embodiments, the method further includes: based on the trained music melody generation model, selecting at least one of the following ablation factors for model parameter tuning: attention mechanism, number of self-attention heads, number of encoder layers, number of decoder layers, length of input music sequence, and dataset type; determining the entropy distribution of pitch and / or rhythmic pattern similarity in the second melody sequence, and using these as objective evaluation parameters to evaluate the generation effect of the parameter-tuned music melody generation model.

[0095] Here, the basic Informer model is selected as the control condition; ablation factors (attention mechanism, number of self-attention heads, number of encoder layers, number of decoder layers, length of input music sequence, and type of dataset (piano dataset or guitar dataset)) are selected for ablation experiments.

[0096] For objective evaluation of the model's generated melodies, we selected Pitch Class Histogram Entropy (low entropy if the pitch of a segment is clear, and high entropy if the pitch of a segment is scattered) and Groove Consistency (rhythm pattern similarity, which helps measure the rhythm of music; if a segment has a clear rhythm, then the rhythm patterns of different measures should be similar, thus producing high entropy). On the other hand, if the rhythm feels unstable, the rhythmic patterns in different measures will also be unstable, leading to... The lower score is used as an objective evaluation parameter, and the calculation formula is as follows:

[0097]

[0098]

[0099] In equation (2), h represents the entropy distribution of pitch. i The probability of each pitch category appearing in music; there are 12 pitch categories in total.

[0100] In equation (3), Indicates rhythmic pattern similarity. Indicates the position of a note in a measure; a value of 1 indicates the position of a note, and a value of 0 indicates the position of a note. Q represents... The dimension is set to 64, and XOR represents the exclusive OR operation. The value is between 0 and 1.

[0101] In some implementations, subjective evaluations of the melody can be added, inviting music lovers to give corresponding scores based on their subjective feelings after listening to it, so as to subjectively evaluate the effect of the generated melody and ensure the vividness and listenability of the generated melody.

[0102] After optimizing the Informer model melody generation model, the proposed models need to be integrated to build a complete melody generation system. This application also provides a music melody generation system, including an operating system, a server, a backend, and a frontend. Specifically: the server implements and encapsulates the aforementioned music melody generation model and designs access interfaces for invocation; the frontend is developed using at least one programming language, including a player and a visual user interface; and the backend is developed using the Spring framework.

[0103] The architecture diagram of the melody generation system designed in this application embodiment is as follows: Figure 7 As shown, the system consists of four parts: operating system, application server, backend, and frontend. The first two layers are the underlying services that maintain system operation. The latter two layers are the focus of this application's system design, revolving around three parts: First, the proposed model is implemented and encapsulated, and access interfaces are designed for invocation. MySQL is used as the database, with a suitable table structure designed to store system data. Finally, a B / S architecture system is built to provide users with a visual interactive interface. The backend is developed using the Spring framework, and the frontend is developed using languages ​​such as HTML and CSS. The client uses a browser to access the web pages.

[0104] The above-described method for generating music melody will now be described with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0105] This application aims to propose an innovative method for representing music events, then construct a music melody generation model based on the Informer, realize melody generation, and verify the quality and originality of the melodies generated by the Informer model. Finally, a music melody generation system is built, providing functions such as melody generation, music recommendation, and music emotion classification.

[0106] Figure 8The music melody generation research and system implementation architecture diagram provided in the embodiments of this application is as follows: Figure 8 As shown, on the one hand, the MIDI-Like event representation is optimized by adding musical knowledge such as measure number, position, tempo, pitch, note rate, and duration to capture higher-level musical information and better represent the music. On the other hand, an Informer-based music melody generation model is constructed, including an encoder, decoder, and self-attention mechanism. Furthermore, a music melody generation system is built, providing functions such as melody generation, music recommendation, and music emotion classification.

[0107] MIDI-Like representation is most widely used in deep learning-based symbolic music generation tasks. It uses "Note on" and "Note off" events to represent the start and end of a note, respectively, and integers between 0 and 127 to represent the pitch. However, it only records the process of musical performance and cannot provide an intuitive representation of the musical content. This research aims to add musical knowledge to the existing MIDI-Like representation, representing MIDI data in a way that resembles how humans read music, in order to better represent music and provide richer input information for music melody generation models.

[0108] Figure 9 The technical roadmap for music melody generation research and system implementation provided in the embodiments of this application is as follows: Figure 9 As shown, 1. Data acquisition and preparation, including Midi data and non-Midi data, where non-Midi data can be edited and processed into Midi format using DAW; 2. Model construction: first, a music melody generation model is constructed based on the Informer network structure, and then the model validity is assessed; 3. Melody generation system construction, including database design, integration interface design, backend logic design, and user interaction design.

[0109] This application constructs an Informer-based music melody generation model, including an encoder, decoder, and self-attention mechanism. It also optimizes the ProbSpare self-attention and Self-attention distillation mechanisms of the Informer model, modifying the attention weight allocation method in the ProbSpare self-attention mechanism and using different numbers of random sampling factors to improve model performance.

[0110] For the ProbSpare self-attention mechanism, the number of random sampling factors in the original Informer's ProbSpare self-attention is modified. This model sets it to 15 for long sequences of musical melodies to better capture musical information.

[0111] The distillation mechanism, including convolutional layers (downConv layers) and maxpooling layers, is implemented in this application by adjusting the parameter weights of the convolutional and maxpooling layers. The distillation mechanism modifies the corresponding parameter weights of each layer; for example, the downConv layer parameters are set to kernel size = 3 and padding = 1, and the maxpool layer parameters are set to kernel size = 3, stride = 2, and padding = 1. This optimization of the distillation mechanism improves the model's performance. Furthermore, it mines contextual information from musical melodies, learns musical features, and weakens the interference of irrelevant information.

[0112] The traditional Transformer model generates vectors one digit at a time until the desired vector length is reached (i.e., the vector length gradually increases from 1 to the required length). In contrast, the Informer model directly specifies the length of the generated vector and generates the target vector all at once, without the gradual increase in length. In other words, unlike the Transformer's step-by-step dynamic decoding (i.e., the input at the next time step depends on the output at the previous time step), the melody generation model provided in this application uses a parallel decoder to output all prediction results at once, improving the inference speed of music sequence prediction.

[0113] The embodiments of this application have at least the following beneficial effects: (1) The melody generation model is designed to handle long sequences, thus having an advantage in generating longer melodies. This is beneficial for music generation tasks, as musical melodies typically involve long time spans. (2) The melody generation model can model musical sequences at different levels. Each attention head can focus on different time steps or features in the sequence, thereby capturing details and forms at different levels. This allows the model to process both global and local information of the musical sequence simultaneously, improving the diversity and creativity of generated music. (3) The melody generation model can flexibly handle multimodal features, such as notes, rhythms, chords, etc. By embedding different types of features into the input, the model can comprehensively utilize multiple types of information when generating music. This can improve the richness and diversity of generated music. (4) The melody generation model uses the ProbSpare self-attention mechanism and the Self-attention distillation mechanism, effectively reducing the complexity of the algorithm, saving computing resources, and improving generation efficiency.

[0114] Based on the foregoing embodiments, this application further provides a music melody generation device. The device includes various modules and units included in each module, which can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0115] Figure 10 This is a schematic diagram of the composition structure of a music melody generation device provided in an embodiment of this application, as shown below. Figure 10 As shown, the device 1000 includes a data acquisition module 1110, an event representation module 1120, a melody prediction module 1130, and a format conversion module 1140, wherein:

[0116] The data acquisition module 1110 is used to acquire raw audio files in MIDI format;

[0117] The event representation module 1120 is used to represent events in the original audio file through a custom musical representation format and obtain a first melody sequence through preprocessing; wherein, the musical representation format includes the number of measures, position, tempo, pitch, note rate, duration, instrument type, and chords;

[0118] The melody prediction module 1130 is used to input the first melody sequence into a trained music melody generation model for prediction to obtain a second melody sequence; wherein, the music melody generation model is built based on an Informer network; the Informer network includes an encoder and a decoder; each layer of the encoder uses a multi-head ProbSparse self-attention mechanism;

[0119] The format conversion module 1140 is used to convert the second melody sequence into a MIDI format file to obtain a musical work.

[0120] In some possible embodiments, the event representation module 1120 includes: a first processing unit, used to represent the original audio file by adding supporting music tags using the open-source MidiTok library, and combine them into feature representations by byte pair encoding; and a second processing unit, used to vectorize, segment, fill and standardize the feature representations to obtain the first melody sequence.

[0121] In some possible embodiments, the event characterization module 1120 further includes a third processing unit for randomly transforming the first melody sequence and adding noise to obtain a new first melody sequence; wherein the random transformation includes at least one of the following: pitch transformation, volume transformation, and tempo transformation.

[0122] In some possible embodiments, the apparatus further includes a model training module for acquiring a public dataset and a custom dataset to obtain a sample dataset; wherein the custom dataset is a melody dataset of different styles and time periods downloaded via the network; for each sample in the sample dataset, event representation is performed and preprocessed to obtain a corresponding third melody sequence; the third melody sequence is input into the Informer network to obtain a prediction sequence; the error is calculated using the prediction sequence and the third melody sequence, and the encoder and decoder are optimized until convergence is obtained to obtain the music melody generation model.

[0123] In some possible embodiments, the apparatus further includes a weight allocation module for setting different probability distribution functions for input elements in the input first melody sequence in the ProbSpare self-attention mechanism to randomly update the attention weight allocation between elements; adding convolutional layers and max pooling after each attention block of the encoder, and adjusting the parameter weights of the convolutional layers and the max pooling layers respectively, so that the features of a single layer are halved in the time dimension.

[0124] In some possible embodiments, the apparatus further includes an optimization evaluation module, used to select at least one of the following ablation factors for model parameter tuning based on the trained music melody generation model: attention mechanism, number of self-attention heads, number of encoder layers, number of decoder layers, length of input music sequence, and dataset type; determine the entropy distribution of pitch and / or rhythmic pattern similarity in the second melody sequence, and use these as objective evaluation parameters to evaluate the generation effect of the parameter-tuned music melody generation model.

[0125] It should be noted that the descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0126] It should be noted that, in the embodiments of this application, if the above-described music melody generation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0127] Correspondingly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in any of the music melody generation methods described in the above embodiments. Correspondingly, embodiments of this application also provide a computer program product, which, when executed by a processor of an electronic device, is used to implement the steps in any of the music melody generation methods described in the above embodiments.

[0128] Based on the same technical concept, this application provides an electronic device for implementing the music melody generation method described in the above method embodiments. Figure 11 This is a hardware entity diagram of an electronic device provided in an embodiment of this application, such as... Figure 11 As shown, the electronic device 1100 includes a memory 1110 and a processor 1120. The memory 1110 stores a computer program that can run on the processor 1120. When the processor 1120 executes the program, it implements the steps in any of the music melody generation methods described in the embodiments of this application.

[0129] The memory 1110 is configured to store instructions and applications executable by the processor 1120, and can also cache data to be processed or already processed by the processor 1120 and various modules in the electronic device (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0130] When processor 1120 executes a program, it implements the steps of any of the above-mentioned music melody generation methods. Processor 1120 typically controls the overall operation of electronic device 1100.

[0131] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0132] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0133] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0134] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0137] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0138] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0139] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause the device automatic test line to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0140] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0141] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0142] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating musical melody, characterized in that, The method includes: Obtain the raw audio file in MIDI format; The original audio file is represented by a custom musical representation format, and a first melody sequence is obtained through preprocessing. The musical representation format includes the number of measures, position, tempo, pitch, note rate, duration, instrument type, and chords. The first melody sequence is input into a trained music melody generation model for prediction to obtain a second melody sequence; wherein, the music melody generation model is built based on an Informer network; the Informer network includes an encoder and a decoder; the encoder uses a multi-head ProbSparse self-attention mechanism; The second melody sequence is converted into a MIDI file to obtain the musical work.

2. The method according to claim 1, characterized in that, The process of representing events in the original audio file using a custom musical representation format and obtaining a first melody sequence through preprocessing includes: Using the open-source MidiTok library, the original audio file is characterized by adding supporting music tags, and then combined into feature representations using byte-pair encoding. The feature representation is vectorized, segmented, filled, and standardized to obtain the first melody sequence.

3. The method according to claim 2, characterized in that, Following standardized operations, the method further includes: The first melody sequence is randomly transformed and noise is added to obtain a new first melody sequence; wherein the random transformation includes at least one of the following: pitch transformation, volume transformation and tempo transformation.

4. The method according to any one of claims 1 to 3, characterized in that, The music melody generation model was trained through the following process: A sample dataset is obtained by acquiring public datasets and custom datasets; wherein, the custom dataset is a dataset of melodies of different styles and time periods downloaded from the network; For each sample in the sample dataset, an event representation is performed and the corresponding third melody sequence is obtained through preprocessing. The third melody sequence is input into the Informer network to obtain the predicted sequence; The error is calculated using the predicted sequence and the third melody sequence, and the parameters of the encoder and decoder are optimized until convergence is achieved to obtain the music melody generation model.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: In the ProbSpare self-attention mechanism, different random sampling factor numbers are set for the input elements in the first melody sequence to randomly update the attention weight distribution between elements; After each attention block of the encoder, a convolutional layer and a max pooling layer are added, and the parameter weights of the convolutional layer and the max pooling layer are adjusted so that the features of a single layer are halved in the time dimension.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the trained music melody generation model, at least one of the following ablation factors is selected for model parameter tuning: attention mechanism, number of self-attention heads, number of encoder layers, number of decoder layers, length of input music sequence, and type of dataset; Determine the entropy distribution of pitch and / or rhythmic pattern similarity in the second melody sequence, and use these as objective evaluation parameters to evaluate the generation effect of the music melody generation model after parameter tuning.

7. A music melody generation system, characterized in that, This includes the operating system, server, backend, and frontend, among which: The server implements and encapsulates the music melody generation model according to any one of claims 1 to 6, and designs an access interface for invocation; the front end is developed using at least one programming language, including a player and a visual user interface; the back end is developed using the Spring framework.

8. A musical melody generation device, characterized in that, The device includes a data acquisition module, an event representation module, a melody prediction module, and a format conversion module, wherein: The data acquisition module is used to acquire raw audio files in MIDI format; The event representation module is used to represent events in the original audio file using a custom musical representation format and obtain a first melody sequence through preprocessing; wherein, the musical representation format includes the number of measures, position, tempo, pitch, note rate, duration, instrument type, and chords; The melody prediction module is used to input the first melody sequence into a trained music melody generation model for prediction to obtain a second melody sequence; wherein, the music melody generation model is built based on an Informer network; the Informer network includes an encoder and a decoder; the encoder uses a multi-head ProbSparse self-attention mechanism; The format conversion module is used to convert the second melody sequence into a MIDI format file to obtain a musical work.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

Images