Symbolic music generation method and system using language model

Abstract

A symbolic music generation system using a language model according to the present invention: encodes music data to generate an input token from input digital music protocol data; performs one or more structural embeddings on the input token; and generates the next output token by combining an input token embedding result and the structural embedding result and inputting the combined result to a language model, wherein the one or more structural embeddings may be initialized before training the language model.

Description

Method and system for generating symbolic music using a language model

[0001] The present invention relates to a method and system for generating symbolic music, and more particularly, to a method and system for generating symbolic music using a large-scale language model using structural embedding.

[0002] Artificial intelligence (AI) technology has recently been showing cutting-edge developments, drawing attention across society. AI encompasses "a computer brain that executes tasks previously reserved for human intelligence," "the engineering and science of creating intelligent machines," and "a set of algorithmic systems designed to think, perceive, and act like humans," enabling computers to perform highly advanced, human-like intellectual abilities.

[0003] AI, combined with augmented reality, the Internet of Things, edge computing, and digital twins, is being touted as a key new technology that will drive the Fourth Industrial Revolution, promising highly integrated smart spaces. Furthermore, AI is gaining traction as a next-generation growth engine capable of evolving industrial ecosystems beyond simply solving standardized problems. It is actively being applied not only to IT, healthcare, agriculture, energy, automobiles, and robotics, but also to knowledge service industries such as distribution, finance, law, education, real estate, advertising, and communications. In other words, AI is integrating with all existing systems, not just those that seek to improve the convenience and quality of daily life, but also across the entire spectrum of our society's culture and arts, preparing for a new era.

[0004] Even in the field of artistic creation, which was once considered difficult for artificial intelligence to handle, new revolutionary results are emerging, and various experiments are being conducted in the field of music, such as composition, performance, and imitation.

[0005] However, these existing AI composition technologies simply produce composition results according to a set pattern, so they have the problem of not faithfully reflecting the user's emotions or situational expressions.

[0006] OpenAI Blog "MuseNet" (https: / openai.com / index / musenet / , 2019,04,25)

[0007] Musebert: Pre-training music representation for music understanding and controllable generation.(ISMIR 2021, 2021.11.07, Ziyu Wang and Gus Xia)

[0008] One embodiment of the present invention aims to provide a more practical and reproducible symbolic music generation method and system by encoding musical structures using structural embeddings of digital music protocol (e.g., MIDI (musical instrument digital interface)) files and utilizing large-scale language models.

[0009] According to one embodiment of the present invention, a symbolic music generation system using a language model comprises: at least one processor; and at least one memory storing instructions or information executed by the at least one processor; wherein operations performed by the instructions or information executed by the processor include: encoding music data to generate input tokens from input digital music protocol data; performing one or more structural embeddings from the input tokens; and combining the input token embedding results with the structural embedding results to input the result to a language model and generating a next output token; wherein the one or more structural embeddings can be initialized before the language model starts learning.

[0010] Here, the input token may include at least one of N×M note-on tokens consisting of N velocities and M pitch numbers; a note-off token with 0 velocities; a time-shift token; and a special token for matching the beginning of a sentence, the end of a sentence, and the length of a sentence.

[0011] Additionally, the one or more structural embeddings may include at least one of a part embedding indicating a time portion of the entire music in which the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token.

[0012] Additionally, the one or more structural embeddings may be initialized using either a first initialization method that randomly initializes all vectors of the one or more structural embeddings, or a second initialization method that initializes time-related embeddings of the one or more structural embeddings using sine wave encoding and applies the first initialization method to non-time-related embeddings.

[0013] Additionally, the first initialization method may include a method of initializing all vectors of the one or more structural embeddings using a truncated normal distribution.

[0014] In addition, the one or more structural embeddings may include at least one of a part embedding indicating a time portion of the entire music in which the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token, and the second initialization method may include a method of preserving continuity of sequential musical properties of notes and pitches by applying sine wave weights to the part embedding and the time embedding, and a method of initializing the type embedding and the PC embedding to a truncated normal distribution.

[0015] Additionally, the embedding input to the language model may include the input token embedding, the one or more structural embeddings, and the positional embedding initialized by the first or second initialization method.

[0016] According to one embodiment of the present invention, a method for generating symbolic music using a language model that generates music using a large-scale language model is executed by at least one processor, and includes the steps of: encoding music data to generate input tokens from input digital music protocol data; performing one or more structural embeddings from the input tokens; and inputting the combined input token embedding results and the structural embedding results into a language model and training the language model to generate a next output token; wherein the one or more structural embeddings can be initialized before the language model starts training.

[0017] At this time, in the encoding step, the input token may include at least one of N×M note-on tokens composed of N velocities and M pitch numbers; a note-off token with a 0 velocities; a time-shift token; and a special token for matching the beginning of a sentence, the end of a sentence, and the length of a sentence.

[0018] In addition, in the step of performing the structural embedding, the one or more structural embeddings may include at least one of a part embedding indicating at which time part of the entire music the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token.

[0019] Additionally, the one or more structural embeddings initialized before the language model starts learning may be initialized using either a first initialization method that randomly initializes all vectors of the one or more structural embeddings or a second initialization method that initializes time-related embeddings among the one or more structural embeddings using sine wave encoding and applies the first initialization method to non-time-related embeddings.

[0020] Additionally, the first initialization method may include a method of initializing all vectors of the one or more structural embeddings using a truncated normal distribution that initializes them in the form of a probability distribution in which values ​​smaller than the minimum value or larger than the maximum value are removed from the normal distribution.

[0021] In addition, the one or more structural embeddings may include at least one of a part embedding indicating, for the input token, a time portion of the entire music in which the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token, and the second initialization method may include a method of preserving continuity of sequential musical properties of notes and pitches by applying sine wave weights to the part embedding and the time embedding, and a method of initializing the type embedding and the PC embedding to a truncated normal distribution.

[0022] Additionally, in the learning step, the embedding input to the language model may include the input token embedding, the structural embedding, and the position embedding initialized by the first or second initialization method.

[0023] A program stored in a computer-readable recording medium according to one embodiment of the present invention can be executed on a computer by at least one of the above-described methods.

[0024] According to one embodiment of the present invention, a symbolic music generation system capable of more effectively reproducing the structural characteristics of music can be provided by applying four structural embeddings constructed from digital music protocol data (e.g., MIDI data) to a large-scale language model.

[0025] Additionally, by changing the way the structural embedding of digital music protocol data is initialized, it has the effect of enabling users to generate music with desired characteristics.

[0026] Figure 1 is a schematic diagram of an electronic device according to one embodiment of the present invention.

[0027] FIG. 2 is a schematic diagram of a method for generating symbolic music using a language model according to one embodiment of the present invention.

[0028] FIG. 3 is a schematic diagram illustrating encoding and structural embedding of digital music protocol data according to one embodiment of the present invention.

[0029] FIG. 4 is a schematic diagram showing the learning process of a symbolic music generation model using a language model according to one embodiment of the present invention.

[0030] FIG. 5 is a schematic diagram showing a process of inferring music using a symbolic music generation model using a language model according to one embodiment of the present invention.

[0031] In order to clarify the technical idea of ​​the present disclosure, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, if a detailed description of a related known function or component is determined to unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In the drawings, components having substantially the same functional configuration are given the same reference numbers and symbols as possible even if they are shown in different drawings. For convenience of explanation, devices and methods are described together when necessary. Each operation of the present disclosure does not necessarily have to be performed in the described order and may be performed in parallel, selectively, or individually.

[0032] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0033] Throughout this disclosure, singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms such as "comprise" or "have" should be understood to indicate the presence of a feature, number, step, operation, component, part, or combination thereof, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. In other words, when it is said throughout this disclosure that a part "comprises" a certain component, unless specifically stated otherwise, this does not mean that other components may be included, but rather that other components may be excluded.

[0034] Expressions such as "at least one" modify the entire list of elements, not individual elements of the list. For example, "at least one of A, B, and C" and "at least one of A, B, or C" refer to only A, only B, only C, both A and B, both B and C, both A and C, all of A, B, and C, or any combination thereof.

[0035] In addition, terms such as “...unit”, “...module”, etc. described in the present disclosure mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0036] Throughout this disclosure, when a part is said to be "connected" to another part, this includes not only cases where the parts are "directly connected," but also cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise specifically stated.

[0037] The expression “configured to” as used throughout this disclosure can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system, together with other devices or components, is “capable of.” For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0038] Artificial intelligence (AI) is a field of computer engineering and information technology that studies how to enable computers to perform human-like tasks, such as thinking, learning, and self-improvement. It aims to enable computers to mimic human intelligent behavior. Furthermore, AI does not exist in isolation; rather, it is closely linked, both directly and indirectly, to other fields of computer science. In particular, efforts are actively underway to incorporate AI elements into various fields of information technology and utilize them to solve problems in those fields.

[0039] Machine learning is a branch of artificial intelligence that empowers computers to learn without explicit programming. Specifically, machine learning is the study and development of algorithms and systems that learn from empirical data, make predictions, and improve their own performance. Rather than executing strictly defined, static program instructions, machine learning algorithms build specific models based on input data to derive predictions or decisions. The term "machine learning" can be used interchangeably with "machine learning."

[0040] Many machine learning algorithms have been developed to classify data. Representative examples include decision trees, Bayesian networks, support vector machines (SVMs), and artificial neural networks (ANNs). Decision trees are an analytical method that performs classification and prediction by diagramming decision rules in a tree-like structure. Bayesian networks are models that represent probabilistic relationships (conditional independence) between multiple variables in a graph structure. Bayesian networks are suitable for data mining through unsupervised learning. Support vector machines are supervised learning models for pattern recognition and data analysis, primarily used for classification and regression analysis. Artificial neural networks model the operating principles and interconnected relationships of biological neurons. They are information processing systems in which numerous neurons, called nodes or processing elements, are connected in layers.

[0041] An artificial neural network (ANN) is a model used in machine learning. It is a statistical learning algorithm inspired by biological neural networks (especially the brain, the central nervous system of animals) in machine learning and cognitive science. Specifically, an ANN can refer to a general model in which artificial neurons (nodes) form a network through the connection of synapses, which change the strength of the synaptic connections through learning, thereby achieving problem-solving capabilities. The term "ANN" can be used interchangeably with the term "neural network."

[0042] An artificial neural network can include multiple layers, each of which can include multiple neurons. Furthermore, an artificial neural network can include synapses, which connect neurons. An artificial neural network can generally be defined by three factors: a) the connection pattern between neurons in different layers, b) a learning process that updates the weights of the connections, and c) an activation function that generates an output value from a weighted sum of the inputs received from the previous layer.

[0043] Artificial neural networks may include, but are not limited to, network models such as Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Multilayer Perceptrons (MLPs), and Convolutional Neural Networks (CNNs). In this specification, the term "layer" may be used interchangeably with the term "layer."

[0044] Artificial neural networks are categorized into single-layer neural networks and multi-layer neural networks based on the number of layers. A typical single-layer neural network consists of an input layer and an output layer. A typical multi-layer neural network consists of an input layer, one or more hidden layers, and an output layer.

[0045] The input layer is the layer that receives external data, and the number of neurons in the input layer is the same as the number of input variables, and the hidden layer is located between the input layer and the output layer. It receives signals from the input layer, extracts characteristics, and transmits them to the output layer. The output layer receives signals from the hidden layer and outputs output values ​​based on the received signals. The input signals between neurons are multiplied by each connection strength (weight) and then added, and if this sum is greater than the threshold of the neuron, the neuron is activated and outputs the output value obtained through the activation function.

[0046] Meanwhile, deep neural networks, which include multiple hidden layers between the input and output layers, are representative artificial neural networks that implement deep learning, a type of machine learning technique. The term "deep learning" can be used interchangeably with "deep learning."

[0047] The workflow of machine learning consists of a series of steps: collecting data for learning and validation, modeling, and then training the model. This can include the processes of collecting training data, inspecting and exploring the data, preprocessing and cleaning the data, modeling, and training.

[0048] 1. Collect training data

[0049] The training data applied to the learning model of this specification can be generated using data collected from multiple samples. The training data may include various digital music protocol data files (e.g., MIDI files) for symbolic music generation. For example, MIDI (musical instrument digital interface) files are digital sound files created using a computer, and music composed using MIDI can be easily edited or synthesized using a computer.

[0050] In this specification, at least one or more different types of training data sets may be used to train a learning model, and each training data set may further include one or more experiment-based results used as feature labels. At least a portion of the training data set may be used to train the learning model, and another portion may be used to validate the learned learning model.

[0051] 2. Checking and exploring data

[0052] Once training data for learning a learning model is collected, the collected training data can be inspected and explored for data structure, noisy data, and data cleaning methods for applying machine learning.

[0053] This data review and exploration phase is called Exploratory Data Analysis (EDA), and EDA can be defined as the process of observing and understanding the collected data from various perspectives. Before data learning, visualizations such as graphs and statistical tests are used to examine independent and dependent variables, variable types, and their data types, allowing for preliminary identification of data characteristics and inherent structural relationships. Through EDA, data distribution and values ​​can be examined to better understand the phenomena expressed by the data and identify potential problems. Furthermore, through the process of examining data from various perspectives, various patterns that might not have been detected during the problem definition phase can be discovered, allowing for modification of existing hypotheses or the development of new ones. Exploratory data analysis can broadly include the process of searching for outliers and analyzing the relationships between data attributes.

[0054] The process of detecting outliers involves determining whether data contains outliers. This process can involve sampling, statistical methods, and visualization methods. Sampling methods extract a random sample from the data to identify overall trends and anomalies in the data values. Statistical methods can utilize summary statistics such as the mean, median, and mode to determine the center of the data, or the range and variance to determine the distribution of the data. Visualization methods can utilize probability density functions, histograms, dotplots, word clouds, time series charts, and maps to determine which statistical indicators are appropriate for each attribute of the collected data. However, when using statistical indicators, it is important to note that the mean reflects all data values ​​in the set, so outliers can affect the value, whereas the median uses the single value in the middle, so it can produce representative results even with outliers.

[0055] The process of analyzing the relationship between data attributes is to find combinations of attributes that have meaningful correlations within the data. The relationship analysis can be performed differently depending on the combination of attributes between qualitative attributes (Categorical Variable; Qualitative) that cannot be expressed numerically but can be arbitrarily quantified and quantitative attributes (Numeric Variable; Quantitative). The qualitative-qualitative relationship (Categorical - Categorical) can be displayed by using cross tables and mosaic plots to count the number of values ​​corresponding to each pair of attribute values. The quantitative-qualitative relationship (Numeric-Categorical) can be visually expressed by observing statistical values ​​(mean, median, etc.) by category or using box plots. The quantitative-quantitative relationship (Numeric-Numeric) can be analyzed for the association between two attributes using correlation coefficients. A correlation coefficient of -1 indicates a negative correlation where the two attributes change in opposite directions, 0 indicates no correlation, and 1 indicates a positive correlation where the two attributes always change in the same direction. The relationship between two attributes with a correlation coefficient can take many forms, and this can be visually represented using a scatter plot.

[0056] 3. Data preprocessing and cleaning

[0057] Once the data has been inspected and explored, data preprocessing is performed to transform it into a format suitable for machine learning training models. Data preprocessing involves refining data and transforming it into a form understandable by the model. Data preprocessing typically includes handling missing data, removing outliers, scaling, categorical data encoding, feature selection and extraction, and data transformation. The detailed data preprocessing steps can be performed in whole or in part, and a separate machine learning model may be used for data preprocessing.

[0058] Handling missing data involves handling missing values ​​in data. Missing values ​​can be displayed as NaN (Not a Number) or blank, or they can be deleted. Filling in or deleting missing values ​​within the data improves data completeness. When filling in missing values, values ​​such as the mean, median, or mode can be used.

[0059] Outlier removal is the process of removing outliers, values ​​that deviate from the normal data pattern. Outliers can degrade model performance and should therefore be removed or replaced. Identifying outliers can be accomplished by deleting the corresponding rows or columns or replacing them with different values.

[0060] Data scaling is the process of adjusting the size of data. Through data scaling, the range of data can be adjusted, and the performance of the model or the convergence speed can be improved. Through data scaling, the characteristics of the data can be adjusted to a similar range, and data scaling can generally be applied with standardization and normalization. Standardization is a method of converting data into a distribution with a mean of 0 and a standard deviation of 1, and is mainly converted using the mean and standard deviation, and the standardized value z is It can be expressed as (x is the original value, μ is the mean, σ is the standard deviation). Normalization is a method to convert the range of data to [0,1] or [-1,1], and mainly converts data using the minimum and maximum values, and the normalized value x norm silver can be expressed as (x is the original value, x min is the minimum, x max is the maximum value).

[0061] Categorical data encoding is the process of converting categorical variables, represented as strings or integers that cannot be directly input into a model, into numerical data types that can be input into the model. Typically, one-hot encoding or label encoding is used to convert categorical variables into numerical data types.

[0062] Feature selection and extraction is a process to improve model performance by selecting the most useful features for model learning or extracting new features. This process can reduce model complexity and prevent overfitting.

[0063] Data transformation is the process of transforming data to extract new information or to improve model understanding. This can include tokenizing text data or preprocessing image data. Data transformation can extract useful features from source data or transform data into an appropriate format, improving model performance.

[0064] Through data preprocessing as described above, the performance of machine learning models can be improved and stability can be secured.

[0065] Meanwhile, when training a learning model based on training data including music information according to one embodiment of the present invention, a process of preprocessing information expressed in natural language and a process of learning a language model based on the preprocessed data may be performed.

[0066] 3-1. Text Preprocessing for Large-Scale Language Models

[0067] If the data collected to analyze the structure of music, such as digital music protocol data, has not been preprocessed to suit the needs, tokenization, cleaning, and normalization can be performed to suit the intended use of the data.

[0068] Tokenization refers to the process of dividing given data into units called tokens, which can be broadly defined as meaningful units. Tokenization can broadly include word tokenization and sentence tokenization.

[0069] Word tokenization refers to cases where tokens are based on words, and in this case, words can include not only individual words but also word phrases and meaningful strings. Word tokenization separates words based on spaces or punctuation marks, such as periods, commas, question marks, semicolons, and exclamation marks. However, removing all punctuation or special characters during tokenization can sometimes result in tokens losing their meaning, necessitating a more precise tokenization algorithm. For example, if a word itself contains punctuation or uses special characters with meaning, simply removing them may not be enough. Therefore, tokenization rules such as the Penn Treebank Tokenization Rules can be applied during tokenization.

[0070] Sentence tokenization refers to the process of dividing text into sentences. Typically, unrefined data is not segmented into sentences, requiring sentence tokenization to suit the intended use. Various rules for sentence tokenization can be defined depending on the language used and how special characters are used within the corpus.

[0071] Tokenization is the process of classifying tokens according to their intended use. Before and after tokenization, cleaning and normalization are performed on text data to suit the intended use. Cleaning removes noise, while normalization integrates words with different representations and transforms them into a single, consistent word.

[0072] Refinement can occur before tokenization to eliminate any interference and facilitate tokenization. However, it can also be performed continuously and iteratively after tokenization to remove any remaining noise. The noise data removed during refinement are meaningless characters. Methods for removing unnecessary words include removing stopwords, low-frequency words, and short words.

[0073] Normalization work includes unifying words with different spellings based on rules, unifying uppercase and lowercase letters, etc. Unifying uppercase and lowercase letters is a normalization method that can reduce the number of words in English-speaking languages. In English-speaking languages, uppercase letters are only used in certain situations such as the beginning of a sentence, and most texts are written in lowercase letters, so unifying uppercase and lowercase letters can mostly be done by converting uppercase letters to lowercase letters.

[0074] Processing natural language in computing systems requires preprocessing, which involves digitizing text. This involves mapping each word in the text to a unique integer. This mapping process can utilize techniques such as integer encoding, padding, and one-hot encoding.

[0075] Integer encoding is a method of assigning integers to words. It creates a vocabulary by sorting words in order of frequency, and assigns integers in order of frequency, starting with the lowest number. Integer encoding performs sentence tokenization on text data containing multiple sentences, and performs word tokenization through parallel refinement and normalization. During this process, words are lowercase to unify the number of words, and stopwords and word length can be deleted. Through this, words can be recorded as keys and the frequency of each word as values. Integer encoding can be performed by sorting the text in order of frequency and assigning integers to words with high frequencies.

[0076] Padding is the process of randomly adjusting the length of sentences of different lengths within a text to the same length. Computing systems can perform parallel computations by grouping sentences of the same length into a single matrix. Specifically, to perform parallel computations, the lengths of sentences of different lengths within a text can be randomly padded with "0" to equalize the integer encoding results. Specifically, the longest sentence in a set of integer-encoded words can be identified, and a "0" can be added to the integer matrix corresponding to the length of the longest sentence. The computing system can then process sentences of the same length as a single matrix, allowing it to perform parallel processing. At this point, the computing system can ignore the "0" word, which is perceived as meaningless. This process of adjusting the size (shape) of data by filling it with a specific value is called padding. Using the number "0" to adjust the length is called zero padding.

[0077] One-hot encoding is a vector representation of words that uses the size of the vector as the dimension of the word set, assigning a value of 1 to the index of the word to be expressed, and 0 to all other indices. The vector expressed in this way is called a one-hot vector. One-hot encoding consists of integer encoding and an index assignment process. After integer encoding is performed and a unique integer is assigned to each word, the unique integer of the word to be expressed is regarded as an index, and a "1" is assigned to the corresponding position, and a "0" is assigned to the index positions of other words. However, one-hot encoding has the disadvantage that the space required to store the vector increases as the number of words increases (the dimensionality of the vector increases), and the similarity between words cannot be identified. To address these shortcomings, techniques that reflect the latent meaning of words and vectorize them into a multidimensional space include LSA (Latent Semantic Analysis), a count-based vectorization method; NNLM, RNNLM, Word2Vec, and FastText, which vectorize based on prediction; and the GloVe method, which uses both count-based and prediction-based methods.

[0078] Meanwhile, for computers to understand and process text, it must be appropriately converted into numbers. Because the performance of natural language processing can vary significantly depending on how words are represented, numerous techniques have been proposed to quantify words. Currently, the most widely used method is word embedding, which vectorizes each word through artificial neural network training.

[0079] Word embedding is a method of representing words as vectors, converting them into dense representations. The resulting word embedding is called a dense vector, or embedding vector. Word embedding methods include LSA, Word2Vec, FastText, and Glove.

[0080] 4. Modeling and Training

[0081] Artificial neural networks can be trained using training data. Here, "training" refers to the process of determining the parameters of an artificial neural network using training data to achieve objectives such as classification, regression analysis, or clustering of input data. Representative examples of artificial neural network parameters include the weights assigned to synapses and the biases applied to neurons.

[0082] An artificial neural network trained using training data can classify or cluster input data based on its patterns. Meanwhile, an artificial neural network trained using training data is referred to herein as a "trained model."

[0083] The following explains the learning methods of artificial neural networks. Learning methods of artificial neural networks can be broadly categorized into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0084] Supervised learning is a machine learning method that infers a function from training data. Among these inferred functions, regression analysis is the process of outputting continuous values, while classification is the process of predicting and outputting the class of an input vector.

[0085] In supervised learning, an artificial neural network is trained with labels for training data. Here, the label can mean the correct answer (or result value) that the artificial neural network should infer when training data is input to the artificial neural network. In this specification, the correct answer (or result value) that the artificial neural network should infer when training data is input is called a label or labeling data. In addition, in this specification, setting a label on training data for learning of the artificial neural network is called labeling the training data. In this case, the training data and the label corresponding to the training data constitute a single training set, and can be input to the artificial neural network in the form of a training set.

[0086] Meanwhile, training data represents multiple features, and labeling the training data can mean that the features represented by the training data are labeled. In this case, the training data can represent the features of the input object in vector form. An artificial neural network can use the training data and labeled data to infer a function regarding the relationship between the training data and the labeled data. Furthermore, the parameters of the artificial neural network can be determined (optimized) by evaluating the function inferred by the artificial neural network.

[0087] Unsupervised learning is a type of machine learning in which training data is not labeled. Specifically, unsupervised learning can be a learning method that trains an artificial neural network to find and classify patterns in the training data itself, rather than the relationship between the training data and the corresponding labels. Examples of unsupervised learning include clustering and independent component analysis (ICA). In this specification, the term "clustering" may be used interchangeably with the term "clustering."

[0088] Examples of artificial neural networks that utilize unsupervised learning include generative adversarial networks (GANs) and autoencoders (AEs).

[0089] Generative adversarial networks (GANs) are a machine learning method in which two different AI components, a generator and a discriminator, compete to improve performance. In this case, the generator is a model that creates new data, capable of generating new data based on original data. The discriminator, a model that recognizes data patterns, can determine whether the input data is original or new data generated by the generator. The generator learns from data that fails to fool the discriminator, while the discriminator learns from data that the generator deceives. Accordingly, the generator can evolve to fool the discriminator as effectively as possible, while the discriminator can evolve to effectively distinguish between original data and data generated by the generator.

[0090] An autoencoder is a neural network that aims to reproduce the input itself as an output. An autoencoder comprises an input layer, at least one hidden layer, and an output layer. In this case, since the number of nodes in the hidden layer is smaller than that in the input layer, the data dimensionality is reduced, leading to compression or encoding. Furthermore, data output from the hidden layer is fed into the output layer. In this case, since the number of nodes in the output layer is larger than that in the hidden layer, the data dimensionality increases, leading to decompression or decoding.

[0091] Meanwhile, autoencoders express input data as hidden layer data by adjusting the connection strengths of neurons through learning. The hidden layer expresses information with a smaller number of neurons than the input layer. The ability to reproduce input data as output implies that the hidden layer has discovered and expressed hidden patterns in the input data.

[0092] Semi-supervised learning is a type of machine learning that utilizes both labeled and unlabeled training data. One technique for semi-supervised learning is to infer labels for unlabeled training data and then use these inferred labels to perform training. This technique can be useful in situations where labeling is expensive.

[0093] Reinforcement learning is the theory that, if an agent is given an environment in which it can determine the optimal action at any given moment, it can find the optimal path through experience without data. Reinforcement learning is primarily implemented using a Markov Decision Process (MDP). A Markov Decision Process is described as follows: first, an environment containing the information necessary for the agent to take the next action is provided; second, how the agent will act in that environment is defined; third, what rewards the agent will receive for performing well and what penalties will be imposed for performing poorly is defined; and fourth, the optimal policy is derived through repeated experience until the future reward reaches its maximum.

[0094] The structure of an artificial neural network is specified by the model configuration, activation function, loss function or cost function, learning algorithm, optimization algorithm, etc., and the hyperparameters are set in advance before learning, and the model parameters are set through learning afterwards, so that the content can be specified.

[0095] For example, factors that determine the structure of an artificial neural network may include the number of hidden layers, the number of hidden nodes included in each hidden layer, the input feature vector, and the target feature vector.

[0096] Hyperparameters include various parameters that must be initially set for learning, such as initial values ​​for model parameters. Furthermore, model parameters include various parameters to be determined through learning. For example, hyperparameters may include initial values ​​for inter-node weights, initial values ​​for inter-node biases, mini-batch size, number of learning iterations, and learning rates. Furthermore, model parameters may include inter-node weights, inter-node biases, and more.

[0097] The loss function can be used as an indicator (standard) to determine the optimal model parameters during the learning process of an artificial neural network. In an artificial neural network, learning refers to the process of manipulating model parameters to reduce the loss function, and the purpose of learning can be seen as determining the model parameters that minimize the loss function. The loss function can mainly use the mean squared error (MSE) or the cross entropy error (CEE), but the present invention is not limited thereto. The cross entropy error can be used when the correct answer label is one-hot encoded. One-hot encoding is an encoding method that sets the correct answer label value to 1 only for neurons corresponding to the correct answer, and sets the correct answer label value to 0 for neurons that are not the correct answer.

[0098] In machine learning or deep learning, learning optimization algorithms can be used to minimize the loss function. Learning optimization algorithms include gradient descent (GD), stochastic gradient descent (SGD), momentum, Nesterov Accelerate Gradient (NAG), Adagrad, AdaDelta, RMSProp, Adam, and Nadam.

[0099] Gradient descent is a technique that adjusts model parameters in a direction that reduces the loss function value by considering the gradient of the loss function at the current state. The direction of model parameter adjustment is called the step direction, and the size of the adjustment is called the step size. Here, the step size can represent the learning rate. Gradient descent obtains the gradient by partially differentiating the loss function with respect to each model parameter, and updates the model parameters by changing the learning rate in the direction of the obtained gradient.

[0100] Stochastic gradient descent is a technique that divides learning data into mini-batches and performs gradient descent on each mini-batch to increase the frequency of gradient descent.

[0101] Adagrad, AdaDelta, and RMSProp are techniques for improving optimization accuracy by adjusting the step size in SGD. In SGD, momentum and NAG are techniques for improving optimization accuracy by adjusting the step direction. Adam combines momentum and RMSProp to improve optimization accuracy by adjusting the step size and step direction. Nadam combines NAG and RMSProp to improve optimization accuracy by adjusting the step size and step direction.

[0102] The learning speed and accuracy of artificial neural networks are significantly influenced by not only the network structure and the type of learning optimization algorithm, but also hyperparameters. Therefore, to obtain a good learning model, it is crucial not only to determine an appropriate artificial neural network structure and learning algorithm, but also to set appropriate hyperparameters.

[0103] Typically, hyperparameters are experimentally set to various values ​​while training an artificial neural network, and the learning results are set to the optimal values ​​that provide stable learning speed and accuracy.

[0104] Figure 1 is a schematic diagram of an electronic device according to one embodiment of the present invention.

[0105] As illustrated in FIG. 1, an electronic device (100) (hereinafter referred to as an electronic device) according to an embodiment of the present invention may include at least one processor (110), a memory (120), and a communication unit (130). The electronic device (100) is a basic configuration for performing a computing environment, and in other embodiments, the electronic device (100) may be implemented by additionally or alternatively including some other components, may be implemented as a single or multiple entities, or may be implemented as only some of the disclosed components. Components or at least some of the components inside or outside the electronic device (100) may be connected to each other through a BUS, a GPIO (General Purpose Input / Output), an SPI (Serial Peripheral Interface), or a MIPI (Mobile Industry Processor Interface), thereby transmitting and receiving data or signals.

[0106] Unless the context clearly indicates otherwise, the processor (110) may refer to a set of one or more processors, and may control components of the processor (110) and the electronic device (100) by executing software (e.g., commands, programs, etc.) stored in at least the memory (120). In addition, the processor (110) may perform various operations such as calculations, processing, data generation or processing, and may read data from or store data in the memory (120). The processor (110) may be composed of at least one core and may include a processor for data analysis, machine learning (ML), or deep learning (DL), such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU). The processor (110) may read software stored in the memory (120) to perform data processing for machine learning (or deep learning) of the present invention. According to one embodiment of the present disclosure, the processor (110) can perform operations for learning a neural network. The processor (110) can perform calculations for learning a neural network, such as processing input data for learning in deep learning, extracting features from the input data, calculating errors, and updating weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) can process learning of a neural network model. For example, the CPU and GPGPU can together process learning of a neural network model and data classification using a neural network model. In addition, in one embodiment of the present disclosure, at least one processor (110) of the electronic device (100) can be used together to process learning of a neural network model and data classification using a neural network model.

[0107] The memory (120) is for storing various data, and the data is data acquired, processed, or used by at least one component of the electronic device (100), and may include software (e.g., commands, programs, etc.). Unless explicitly expressed otherwise in the context, the memory (120) may refer to a set of one or more memories, and may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, an optical disk, and a web storage that performs a storage function on the Internet. The instructions or programs or software stored in the memory (120) may be used to refer to an operating system, an application for controlling components of the electronic device (100), or middleware that provides various functions to the application so that the application can utilize the components of the electronic device (100). In one embodiment, when the processor (110) performs a specific operation, the memory (120) may store instructions that are performed by the processor (110) and correspond to the specific operation.

[0108] The communication unit (130) performs wireless or wired communication between the electronic device (100) and another device (e.g., a user terminal or another server), and the communication unit (130) can use wireless communication systems according to methods such as eMBB, URLLC, MMTC, LTE, LTE-A, NR, UMTS, GSM, CDMA, WCDMA, TDMA, FDMA, OFDMA, SCFDMA, WiBro, WiFi, Bluetooth, NFC, GPS, or GNSS. In addition, the communication unit (130) can use various wired communication systems such as USB, HDMI, RS-232 (Recommended Standard-232), POTS (Plain Old Telephone Service), PSTN (Public Switched Telephone Network), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and local area network (LAN). In one embodiment of the present invention, the communication unit (130) can be configured regardless of the communication mode such as wired or wireless, and can be configured with various communication networks such as a personal area network (PAN), a wide area network (WAN), etc. Additionally, the network may be the well-known World Wide Web (WWW), or may utilize a wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. The technologies described in one embodiment of the present invention can also be used in other networks mentioned above.

[0109] An electronic device (100) according to one embodiment of the present invention can configure a symbolic music generation system (100) using a large-scale language model or execute software that configures a symbolic music generation method using a large-scale language model.

[0110] A symbolic music generation system (100) according to an embodiment of the present invention can improve upon the shortcomings of existing music generation techniques. Existing music generation techniques require domain-specific annotations, such as bars and beats, that are missing from raw digital music protocol data (e.g., MIDI data) for tokenization. Furthermore, existing token embedding methods have shown little effectiveness without domain-specific annotations. To improve upon these limitations or shortcomings, one embodiment of the present invention proposes a music generation framework based on digital music protocol data.

[0111] The music generation framework according to one embodiment of the present invention can effectively encode musical structure using two structural embeddings without domain-specific annotations of digital music protocol data. The music generation framework of the present invention can generate practical and reproducible music by utilizing a large-scale language model.

[0112] A symbolic music generation method according to an embodiment of the present invention may include a technique for expressing music as symbols or codes and creating new music through algorithms and computer programs. Using this symbolic music generation method, the patterns and structures of music can be analyzed and new music can be generated based on these. The symbolic music generation method may include the processes of music data representation, algorithms, model selection and learning, and music generation. When generating symbolic music, the beginning and end of each note, pitch, and velocity of the music can be expressed as symbols within a digital music protocol file.

[0113] Various algorithms or machine learning models can be used to generate symbolic music. For example, the Music Transformer, a deep learning model that applies the Transformer model, used in natural language processing, to music generation, can be used. The Music Transformer model can learn the structural patterns of music and generate music with a consistent style. By learning a large amount of music data using a selected algorithm, the Music Transformer model can understand the patterns and rules of music. This learned model can then be used to generate new music. Through this process, symbolic music generation technology can be used to create creative and novel music.

[0114] These symbolic music generation techniques are rapidly growing in the field of machine learning, and the number of music datasets for training large-scale music generation models is steadily increasing. Typically, the Transformer architecture can be applied to music due to its sequential nature. Frequently used in Natural Language Processing (NLP), the Transformer architecture requires vertical (multiple notes played simultaneously) and temporal (all notes played for a specific duration) structures in music data, which necessitates a different tokenization procedure than the typical positional encoding used in text generation.

[0115] MethodToken TypesREMINote-On, Duration, Tempo, Position, Bar, ChordCPTrack, Tempo, Position, Bar, Chord, Pitch, Duration, Velocity, FamilyMIDI-likeNote-On, Note-Off, Time-Shift, VelocityMuseNetPitch-Volume-Instrument, Wait(Time-Shift)

[0116] [Table 1] Examples of existing methods for tokenizing digital music protocol (MIDI) data Among the existing publicly available tokenization methods for digital music protocol data (MIDI data), REMI (Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions, Yu-Siang Huang, 2020) and CP (compound word representation, Compound word transformer: Learning to compose full-song music over dynamically directed hypergraphs, Wen-Yi Hsiao, 2021) tokenize based on annotations generated by the worker in the MIDI data. MIDI-like tokenization (MUSIC TRANSFORMER: GENERATING MUSIC WITH LONG-TERM STRUCTURE, Cheng-Zhi Anna Huang, 2018) is known to have the highest compatibility with large-scale music data on cloud-based platforms. However, these publicly known MIDI data tokenization methods fundamentally require annotations for the temporal division of bars or chords in MIDI data. Creating annotations in MIDI data is typically done manually by human operators, a process that is time-consuming and expensive.

[0117] Another approach is to use heuristic algorithms on digital music protocol data (e.g., MIDI data). However, these methods are only reliably applicable when the input music progresses at a constant speed or tempo and represents well-defined bars and beats. However, most MIDI data transcribed directly from live music audio frequently introduces notes that deviate from the scripted tempo when played by the performer. This means that heuristic algorithms are difficult to apply to digital music protocol data (e.g., MIDI data) derived from live music audio.

[0118] MuseNet was proposed as a vanilla MIDI tokenization method to effectively encode musical structure without domain-specific annotations in digital music protocol data (e.g., MIDI data). MuseNet introduces additional structural embeddings to better extract the structural context of music from MIDI-based tokens, demonstrating high performance in prompt-based music generation. Released by OpenAI in 2019, MuseNet is a deep neural network for music composition. It learns from digital music protocol data and has a structure that predicts the next key of an input key.

[0119] A symbolic music generation system (100) according to one embodiment of the present invention can encode a musical structure based on raw digital music protocol data (e.g., MIDI data) based on the structural embedding proposed by MuseNet. Specifically, one embodiment can build a structural embedding by taking into account noise in raw digital music protocol data (e.g., MIDI data) by training a vanilla GPT-2 model that implements the structural embedding of MuseNet.

[0120] As illustrated in FIG. 2, a method for generating symbolic music using a language model implemented in a symbolic music generation system (100) using a language model according to one embodiment of the present invention may include a step of constructing a data set (S110); a step of encoding music data (S120); a step of performing structural embedding (S130); a step of training a music generation model (S140); a step of initializing at least one of the structural embeddings during training (S150); and a step of performing inference using the trained music generation model (S160).

[0121] Step for building a data set (S110)

[0122] The data sets for model learning of the symbolic music generation system (100) of one embodiment of the present invention may be Pop1k7 and GiantMIDi-Piano data sets, which are composed of digital music protocol data files (e.g., MIDI files) transcribed from source audio recordings collected from a music streaming service (e.g., YouTube). The training set for model learning may be composed of 1,748 songs from Pop1k7 and 5,842 songs from GiantMIDi-Piano after pruning music of more than 10 minutes from GiantMIDi-Piano. Additionally, the training set may be subjected to two data augmentations, which may include, first, pitch transposition by one of the half steps {-3, -2, -1, 0, 1, 2, 3} and second, time stretching, in which the overall tempo is scaled to one of the steps {0.95, 0.975, 1.1, 1.025, 1.05}. Consequently, the training set may include a total of 17,383 hours of transcribed digital music protocol data (e.g., MIDI data). A test data set for testing the model may include all songs in the MAESTRO data set, which consists of 1,276 live MIDI piano performances performed by virtuoso pianists. Although the data set of the present embodiment is structured as described above, it is not limited thereto, and the structure of the data set may be freely modified by those skilled in the art.

[0123] Step of encoding music data (S120)

[0124] A symbolic music generation system (100) of one embodiment can encode music data to generate input tokens from input digital music protocol data. At this time, an event-based tokenization method of MuseNet, a music generation model, can be applied to encode music data into a token sequence. In this tokenization method, all meta information in the digital music protocol data is deleted so that only different input tokens are used. The input tokens are composed of 32 different velocities and 128 pitch numbers (e.g., <v64:c4>) consisting of 4,096 note-on tokens and 128 note-off tokens with a speed of 0 (e.g. <v0:c4>) and 100 time-shift tokens (e.g. <wait:2>) can be included. Also, to arrange pieces with different durations into a fixed sequence length, <bos>(being of sequence; beginning of a sentence), <eos>(end of sequence; end of sentence), <pad>Special tokens such as (pad token; a token to adjust the length of a short sentence to a longer sentence) may also be used.

[0125] Step of performing structural embedding (S130)

[0126] As illustrated in FIG. 3, the symbolic music generation system (100) of one embodiment can perform one or more structural embeddings from input tokens. For example, the symbolic music generation system (100) can construct four structural embedding layers for digital music protocol data based on the structural embedding of MuseNet, a music generation model. As illustrated in FIG. 3, the four embedding layers output structural embedding results, and the structural embeddings can be projected by being linked to token embeddings and then added to position embeddings. The structural embeddings can be pre-trained by a first model such as MuseNet, and then output and stored. Alternatively, the structural embeddings can be output based on input tokens by adding an embedding layer to the front end of the symbolic music generation model of the present embodiment. Structural embeddings can include Part embeddings, Type embeddings, Time embeddings, and PC (Pitch-Class; pitch class, note type) embeddings, and the class index for each embedding can be configured as follows.

[0127] Part embeddings indicate which time segment of the entire music (song) each token of digital music protocol data is in. Each token of the music can be assigned to one of 128 equally separated parts in real time encoded through time shift tokens. Consequently, each token is indexed from 0 to 128, where 0 represents a non-note token, for example, <bos>me <eos>, and 1 represents the actual beginning of the music (song). That is, the Part embedding divides the total time length of the music into 128 equally, and then calculates the cumulative time of each token using the time token within the sequence in the event token sequence, and the part class of each token can be determined according to the cumulative time of the token.

[0128] Type embeddings can indicate the type of each token. The type class for each token can be determined based on whether the token is associated with an event: a note-on token (0), a note-off token (1), a time-shift token (2), or a special token (3). These type embeddings allow the model to more easily distinguish between different token types and learn inter-token dependencies within each token type. Without type embeddings, all tokens would be learned independently, regardless of their unique type.

[0129] The time embedding represents how much relative time has passed for each token, in 10-ms units, since the most recent time-shift token (the time-shift token). The time embedding can be directly assigned to the note-on or note-off token that follows each time-shift token, allowing for a more direct encoding of temporal progression. All time-shift tokens are assigned a time embedding of 0, and when a time token appears in the sequence, its time-shift value (i.e., <time-2>In this case, the time shift value is 2) and is assigned to the time class for the next non-time token. This embodiment uses relative time rather than absolute time because the maximum length of each music (song) may vary in the data set.

[0130] PC embeddings represent the pitch class of each note-on token or note-off token. A pitch class (PC) refers to the set of all notes with the same note name, and refers to a set of specific pitches with the same note name existing in all octaves. Since each note-on token or note-off token contains not only the pitch number of the digital music protocol data but also the velocity of the digital music protocol data, the pitch class of each token is assigned to the PC class of the PC embedding, and other tokens can be assigned 0. MuseNet originally used only the relative position of each note within the code, leaving the interpretation of the chord as a group of notes unclear. By utilizing the pitch class of each note-related token, the timbre of each token can be clearly expressed. In other words, by incorporating PC embeddings, the model can directly accumulate timbre information through self-attention, allowing the harmonic context of the input to be analyzed.

[0131] Learning stage (S140)

[0132] As illustrated in FIG. 4, a symbolic music generation system (100) according to an embodiment of the present invention can learn (train) a symbolic music generation model using a first model, a music generation module, MuseNet, and a second model, a large-scale language model, GPT-2, as a backbone architecture. When training a symbolic music generation model, the four structural embeddings described above are concatenated to input token embeddings along the feature axis, and the resulting embeddings can be projected into a hidden embedding dimension through a fully-connected layer. Before providing the embeddings as input to GPT-2, trainable sequence positional encodings, which are randomly initialized as in vanilla GPT-2, can be added. Information about the position of tokens can be injected into the language model through positional encoding. During training, the symbolic music generation model can project the four structural embeddings into the input token size for predicting the next token after they are concatenated. A symbolic music generative model can perform a task similar to language modeling for training: predicting the next token.

[0133] In this embodiment, the symbolic music generation system (100) according to one embodiment can use AdamW with a learning rate of 1e-4 that linearly decreases to 0 after 1,000 warm-up steps for training the model. A maximum sequence length of 1,024 can be used, and a total of 144,786 training iterations or 6 epochs can be executed. The GPT-2 architecture is used as the base optimizer for training, and the maximum length of a token can be set to 1,024. All test samples are inferred from token prompts by MuseNet, and a short token sequence at the beginning of each music (song) of MAESTRO can be used as a prompt. X = [x1, …, x N ] is a sequence of input tokens, where N is the length of the sequence, and at each step the model learns L = -∑logp(x n | x <i ) to optimize x <i in x i Learn how to predict .

[0134] Step for initializing structural embedding (S150)

[0135] To verify the effectiveness of structural embedding of a symbolic music generation model, a symbolic music generation system (100) according to one embodiment can initialize at least some of the four structural embeddings. The initialization of the structural embeddings can include a method of randomly initializing all four structural embeddings (truncated normal initialization) (a first initialization method) and a method of initializing the part embedding and the time embedding using sinusoidal weights and initializing the type embedding and the PC embedding using the first initialization method (a second initialization method).

[0136] The first initialization method is to initialize the values ​​of the initial embedding to random values ​​or a specific distribution shape, and initialize the embedding to a probability distribution shape that removes values ​​smaller than the minimum value or larger than the maximum value from the normal distribution.

[0137] The second initialization method applies sinusoidal weights to the temporal embeddings (part, time) and a truncated normal distribution initialization to the other embeddings (type, PC). This initialization method is inspired by the use of sinusoidal encoding to preserve the continuity of sequential musical properties such as notes and pitches. A sinusoidal lookup table (SLO) can be applied to the temporal embeddings (part, time).

[0138]

[0139]

[0140] Here, k is the class index of the part embedding or temporal embedding, i is the feature index, d is the hidden size, and w is a scaling factor. Using two different values ​​for w allows us to express the properties of the part embedding and the temporal embedding without interference, as the embeddings are orthogonal. Considering that the part embedding consists of much larger units than the time embedding (10 ms), we can heuristically set w = 10 and w = 1, respectively.

[0141] The symbolic music generation model equipped with structured embeddings randomly initialized according to the first initialization method is named GPT2-RE, and the model equipped with structured embeddings initialized according to the second initialization method is named GPT2-SE, so that their performance can be compared with that of GPT2, a vanilla language model not equipped with structured embeddings.

[0142] Step of inference using the learned music generation model (S160)

[0143] As illustrated in FIG. 5, a symbolic music generation system (100) according to one embodiment can infer the next token for an input token using a trained symbolic music generation model that applies four structural embeddings. During inference, the symbolic music generation model generates a sequence through autoregressive generation when given a token prompt from digital music protocol data. Each time a token is generated as an output for an input token during inference, feature values ​​are generated using a rule-based algorithm to perform additional embedding, which can then be used to generate the token for the next time step. That is, after generating a token, the four structural embeddings (part, type, time, PC) of the token can be inferred from all subsequent tokens by the rule-based module and provided to the model for the generation step for the output of the next token. In this embodiment, sampling is repeated five times to reduce the randomness of the random generation during inference, and all generated samples can be truncated to a maximum length of one minute to save computational costs.

[0144] The symbolic music generation system (100) using a large-scale language model according to an embodiment of the present invention, having the configuration described above, and the symbolic music generation method implemented by the system can be evaluated in performance according to the initialization method of structural embedding.

[0145] The symbolic music generation model equipped with structured embeddings randomly initialized according to the first initialization method is named GPT2-RE, and the model equipped with structured embeddings initialized according to the second initialization method is named GPT2-SE, so that their performance can be compared with that of GPT2, a vanilla language model not equipped with structured embeddings.

[0146] The qualitative evaluation of the three models shows that the model that used randomly initialized four embeddings ("GPT2-RE") performed the best qualitatively, followed by the model that initialized temporal embeddings with sinusoidal encoding ("GPT2-SE"), and the model that did not use any of the four embeddings ("GPT2") received the lowest score.

[0147] Quantitatively, GPT2-SE scored the best in the metric of how repetitive the note pattern is (structure indicator; SI) and how well the chord progression follows and is typical of the distribution of the correct dataset (ground truth, GT) (chord progression variation rationality; CPVR), followed by GPT2-RE, and GPT2 scored the lowest.

[0148] This demonstrates that the music generation performance of the symbolic music generation model is further improved when additional embeddings are combined, while qualitative improvements beyond simply repeating patterns or typical chord progressions are required to achieve a qualitatively good evaluation.

[0149] Additionally, the method of initializing the time-related embedding with sinusoidal encoding (the second initialization method) tends to create more repetitive patterns for relatively shorter sections compared to the random initialization method (the first initialization method), and thus is expected to be highly useful when creating music that requires repetitive patterns.

[0150] Due to this, when implementing and using a symbolic music generation model, the practicality of the symbolic music generation system (100) can be increased by selecting a different embedding initialization method depending on what kind of music characteristics the user wants to create.

[0151] Meanwhile, embodiments of the present invention may be implemented as an application-specific integrated circuit (ASIC) manufactured to suit the special functions of a specific application field and device.

[0152] An application-specific integrated circuit is also called an application-specific semiconductor. Unlike standard semiconductors that have set specifications and can be applied to any electronic product or application as long as certain requirements are met, an application-specific semiconductor is an integrated circuit that a semiconductor manufacturer manufactures according to a specific order for a specific product or function. In other words, an application-specific semiconductor is designed and manufactured to perform only the functions required for a specific device or specific function. Depending on the design method, application-specific semiconductors are largely divided into full custom ICs, which design and manufacture the circuit from scratch according to the user's needs, and semi-custom ICs, which design and manufacture the circuit using some of the standardized design.

[0153] Application-specific semiconductors are primarily used in communications systems, high-performance computing systems, consumer electronics, automobiles, industrial automation, medical devices, military, and aerospace industries. Recently, they are being applied to AI semiconductors that perform large-scale calculations required for AI implementation with high performance and power efficiency.

[0154] Application-specific integrated circuits (ASICs) are core components of network routers, switches, and modems in communication systems, performing data packet processing, protocol conversion, and signal processing to deliver high throughput and low latency. In high-performance computing systems, ASICs are key components for high-speed and parallel processing. In consumer electronics such as digital cameras, smartphones, tablets, and game consoles, ASICs provide high-performance and low-power solutions required to perform specific functions. In the automotive industry, ASICs control various electronic systems within vehicles, and in industrial automation systems, ASICs provide solutions for high-precision control and high-performance processing.

[0155] An application-specific integrated circuit to which an embodiment of the present invention is applied includes a memory in which an individual memory interface (I / F) is implemented, and may include a plurality of functional blocks that request memory access. Each functional block may be a direct memory access (DMA) functional block, a processor, a video processor, a cache controller, a decompression block, or a data path block. The basic configuration of the application-specific integrated circuit may include a transistor that amplifies or switches an electrical signal, a logic gate that is a circuit that performs a logical function by combining transistors, a memory cell that stores data, an analog circuit that is a circuit that processes a continuous voltage or current by combining transistors, and an IP core (Intellectual Property Core) such as a microprocessor, DSP, or graphic core that is pre-designed to perform a specific function.

[0156] The ASIC may also include a separate memory I / F interfacing with individual memories and an embedded memory I / F interfacing with embedded memories. The separate memory I / F is connected to each functional block, receives memory access signals (e.g., control signals, address signals, and data signals), and generates signals for controlling the individual memories based on these input signals. The embedded memory I / F is connected to each functional block, receives memory access signals (e.g., control signals, address signals, and data signals), and generates modified memory access signals for controlling the embedded memories based on these input signals. The separate memory I / F and the embedded memory I / F may be designed within the memory control block of the ASIC to provide a memory control structure that can be flexibly applied to both the individual memories and the embedded memories.

[0157] Additionally, an application-specific integrated circuit (ASIC) for an artificial neural network (ANN) may be configured to include a plurality of neurons arranged in an array and a plurality of synaptic circuits, each neuron including a register, a microprocessor, and at least one input, and each synaptic circuit including a memory for storing synaptic weights. Each neuron of the ASIC may be connected to at least one other neuron through one of the plurality of synaptic circuits.

[0158] Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and / or other program modules that may be executed on one or more computers and / or as a combination of hardware and software.

[0159] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0160] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0161] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0162] It is to be understood that the specific order or hierarchy of steps in the processes presented are merely examples of exemplary approaches. It is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0163] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein. < / eos> < / bos> < / pad> < / eos> < / bos>

Claims

1. A symbolic music generation system using a language model, at least one processor; and At least one memory for storing instructions or information executed by at least one processor; The operation performed by the above command or information executed by the above processor is, An act of encoding music data to generate an input token from the input digital music protocol data; An operation of performing one or more structural embeddings from the input tokens; and An operation of combining the input token embedding result and the structural embedding result to input it to a language model and generating the next output token; wherein said one or more structural embeddings are initialized before said language model begins training. A symbolic music generation system using language models.

2. In claim 1, The above input token is, N×M note-on tokens consisting of N velocities and M pitch numbers; A note-off token with speed 0; time-shift token; and Contains at least one of the special tokens for matching the beginning of a sentence, the end of a sentence, and the length of a sentence; A symbolic music generation system using language models.

3. In claim 1, One or more of the above structural embeddings, At least one of a part embedding indicating at which time part of the entire music the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token. A symbolic music generation system using language models.

4. In claim 1, One or more of the above structural embeddings, A first initialization method that randomly initializes all vectors of the one or more structural embeddings, or a second initialization method that initializes time-related embeddings among the one or more structural embeddings using sine wave encoding and applies the first initialization method to non-time-related embeddings, A symbolic music generation system using language models.

5. In claim 4, The first initialization method includes a method of initializing all vectors of the one or more structural embeddings using a truncated normal distribution. A symbolic music generation system using language models.

6. In claim 4, One or more of the above structural embeddings, The input token comprises at least one of a part embedding indicating a time part of the entire music in which the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token, The above second initialization method is, A method for preserving the continuity of sequential musical properties of notes and pitches by applying sine wave weights to the part embedding and the time embedding, and a method for initializing the type embedding and the PC embedding with a truncated normal distribution. A symbolic music generation system using language models.

7. In claim 4, The embedding input to the language model includes the input token embedding, the one or more structural embeddings, and the positional embedding initialized by the first or second initialization method. A symbolic music generation system using language models.

8. A symbolic music generation method using a language model that generates music using a large-scale language model, wherein the symbolic music generation method is executed by at least one processor, A step of encoding music data to generate an input token from input digital music protocol data; performing one or more structural embeddings from the input tokens; and A step of combining the input token embedding result and the structural embedding result to input to a language model and learning to generate the next output token; wherein said one or more structural embeddings are initialized before said language model begins training. A method for generating symbolic music using language models.

9. In claim 8, In the above encoding step, The input tokens include at least one of N×M note-on tokens consisting of N velocities and M pitch numbers; a note-off token with 0 velocities; a time-shift token; and special tokens for matching the beginning of a sentence, the end of a sentence, and the length of a sentence. A method for generating symbolic music using language models.

10. In claim 8, In the step of performing the above structural embedding, The one or more structural embeddings include, for the input token, at least one of a part embedding indicating at which time part of the entire music the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token. A method for generating symbolic music using language models.

11. In claim 8, The one or more structural embeddings initialized before the language model starts learning are: A first initialization method that randomly initializes all vectors of the one or more structural embeddings, or a second initialization method that initializes time-related embeddings among the one or more structural embeddings using sine wave encoding and applies the first initialization method to non-time-related embeddings, A symbolic music method using language models.

12. In claim 11, The first initialization method includes a method of initializing all vectors of the one or more structural embeddings using a truncated normal distribution that initializes the vectors in the form of a probability distribution in which values ​​smaller than the minimum or larger than the maximum are removed from the normal distribution. A method for generating symbolic music using language models.

13. In claim 11, The one or more structural embeddings include, for the input token, at least one of a part embedding indicating at which time part of the entire music the input token is located, a type embedding indicating that the type of the input token is one of a note-on token, a note-off token, a time-shift token, or a special token, a time embedding indicating that the input token is a time elapsed after the time-shift token, and a PC embedding indicating a pitch class (PC) of the note-on token or the note-off token, The above second initialization method is, A method for preserving the continuity of sequential musical properties of notes and pitches by applying sine wave weights to the part embedding and the time embedding, and a method for initializing the type embedding and the PC embedding with a truncated normal distribution. A method for generating symbolic music using language models.

14. In claim 11, In the above learning stage, The embedding input to the language model includes the input token embedding, the structural embedding, and the positional embedding initialized by the first or second initialization method. A method for generating symbolic music using language models.

15. A program stored in a computer-readable recording medium that causes a computer to execute the method of any one of claims 8 to 14.