Methods, apparatus, and systems for multi-track symbolic music generation

Abstract

The disclosure relates to a computer-implemented method for audio generation. The method includes obtaining a first set of single-track audio data, wherein the single-track audio data is represented in a text-based single-track musical notation; obtaining a set of predefined character sequences, each predefined character sequence associated with a type of music elements in the single-track musical notation; tokenizing the first set of single-track audio data based on the set of predefined character sequences; and training a first audio generation model based at least in part on the tokenized first set of signal-track audio data. The disclosure further relates to corresponding apparatus, computer programs, and computer-readable storage media.

Description

D24018W001METHODS, APPARATUS, AND SYSTEMS FOR MULTI-TRACK SYMBOLIC MUSIC GENERATIONCross-Reference to Related Applications

[0001] This application claims the benefit of priority from PCT Application No. PCT / CN2024 / 114825 filed on 27 August 2024, and U.S. Provisional Application No. 63 / 858,168 filed on 5 August 2025, each of which is incorporated by reference herein in its entirety.Technical Field

[0002] The present disclosure relates to techniques of symbolic music generation and, in particular, to generative music using generative models, for example for providing immersive and interactive experience.Background

[0003] Generative music using generative models has become a focus in the field of audio generation and processing. In contrast to mono / stereo music, multi-track music consists of several single tracks representing dedicated types / categories of instruments / sounds. Since discrete elements are available, multi-track music provides much more flexibility for users to interact with, which includes for example the following benefits:1) mixing style can be customized and is replicable for different scenarios.2) arbitrary music effect (e.g., equalization EQ, dynamic range) can be applied for arbitrary track.3) Less limitation to rendering configurations. That means, it could be rendered to stereo for headphones or channel-based format (e.g., 5.1, 7.1) for speakers.

[0004] In addition, spatial metadata can be added to each track to produce object-based music, which is of interest for example in XR / Gaming use-cases as it is able to offer immersive and interactive experience. Besides, multi-track music generation can be applied to user-D24018W001 generated content creation in terms of generating accompaniment for the user recorded / generated music, which can provide assistance towards premium music content.

[0005] Music generation approaches generally fall into two categorical types: audio generation and symbolic generation. The audio generation approaches generate audible music. With the latest methods, both the melody and sophisticated performance skills are well presented in the generated audio. Among the approaches, most of them focus on generating music in a mono / stereo audio format, which is a mix containing several elements, e.g., instruments, vocal, reverberations etc. Such mono / stereo music is, however, less flexible in terms of manipulation and interaction. In contrast, symbolic generation generates score information, which is similar to the outcome of human composition. The generated score needs to be interpreted and converted to audio for replay and listening. In addition to the pure score generation that generates score only, there are approaches that can generate score as well as the corresponding performance and / or rendering information, which is referred to as performance generation.

[0006] For audio generation approaches, multiple tasks including score composition, mixing style, and performance skills are expected to be solved simultaneously, while some of these aspects are not necessarily considered as they are in contradiction to the requirement of flexibility. In contrast, symbolic generation approaches concentrate on the score composition and leave the other processes to the user for flexibility considerations.

[0007] There is thus a need for improved techniques with respect to music generation approaches which allow for enhanced flexibility by taking into account the various aspects of audio generation. In particular, there is a need for such techniques that can further improve feasibility and flexibility of existing symbolic generation approaches.Summary

[0008] For symbolic music generation, there are two commonly used music representation methods: MIDI (i.e., Musical Instrument Digital Interface) and ABC notation. MIDI is a digital communication protocol to transmit music messages among electronic music devices. It models a music content to event-based text strings as event messages, wherein the event messages should comprise information of Note-on / off, Velocity, and tempo. On the other hand, an ABC notation is a Text Markup Language (TML) for representing musical scores. The motivation of developing ABC notations is to effectively represent musical scores and make itD24018W001 comprehensive for both humans and computers. More specifically, an ABC notation is designed to express almost identical information to what pentatonic scores express. Compared to MIDI, an ABC notation can explicitly express musical components / elements, such as bars, looping, quarter notes, eighth notes, and rests, etc. Given these characteristics, the potential of music generation using an ABC notation is continuously explored.

[0009] It is therefore appreciated that music representation used for symbolic music generation, especially an ABC notation, is a form of text, and therefore is comprehensive for computer-orientated algorithms. Accordingly, machine learning techniques, such as the use of pre-trained language models which have been developed for performing natural language processing (NLP) tasks (e.g., processing of text), can also be leveraged to such music generation using an ABC notation.

[0010] The present disclosure provides computer-implemented methods for audio generation, apparatus for audio generation, computer programs, and computer-readable storage media, having the features of the respective independent claims.

[0011] One aspect of the present disclosure relates to a computer-implemented method for audio generation. The method may include obtaining a first set of single-track audio data. In particular, the single-track audio data may be represented in a text-based single-track musical notation. The method may further include obtaining a set of predefined character sequences. Notably, each predefined character sequence may be associated with a type of music elements in the single-track musical notation. Also, the method may include tokenizing the first set of single- track audio data based on the set of predefined character sequences. In addition, the method may also include training a first audio generation model based at least in part on the tokenized first set of signal-track audio data.

[0012] Configured as above, by incorporating the domain knowledge of music (i.e., as a predefined character sequence associated with a type of music elements in the single-track musical notation) into the tokenization process, the effectiveness of training an audio generation model can be improved, especially for learning long-term and recurring patterns in ABC notations. Thereby, the proposed audio generation method allows the generated audio / music overperform that based on the existing approaches, since the generated audio / music will be more expressive, and can have longer durations and more compact musical structures.D24018W001

[0013] In some embodiments, the first set of single-track audio data may be tokenized by mapping each of the set of predefined character sequences to a respective single token. Alternatively or in addition, the method may further include determining the set of predefined character sequences based on the single-track musical notation. For example, the set of predefined character sequences may be determined based on musical theory and may comprise one or more of a letter, a number, and a symbol descriptive of one or more of a note, a note length, a chord, and a rest in the single-track musical notation.

[0014] In some embodiments, the tokenization of the first set of single-track audio data may be based on Byte Pair Encoding (BPE). In such cases, the method may further include introducing the set of predefined character sequences as initial vocabulary to the BPE.

[0015] In some embodiments, the method may also include generating a second set of single-track audio data using the trained first audio generation model. More specifically, the method may additionally include detokenizing the second set of single-track audio data for generating single-track audio for output by a playback device.

[0016] In some embodiments, the text-based single-track musical notation may include an ABC notation.

[0017] In some embodiments, the first audio generation model may include a Natural Language Processing (NLP) model based on a neural network. In such cases, the training of the first audio generation model may include providing the tokenized first set of signal-track audio data as input to the NLP model; predicting, by the first audio generation model, a subsequent token from one or more known tokens in the tokenized first set of signal-track audio data; determining an error function for the first audio generation model by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of signal-track audio data; and adjusting parameters of the first audio generation model based on the determined error function.

[0018] In some embodiments, the method may further include obtaining a first set of multi-track audio data, wherein the multi-track audio data is represented in a text-based multitrack musical notation; tokenizing the first set of multi-track audio data based on the set of predefined character sequences; and training a second audio generation model based at least in part on the first audio generation model and the tokenized first set of multi-track audio data. Accordingly, the method may also include generating a second set of multi-track audio dataD24018W001 using the trained second audio generation model. More specifically, the method may additionally include detokenizing the second set of multi-track audio data for generating multi-track audio for output by a playback device.

[0019] In some embodiments, the method may further include, prior to tokenizing the first set of multi-track audio data, organizing the first set of multi-track audio data for training the second audio generation model. In particular, organizing the first set of multi-track audio data may include inserting one or more fields into the first set of multi-track audio data. Specifically, the one or more fields may indicate a track type of each audio track in the first set of multi-track audio data. Alternatively or in addition, organizing the first set of multi-track audio data may include grouping a number of bars for a plurality of audio tracks in the first set of multi-track audio data.

[0020] In some embodiments, training the second audio generation model may include finetuning the first audio generation model with the tokenized first set of multi-track audio data by retaining one or more layers of the first audio generation model for the second audio generation model and retraining other layers of the first audio generation model for the second audio generation model. Besides, training the second audio generation model may further include providing the tokenized first set of multi-track audio data as input to the second audio generation model.

[0021] In some embodiments, the method may also include obtaining a set of predefined further character sequences. Notably, each predefined further character sequence may be associated with a type of music elements in the multi-track musical notation. The method may additionally include tokenizing the first set of multi-track audio data based on the set of predefined further character sequences.

[0022] In some embodiments, the text-based multi-track musical notation may include an ABC notation.

[0023] In some embodiments, the first audio generation model and the second audio generation model may include a Natural Language Processing (NLP) model based on a neural network. In such cases, the training of the second audio generation model may include providing the tokenized first set of multi-track audio data as input to the NLP model; predicting, by the second audio generation model, a subsequent token from one or more known tokens in the tokenized first set of multi-track audio data; determining an error function for the second audioD24018W001 generation model by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of multi-track audio data; and adjusting parameters of the second audio generation model based on the determined error function.

[0024] Configured as above, by organizing the training data for the subsequent tokenization and training processes (as described above), complex musical structures contained in multi-track audio can be effectively represented, thereby improving feasibility and flexibility of the existing text-based audio generation approaches.

[0025] In view of the above, the present disclosure proposes a computationally efficient method for audio generation, especially for generating audio using a text-based musical notation (e.g., an ABC notation). It is appreciated that by introducing the musical information which is predefined in the musical notation (e.g., as predefined notes / character sequences) as initial vocabulary to the tokenizer, tokenization of the musical notation can be performed at a sub-word level, which effectively reduces the size / length of the musical notation for the subsequent processing. In other words, the musical notation will become more compact, which allows for evaluation of more long-term dependency of the musical notation. Accordingly, the subsequent machine learning model (e.g., the NLP) can more accurately learn long-term structures of the music. In addition to single-track audio / music, the proposed method according to the present disclosure also allows for effective generation of multi-track audio / music.

[0026] According to another aspect, an apparatus for audio generation is provided. The apparatus may include a processor and a memory coupled to the processor and storing instructions for the processor. The processor may be configured to perform all steps of the methods according to preceding aspects and their embodiments.

[0027] According to a further aspect, a computer program is described. The computer program may comprise executable instructions for performing the methods or method steps outlined throughout the present disclosure when executed by a computing device (e.g., processor).

[0028] According to another aspect, a computer-readable storage medium is described. The storage medium may store a computer program adapted for execution on a computing device (e.g., processor) and for performing the methods or method steps outlined throughout the present disclosure when carried out on the computing device.D24018W001

[0029] It should be noted that the methods and systems including their preferred embodiments as outlined in the present disclosure may be used stand-alone or in combination with the other methods and systems disclosed in this document. Furthermore, all aspects of the methods and systems outlined in the present disclosure may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.

[0030] It will be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, the details of the disclosed method(s) can be realized by the corresponding apparatus, and vice versa, as the skilled person will appreciate. Moreover, any of the above statements made with respect to the method(s) (and, e.g., their steps) are understood to likewise apply to the corresponding apparatus (and, e.g., their blocks, stages, units), and vice versa.Brief Description of the Drawings

[0031] The invention is explained below in an exemplary manner with reference to the accompanying drawings, whereinFig. 1 illustrates an exemplary music clip in an original ABC notation;Fig. 2 illustrates an exemplary music clip in a processed ABC notation;Fig. 3 illustrates an exemplary music clip in a tokenized single-track ABC notation after processing by the BPE tokenizer in accordance with embodiments of the present disclosure; Fig. 4 illustrates an exemplary music clip in a tokenized single-track ABC notation after processing by the proposed ABC tokenizer based on the predefined character sequences in accordance with embodiments of the present disclosure;Fig. 5(a) schematically illustrates an exemplary apparatus / system for single-track music generation in accordance with embodiments of the present disclosure;Fig. 5(b) schematically illustrates an exemplary apparatus / system for multi-track music generation in accordance with embodiments of the present disclosure;Fig. 6 illustrates an exemplary music clip in an organized multi-track ABC notation after processing by the proposed multi-track ABC organizer in accordance with embodiments of the present disclosure;Fig. 7 shows an example flowchart of a method 700 for audio generation in accordance with embodiments of the disclosure; andD24018W001Fig. 8 schematically illustrates an exemplary apparatus for implementing the proposed audio generation methods in accordance with embodiments of the disclosure.Detailed Description

[0032] In the following, example embodiments of the disclosure will be described with reference to the appended figures. Identical elements in the figures may be indicated by identical reference numbers, and repeated description thereof may be omitted.

[0033] When using an ABC notation for symbolic music generation, approaches based on the recurrent neural network (i.e., the RNN-based approaches) may be adopted for music composition in the ABC notation. Furthermore, the transformer architecture may be used for single-track melody generation. Also, a method with bar patching and control code may be used to reduce the generation time and to make the generated melody matches the desired music form.

[0034] With the transformer architecture applied to Nature Language Processing (NLP), symbolic music generation may be served as a variant of performing an NLP task. Specifically, symbolic music representation may be tokenized before being fed into the NLP model. To this end, a character-level tokenizer may be used in music generation methods based on ABC notations, which separates the individual characters of a given ABC notation. In other words, each individual character of the ABC notation may be taken as a (single) token which may then be fed into the NLP model for the model to learn the music notation and to generate corresponding audio data.

[0035] Fig. 1 illustrates an exemplary music clip in an original ABC notation. The ABC notation may contain letter notations with a-g, A-G, and z to represent the corresponding notes and rests. Additional elements may also be used to place added value on these, such as sharp, flat, raised or lowered octave, note length, key, and ornamentation. For example, letters like “A”, “B”, “c”’ etc. may stand for a corresponding note, numbers like “3”, “4”, “1 / 4” etc. may stand for corresponding note length, combination of letters like “CB”, “FD” etc. may stand for a corresponding chord, and a specific letter (and its combination with a number) like “z”, “z2”, “z4” may stand for a rest in the ABC notation. Other examples for an ABC notation may be known to the person skilled in the art. It is appreciated that this form of notation (i.e., ABCD24018W001 notation) uses ASCII (American Standard Code for Information Interchange) to imitate standard musical notation (e.g., bar lines, tempo marks, etc.), which allows for facilitating the sharing of music online and also for simplifying the audio generation process at a computer. It is further noted that such an ABC notation can also be translated into traditional music notation using one of the ABC conversion tools.

[0036] It is noted that the example as shown in Fig. 1 may be referred to a single-track musical notation (i.e., single-track ABC notation). However, the similar concept may also be applied to a multi-track musical notation (i.e., multi-track ABC notation) where various dedicated types / categories of instruments / sounds may be represented.

[0037] Fig. 2 illustrates an exemplary music clip in a processed ABC notation. More specifically, the ABC notation as shown in Fig. 2 corresponds to the result after processing the original ABC notation of Fig. 1 using a character-level tokenizer (i.e., the character-level tokenized notation). As mentioned above, by means of character-level tokenization, the individual characters of the ABC notation of Fig. 1 are separated from each other (e.g., by space) as a single token as input to a subsequent machine learning model (e.g., NLP model). In other words, the subsequent machine learning model may take the tokenized ABC notation as training input and produce output results that can then be used to adjust / train the machine learning model.

[0038] By tokenizing the original ABC notation, the (tokenized) ABC notation become more understandable to be recognized by the machine learning model for the training process. In the exemplary tokenized result shown in Fig. 2, each individual character is separated from one another by space, which results in a total length of the processed / tokenized ABC notation longer than the original ABC notation as shown in Fig. 1.

[0039] Accordingly, by separating / splitting the individual characters of the original ABC notation (e.g., by space), a resulting stream with very long length may be generated to represent the input content (e.g., a music score), depending on the size of the input content. Such a character-lever tokenizer may work well in case of a short length of the original ABC notation concerned. For content including an ABC notation of longer length, the character-level tokenizer may generate excessively long streams of input tokens which the machining learning model cannot handle. For example, such long representation of an ABC notation may make it difficult for the learning model to learn the recurring patterns in the ABC notation. This may be the caseD24018W001 of a long ABC notation when a duration of the music is long, or there are multiple instruments / sound types included in the ABC notation (i.e., a multi-track ABC notation). Besides, if the tokenization is performed at the character level, it would be difficult for the machine learning model to learn the recurring patterns in the ABC notation, such as a chord.

[0040] In order to overcome the above mentioned issues, the present disclosure proposes an improved tokenization method for processing an (original) ABC notation using a machine learning model (e.g., the NLP model). The (original) ABC notation (which is text-based) may contain e.g., a single-track musical notation (single-track ABC notation), or a multi-track musical notation (multi-track ABC notation). In an embodiment, the (original) ABC notation may be tokenized based on Byte Pair Encoding (BPE) which is able to handle both frequent and rare words in a corpus. More specifically, a set of predefined character sequences (e.g., the predefined notes) may be obtained / determined from the original ABC notation (either single- track or multi-track).

[0041] In the case of a single-track ABC notation, each predefined character sequence may be associated with a type of music elements in the single-track ABC notation. In the case of a multi-track ABC notation, each predefined character sequence may be associated with a type of music elements in the multi-track ABC notation. The music elements of an ABC notation may be referred to the letter notations (e.g., a-g, A-G, and z, etc.) to represent the corresponding notes and rests, or additional elements for placing added value to symbolize, for example, sharp, flat, raised or lowered octave, note length, key, and ornamentation, as described above.

[0042] The set of predefined character sequences may be determined based on musical theory and examples of the predefined character sequences may include a letter, a number, and a symbol describing e.g., a note, a note length, a chord, and a rest in a single-track ABC notation (e.g., the exemplary ABC notation shown in Fig. 1). For example, the predefined character sequences may include letters like “c”, “C”, “D”’ etc. that stand for a corresponding note, numbers like “2”, “1”, “1 / 4” etc. that stand for the corresponding note length, a combination of letters like “CEG”, “GBD” etc. that stands for a corresponding chord, and a specific letter (and its combination with a number) like “z”, “z2”, “z4” that stands for a rest in the single-track ABC notation. Similar examples of the predefined character sequences are also applicable to a multitrack ABC notation. It is noted that the above mentioned exemplary predefined characterD24018W001 sequences are non-limiting examples for the exemplary single-track / multi-track ABC notations as explicitly illustrated in the present disclosure, and other examples for a general ABC notation may be known to the person skilled in the art.

[0043] More specifically, the proposed ABC tokenizer according to the present disclosure introduces the predefined character sequences (the predefined notes) as initial vocabulary to the BPE tokenizer. The BPE tokenizer may figure out the common patterns in a sequence obtained from the original ABC notation, and may then use a new symbol to represent the repeated characters (e.g., the notes) (i.e., characters / notes that have appeared several times). Taking into account the predefined character sequences as initial vocabulary, the proposed ABC tokenizer may map each of the set of predefined character sequences to a respective single token using the BPE.

[0044] Fig. 3 illustrates an exemplary music clip in a tokenized single-track ABC notation after processing by the BPE tokenizer in accordance with embodiments of the present disclosure. That is, the ABC notation as shown in Fig. 3 corresponds to the result after processing the original ABC notation of Fig. 1 using the BPE tokenizer. As indicated above, the BPE tokenizer per se (without taking into account any predefined character sequences as initial vocabulary) may search for repeated characters and replace them with a corresponding new symbol. Thus, comparing the tokenized result of Fig. 3 with the original ABC notation of Fig. 1, the repeated characters “z” directly followed by “B” may be treated as a new symbol of their combination “zB” (i.e., as a single token), while other characters may still be separated from each other e.g., by space as individual tokens.

[0045] By further providing the predefined character sequences as initial vocabulary to the BPE, the original ABC notation can be tokenized in a more structured manner, which assists to enhance the tokenization efficiency of the BPE. Specifically, using the above given examples of the predefined character sequences as initial vocabulary to the BPE, the BPE will recognize, for example, the number “1 / 4” as a note length and tokenize it accordingly. Similar processes also apply to other predefined character sequences as initial vocabulary to the BPE.

[0046] Thus, the subsequent machine learning model (e.g., NLP as a language model) can understand the meaning of the respective symbol / token in the ABC notation that correspond to each of the predefined character sequences, and can distinguish between the different predefinedD24018W001 character sequences. For example, the language model may recognize that letter “C” represents a note, while number “2” rather represents a note length. The corresponding result is shown in Fig. 4 illustrating an exemplary music clip in a tokenized single-track ABC notation after processing by the proposed ABC tokenizer based on the predefined character sequences as initial vocabulary to the BPE in accordance with embodiments of the present disclosure. As shown in Fig. 4, character sequences “z4”, “z2”, “1 / 4” etc. are treated as individual tokens for training the language model to generate audio data for the given ABC notation.

[0047] Accordingly, taking the original ABC notation presented in Fig 1 as an example, the tokenized results of using the BPE only and of using the proposed ABC tokenizers (as shown in Fig. 3 and Fig. 4, respectively) reveals reduced length of representation of the ABC notation for the same content (music score). In other words, for a very long music score, the proposed ABC tokenizer allows for reducing the length of representation of the ABC notation (up to 30% of token reduction compared to the original length), which makes the representation more compact (i.e., fewer tokens are required for the same music score) compared to the representation length of the original ABC notation and the tokenized result based only on the BPE.

[0048] It is therefore appreciated that, compared to the character-level tokenizer, the proposed tokenization approach taking into account the predefined character sequences as initial vocabulary to the BPE improves the tokenization of an ABC notation to a sub-word level. Meanwhile, since the domain knowledge of music (which may be acquired based on music theory) is incorporated into the BPE method by means of the predefined character sequences (predefined notes), the effectiveness of the machine learning model (e.g., the NLP model) learning the patterns in ABC notations can be improved.

[0049] It is noted that the above described exemplary tokenized results as shown in Figs. 3 and 4 merely depict some possible embodiments for tokenizing the original ABC notation of Fig. 1 based on the BPE and the proposed ABC tokenizer, respectively, which shall not be regarded as limiting the scope of implementing the tokenization method according to the present document.

[0050] Fig. 5(a) schematically illustrates an exemplary apparatus / system for single-track music generation in accordance with embodiments of the present disclosure. The system 500aD24018W001 may be or may be part of a computing device, a server, or a distributed system between a computing device and a server which is suitable for generating audio data. For example, the system 500a may comprise a headset, computer, mobile phone or any other audio generation devices for generating audio. In detail, the system 500a obtains a first set of single-track audio data 501a, e.g., raw data of an (original) single-track ABC notation, and comprises a tokenizing unit 502a and a training unit 504a for processing the first set of single-track audio data 501a.

[0051] The tokenizing unit 502a may be implemented with the proposed ABC tokenizer as described above and is configured to tokenize the first set of single-track audio data 501a based on a set of predefined character sequences. The predefined character sequences may be obtained / determined based on the music elements in the single-track ABC notation as described above. The tokenizing unit 502a may then output the tokenized data 503 a which will then be provided to the training unit 504a.

[0052] The training unit 504a may include a machine learning model (e.g., a language model such as a Natural Language Processing (NLP) model based on a neural network) and is configured to train the machine learning model based at least in part on the tokenized first set of signal-track audio data 503a. Herein, the machine learning model may be referred to as a first audio generation model for generating a second set of single-track audio data. After the training process by the training unit 504a is accomplished, the trained first audio generation model 505 may be obtained and retained for processing further input audio data (in order to generate the second set of single-track audio data). It is further noted that the system 500a may be further configured to detokenize the second set of single-track audio data for generating single-track audio for output by a playback device.

[0053] Specifically, the training unit 504a may have a model structure based on a decoder-only, unidirectional Transformer, for example, which may contain a plurality of layers (e.g., 12 layers), each of which has a specific hidden size and a certain number of attention heads (e.g., a hidden size of 768 and 12 attention heads). In addition, the training of the model may be operated in an auto-regressive manner with the loss function cross entropy, for example. Other examples of implementing the machine learning model for the training unit 504a may be known to the person skilled in the art.D24018W001

[0054] In some embodiments, the first audio generation model comprises the NLP model, and the training of the first audio generation model comprises providing the tokenized first set of signal-track audio data 503a as input to the NLP model. The first audio generation model may then predict a subsequent token from one or more known tokens in the tokenized first set of signal-track audio data 503a. Subsequently, an error function for the first audio generation model may be determined by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of signal-track audio data 503a. Based on the determined error function, parameters of the first audio generation model may be adjusted / tuned accordingly, so that a pre-trained model 505 may be obtained for processing further input audio data.

[0055] Fig. 5(b) schematically illustrates an exemplary apparatus / system for multi-track music generation in accordance with embodiments of the present disclosure. Similar to the system 500a in Fig. 5(a), the system 500b may be or may be part of a computing device, a server, or a distributed system between a computing device and a server which is suitable for generating audio data. For example, the system 500b may comprise a headset, computer, mobile phone or any other audio generation devices for generating audio. The multi-track music generation system 500b may in general have components performing similar functions as the single-track music generation system 500a. Compared to the single-track music generation system 500a, the multi-track music generation system 500b may generate multi-track music represented in ABC notations, where each track corresponds to a dedicated type / category of instruments / sounds .

[0056] Specifically, the multi-track music generation according to the present disclosure may be implemented using a two-stage machine / deep learning approach. In the first stage, a single-track generation model (denoted by Model-S) is trained, as described above with respect to the training of the first audio generation model in the system 500a. The resulting Model-S will be served as the pre-trained model (similar to the pre-trained model 505 obtained in the system 500a) for the second stage (as illustrated by the system 500b). In the second stage, a multi-track ABC generation model (denoted by Model-M) 508 is obtained by finetuning the Model-S with the multi-track ABC notation data. For the sake of clarity, the same reference sign 505 is also applied to the (resulting) pre-trained Model-S for the system 500b as illustrated in Fig. 5(b).D24018W001

[0057] In detail, the system 500b obtains a first set of multi-track audio data 501b, e.g., raw data of an (original) multi-track ABC notation, and comprises a tokenizing unit 502b and a fine-tuning unit 504b for processing the first set of multi-track audio data 501b. The tokenizing unit 502b may be implemented with the proposed ABC tokenizer as described above (similar to the tokenizer 502a) and is configured to tokenize the first set of multi-track audio data based on a set of predefined further character sequences. The predefined further character sequences may be obtained / determined based on the music elements in the single-track ABC notation as described above, and / or the predefined further character sequences may be obtained / determined based on the music elements in the multi-track ABC notation using the similar concept as for the single- track ABC notation.

[0058] The tokenizing unit 502b may then output the tokenized data 503b which will then be provided to the fine-tuning unit 504b. Similar to the training unit 504a of the system 500a, the fine-tuning unit 504b may also include a machine learning model (e.g., a language model such as a Natural Language Processing (NLP) model based on a neural network) and is configured to train (or fine tune the parameters of) the machine learning model based at least in part on the first audio generation model obtained in the first stage (i.e., the Model-S 505) and the tokenized first set of multi-track audio data 503b. Herein, the machine learning model may be referred to as a second audio generation model (i.e., the Model-M 508) for generating a second set of multi-track audio data. That is, the second audio generation model 508 may be trained, or the parameters thereof may be fine-tuned based at least in part on the (pre-trained) first audio generation model 505 and the tokenized first set of multi-track audio data 503b.

[0059] After the training / fine-tuning process by the fine-tuning unit 504b is accomplished, the trained second audio generation model 508 may be obtained and retained for processing further input (multi-track) audio data. It is further noted that the system 500b may be further configured to detokenize the second set of multi-track audio data for generating multi-track audio for output by a playback device.

[0060] Specifically, the fine-tuning unit 504b may have a similar structure as the training unit 504a, which may contain a plurality of layers, each of which has a specific hidden size and a certain number of attention heads, as described above. For training the second audio generation model 508, the fine-tuning unit 504b may finetune the (pre-trained) first audio generation modelD24018W001505 with the tokenized first set of multi-track audio data 503b by retaining one or more layers of the first audio generation model 505 for the second audio generation model 508 and retraining other layers of the first audio generation model 505 for the second audio generation model 508, and providing the tokenized first set of multi-track audio data 503b as input to the second audio generation model 508 (via the fine-tuning unit 504b).

[0061] As indicated above, the fine-tuning unit 504b comprises the NLP model. In some embodiments, the training of the second audio generation model 508 comprises providing the tokenized first set of multi-track audio data 503b as input to the NLP model. The second audio generation model may then predict a subsequent token from one or more known tokens in the tokenized first set of multi-track audio data 503b. Subsequently, an error function for the second audio generation model 508 may be determined by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of multi-track audio data 503b. Based on the determined error function for the second audio generation model 508, parameters of the second audio generation model 508 may be adjusted / tuned accordingly for processing further input (multi-track) audio data. For example, for retraining one or more layers of the first audio generation model 505a as mentioned above, corresponding weight of a layer of the first audio generation model 505 which is to be retrained may be updated based on the determined error function.

[0062] It is appreciated that the proposed ABC tokenizer for tokenizing ABC notations according to the present disclosure may be applied to both stages (as shown in Figs. 5(a) and 5(b)). In particular, the proposed ABC tokenizer may define several types of notes based on musical theory, containing musical notes, note length, chords, etc., as the predefined character sequences (notes). The predefined sequences / notes are then introduced as the initial vocabulary to the BPE tokenizer. Compared to the character-level tokenizer, the proposed approach upgrades the ABC notation from the character level to a sub-word level. Furthermore, as the domain knowledge of music is incorporated into the BPE method by means of the pre-defined notes, the effectiveness is improved regarding learning long-term and recurring patterns in ABC notations. As a result, the generated music may overperform the existing approaches in terms of more expressive audio representation, longer durations, and more compact musical structures.D24018W001

[0063] In order to achieve multi-track ABC notation generation with further improved performance, a multi-track ABC organizer is proposed in the second stage for the implementation of the system 500b, to organize the training data, which introduces an additional field to indicate a type of instrument / sound for each track. In addition, the multi-track ABC notation is further organized by grouping a certain number of bars for all tracks together. By means of this approach, a complex structure of multi-track musical notations can be effectively represented.

[0064] Accordingly, the system 500b may further comprise a data organizing unit 506 configured to organize the first set of multi-track audio data 501b for training the second audio generation model 508 prior to tokenizing the first set of multi-track audio data 501b. More specifically, the data organizing unit 506 may insert one or more fields into the first set of multitrack audio data 501b which indicate a track type of each audio track in the first set of multitrack audio data 501b. Alternatively or in addition, the data organizing unit 506 may group a number of bars for a plurality of audio tracks in the first set of multi-track audio data 501b. Subsequently, the organized multi-track data 507 may be provided to the tokenizing unit 502b for further processing.

[0065] Fig. 6 illustrates an exemplary music clip in an organized multi-track ABC notation after processing by the proposed multi-track ABC organizer in accordance with embodiments of the present disclosure. As indicated above, the multi-track ABC organizer is employed in the multi-track music generation system 500b to organize the training data for fine- tuning.

[0066] As specifically shown in Fig. 6, one or more additional fields, notated by “V: num”, for example, are inserted into the (original) ABC notation to indicate a type of instrument / sound for each track. For example, the field “V: 1” and “V:2” can be used to indicate piano and bass, respectively. In addition, a fixed number of bars from all tracks are grouped by adding an additional notation, for example “%bar”. By means of organization of multi-track ABC notations, complex musical structures comprising multiple tracks can be aware for the model training (fine-tuning) process.

[0067] It is noted that the above described exemplary organized result as shown in Fig. 6 merely depicts a possible embodiment for organizing an original multi-track ABC notation basedD24018W001 on the proposed ABC organizer, which shall not be regarded as limiting the scope of implementing the methods for multi-track musical notation generation according to the present document.

[0068] Fig. 7 shows an example flowchart of a method 700 for audio generation in accordance with embodiments of the disclosure. Method 700 may be implemented in software, hardware, or combinations thereof e.g., at an audio generating device, as or as part of a computing device, a server, or a distributed system between a computing device and a server which is suitable for generating audio data. Specific implementations may include a headset, computer, mobile phone, etc., or any other audio generation devices for generating audio. The method 700 comprises processing chains formed by steps S710 through S740 for generating audio data (single-track and / or multi-track) to be further processed by other rendering stages (e.g., for output by a playback device).

[0069] The method 700 comprises step S710 of obtaining a first set of single-track audio data. In particular, the single-track audio data may be represented in a text-based single-track musical notation (e.g., ABC notation). The method further comprises step S720 of obtaining a set of predefined character sequences. Notably, each predefined character sequence may be associated with a type of music elements in the single-track musical notation. In addition, the method comprises step S730 of tokenizing the first set of single-track audio data based on the set of predefined character sequences. Also, the method further comprises step S740 of training a first audio generation model based at least in part on the tokenized first set of signal-track audio data.

[0070] It is further noted that steps S710 through S740 may be performed for each of a plurality of processing cycles of an audio generating device / apparatus / system and do not need to be performed in the order shown in Fig. 7.

[0071] While methods and processing chains have been described above, it is understood that the present disclosure likewise relates to apparatus (e.g., computer apparatus or apparatus having processing capability in general) for implementing these methods and processing chains (or techniques in general).

[0072] An example of such apparatus 800 is schematically illustrated in Fig. 8. The apparatus 800 comprises a processor 801 and a memory 802 coupled to the processor 801. TheD24018W001 memory 802 may store instructions for execution by the processor 801. The processor 801 may be adapted to implement the processing chains described throughout the disclosure and / or to perform methods (e.g., methods of generating audio data for audio rendering) described throughout the disclosure. The apparatus 800 may receive inputs (e.g., a first set of single-track audio data, a first set of multi-track audio data, etc.) and generate outputs (e.g., a second set of single-track audio data, a second set of multi-track audio data, etc.), and may be used for implementing the above described audio generation systems 500a, 500b as illustrated in Figs. 5(a) and 5(b).

[0073] Aspects of the systems described herein may be implemented in an appropriate computer-based sound processing network environment (e.g., server or cloud environment) for processing digital or digitized audio files. Portions of these systems may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

[0074] One or more of the components, blocks, processes or other functional components may be implemented through a computer program that controls execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, physical (non- transitory), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.

[0075] Specifically, it should be understood that embodiments may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components were implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this detailed description, would recognize that, in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored on non-transitory computer-readable medium)D24018W001 executable by one or more electronic processors, such as a microprocessor and / or application specific integrated circuits (“ASICs”). As such, it should be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components, may be utilized to implement the embodiments. For example, computer-implemented neural networks described herein can include one or more electronic processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various components.

[0076] While one or more implementations have been described by way of example and in terms of the specific embodiments, it is to be understood that one or more implementations are not limited to the disclosed embodiments. To the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art.Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0077] Also, it is to be understood that the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted,” “connected,” “supported,” and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings.

[0078] Various Aspects and implementations of the invention may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims.EEE1. A computer-implemented method for audio generation, comprising: obtaining a first set of single-track audio data, wherein the single-track audio data is represented in a text-based single-track musical notation; obtaining a set of predefined character sequences, each predefined character sequence associated with a type of music elements in the single-track musical notation; tokenizing the first set of single-track audio data based on the set of predefined character sequences; andD24018W001 training a first audio generation model based at least in part on the tokenized first set of signaltrack audio data.EEE2. The method according to EEE1, wherein the first set of single-track audio data is tokenized by mapping each of the set of predefined character sequences to a respective single token.EEE3. The method according to EEE1 or EEE2, further comprising determining the set of predefined character sequences based on the single-track musical notation, wherein the set of predefined character sequences is determined based on musical theory and comprises one or more of a letter, a number, and a symbol descriptive of one or more of a note, a note length, a chord, and a rest in the single-track musical notation.EEE4. The method according to any one of EEE1 to EEE3, wherein the tokenization of the first set of single-track audio data is based on Byte Pair Encoding (BPE), the method further comprising introducing the set of predefined character sequences as initial vocabulary to the BPE.EEE5. The method according to any one of EEE 1 to EEE4, further comprising generating a second set of single-track audio data using the trained first audio generation model.EEE6. The method according to EEE5, further comprising detokenizing the second set of single- track audio data for generating single-track audio for output by a playback device.EEE7. The method according to any one of EEE1 to EEE6, wherein the text-based single-track musical notation comprises an ABC notation.EEE8. The method according to any one of EEE1 to EEE7, wherein the first audio generation model comprises a Natural Language Processing (NLP) model based on a neural network, and the training of the first audio generation model comprises: providing the tokenized first set of signal-track audio data as input to the NLP model; predicting, by the first audio generation model, a subsequent token from one or more known tokens in the tokenized first set of signal-track audio data; determining an error function for the first audio generation model by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of signal-track audio data; andD24018W001 adjusting parameters of the first audio generation model based on the determined error function.EEE9. The method according to any one of EEE1 to EEE8, further comprising: obtaining a first set of multi-track audio data, wherein the multi-track audio data is represented in a text-based multi-track musical notation; tokenizing the first set of multi-track audio data based on the set of predefined character sequences; and training a second audio generation model based at least in part on the first audio generation model and the tokenized first set of multi-track audio data.EEE 10. The method according to EEE9, further comprising generating a second set of multitrack audio data using the trained second audio generation model.EEE11. The method according to EEE9 or EEE10, further comprising, prior to tokenizing the first set of multi-track audio data, organizing the first set of multi-track audio data for training the second audio generation model.EEE 12. The method according to EEE11, wherein organizing the first set of multi-track audio data comprises inserting one or more fields into the first set of multi-track audio data, the one or more fields indicating a track type of each audio track in the first set of multi-track audio data.EEE 13. The method according to EEE11 or EEE 12, wherein organizing the first set of multitrack audio data comprises grouping a number of bars for a plurality of audio tracks in the first set of multi-track audio data.EEE 14. The method according to any one of EEE9 to EEE 13, wherein training the second audio generation model comprises finetuning the first audio generation model with the tokenized first set of multi-track audio data by retaining one or more layers of the first audio generation model for the second audio generation model and retraining other layers of the first audio generation model for the second audio generation model, and providing the tokenized first set of multi-track audio data as input to the second audio generation model.EEE15. The method according to any one of EEE9 to EEE14, further comprising: obtaining a set of predefined further character sequences, each predefined further character sequence associated with a type of music elements in the multi-track musical notation; andD24018W001 tokenizing the first set of multi-track audio data based on the set of predefined further character sequences.EEE 16. The method according to any one of EEE9 to EEE15, wherein the text-based multi-track musical notation comprises an ABC notation.EEE17. The method according to any one of EEE9 to EEE16, wherein the first audio generation model and the second audio generation model comprise a Natural Language Processing (NLP) model based on a neural network, and the training of the second audio generation model comprises: providing the tokenized first set of multi-track audio data as input to the NLP model; predicting, by the second audio generation model, a subsequent token from one or more known tokens in the tokenized first set of multi-track audio data; determining an error function for the second audio generation model by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of multi-track audio data; adjusting parameters of the second audio generation model based on the determined error function.EEE 18. The method according to EEE 10 or any method depending on EEE 10, further comprising detokenizing the second set of multi-track audio data for generating multi-track audio for output by a playback device.EEE19. An apparatus for audio generation, comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of EEE 1 to EEE 18.EEE20. A computer program comprising instructions that, when executed by a computing device, cause the computing device to perform all steps of the method according to any one of EEE1 to EEE 18.EEE21. A computer-readable storage medium storing the computer program according to EEE20.

Claims

D24018W001CLAIMS1. A computer-implemented method for audio generation, comprising: obtaining a first set of single-track audio data, wherein the single-track audio data is represented in a text-based single-track musical notation; obtaining a set of predefined character sequences, each predefined character sequence associated with a type of music elements in the single-track musical notation; tokenizing the first set of single-track audio data based on the set of predefined character sequences; and training a first audio generation model based at least in part on the tokenized first set of signal-track audio data.

2. The method according to claim 1 , wherein the first set of single-track audio data is tokenized by mapping each of the set of predefined character sequences to a respective single token.

3. The method according to claim 1 or 2, further comprising determining the set of predefined character sequences based on the single-track musical notation, wherein the set of predefined character sequences is determined based on musical theory and comprises one or more of a letter, a number, and a symbol descriptive of one or more of a note, a note length, a chord, and a rest in the single-track musical notation.

4. The method according to any one of the preceding claims, wherein the tokenization of the first set of single-track audio data is based on Byte Pair Encoding (BPE), the method further comprising introducing the set of predefined character sequences as initial vocabulary to the BPE.

5. The method according to any one of the preceding claims, further comprising generating a second set of single-track audio data using the trained first audio generation model.D24018W0016. The method according to claim 5, further comprising detokenizing the second set of single-track audio data for generating single-track audio for output by a playback device.

7. The method according to any one of the preceding claims, wherein the text-based single-track musical notation comprises an ABC notation.

8. The method according to any one of the preceding claims, wherein the first audio generation model comprises a Natural Language Processing (NLP) model based on a neural network, and the training of the first audio generation model comprises: providing the tokenized first set of signal-track audio data as input to the NLP model; predicting, by the first audio generation model, a subsequent token from one or more known tokens in the tokenized first set of signal-track audio data; determining an error function for the first audio generation model by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of signal-track audio data; and adjusting parameters of the first audio generation model based on the determined error function.

9. The method according to any one of the preceding claims, further comprising: obtaining a first set of multi-track audio data, wherein the multi-track audio data is represented in a text-based multi-track musical notation; tokenizing the first set of multi-track audio data based on the set of predefined character sequences; and training a second audio generation model based at least in part on the first audio generation model and the tokenized first set of multi-track audio data.

10. The method according to claim 9, further comprising: generating a second set of multi-track audio data using the trained second audio generation model.D24018W00111. The method according to claim 9 or 10, further comprising, prior to tokenizing the first set of multi-track audio data, organizing the first set of multi-track audio data for training the second audio generation model.

12. The method according to claim 11, wherein organizing the first set of multi-track audio data comprises inserting one or more fields into the first set of multi-track audio data, the one or more fields indicating a track type of each audio track in the first set of multi-track audio data.

13. The method according to claim 11 or 12, wherein organizing the first set of multitrack audio data comprises grouping a number of bars for a plurality of audio tracks in the first set of multi-track audio data.

14. The method according to any one of claims 9 to 13, wherein training the second audio generation model comprises finetuning the first audio generation model with the tokenized first set of multi-track audio data by retaining one or more layers of the first audio generation model for the second audio generation model and retraining other layers of the first audio generation model for the second audio generation model, and providing the tokenized first set of multi-track audio data as input to the second audio generation model.

15. The method according to any one of claims 9 to 14, further comprising: obtaining a set of predefined further character sequences, each predefined further character sequence associated with a type of music elements in the multi-track musical notation, and tokenizing the first set of multi-track audio data based on the set of predefined further character sequences.

16. The method according to any one of claims 9 to 15, wherein the text-based multitrack musical notation comprises an ABC notation.D24018W00117. The method according to any one of claims 9 to 16, wherein the first audio generation model and the second audio generation model comprise a Natural Language Processing (NLP) model based on a neural network, and the training of the second audio generation model comprises: providing the tokenized first set of multi-track audio data as input to the NLP model; predicting, by the second audio generation model, a subsequent token from one or more known tokens in the tokenized first set of multi-track audio data; determining an error function for the second audio generation model by comparing the predicted subsequent token with a corresponding known token subsequent to the one or more known tokens in the tokenized first set of multi-track audio data; adjusting parameters of the second audio generation model based on the determined error function.

18. The method according to claim 10 or any claim depending on claim 10, further comprising detokenizing the second set of multi-track audio data for generating multi-track audio for output by a playback device.

19. An apparatus for audio generation, comprising a processor and a memory coupled to the processor and storing instructions for the processor, wherein the processor is configured to perform all steps of the method according to any one of claims 1 to 18.

20. A computer program comprising instructions that, when executed by a computing device, cause the computing device to perform all steps of the method according to any one of claims 1 to 18.

21. A computer-readable storage medium storing the computer program according to claim 20.

Images