Computing system and method for music generation

Abstract

There is provided a music generation system comprising a processor and a memory operatively coupled to the processor and storing a rhythm template database comprising a plurality of rhythm templates, and a music generation program stored in the memory and executed by the processor to be configured to receive a user input of lyrics, the method includes identifying a lyric, identifying a plurality of syllables in the lyric, determining a syllable pattern of the identified plurality of syllables, matching the syllable pattern with a selected rhythm template of a plurality of rhythm templates, generating a melody based on the selected rhythm template, generating a music file encoding the melody and the lyric, and outputting the music file encoding the melody and the lyric.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 17 / 808,975, filed on June 24, 2022, entitled “COMPUTING SYSTEM AND METHODFOR MUSIC GENERATION,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Programs have been developed that can generate music based on lyrics input by a user. However, the music generated by such programs generally lacks the musical qualities that many people appreciate, and is therefore not very song-like. For example, automatically generated music from such programs may suffer from misalignment of lyrics and melody notes, scattered or incoherent organization and song structure, mismatched rhythm tracks, and a lack of pleasant repetitive melodies. Therefore, such programs have not achieved widespread use. Therefore, there are currently obstacles to rapid song development using such programs. Summary of the invention

[0004] In view of the above, a music generation system is provided, including a processor and a memory and a music generation program, the memory being operatively coupled to the processor and storing a rhythm template database including a plurality of rhythm templates, the music generation program being stored in the memory and executed by the processor to be configured to: receive user input of lyrics, identify a plurality of syllables in the lyrics, determine a syllable pattern in the identified plurality of syllables, match the syllable pattern with a selected rhythm template from a plurality of rhythm templates, generate a melody based on the selected rhythm template, generate a music file encoding the melody and lyrics, and output the music file encoding the melody and lyrics.

[0005] This summary is provided to introduce in simplified form some concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A schematic diagram of a computing system according to an example of the present disclosure is shown.

[0007] Figure 2 Shows Figure 1 Detailed diagram of a song profile.

[0008] Figure 3 Shows Figure 1A detailed diagram of the song structure settings.

[0009] Figure 4 Shows Figure 1 Detailed diagram of the sheet music template.

[0010] Figure 5 is a flow chart of a method for generating music from lyrics input according to an example embodiment of the present disclosure.

[0011] Figure 6 is a flowchart of a method for assigning a rhythm template to a song part and generating a rhythmically repeated song structure according to an example embodiment of the present disclosure.

[0012] Figure 7 Assign the rhythm pattern to Figure 6 An illustration of a method for performing a song portion.

[0013] Figure 8 is used to rate the rhythm patterns for assigning them to Figure 6 An illustration of a method for performing a song portion.

[0014] Fig. 9 An example computing environment for the present disclosure is shown. DETAILED DESCRIPTION

[0015] In view of the above problems, a system and method for generating music based on lyrics input by a user are provided. Figure 1 , the music generation system 10 includes a music generation computing device 12, which includes a processor 14, a volatile memory 16, an input / output module 18, and a non-volatile memory 24 storing a music generation program 26, the music generation program 26 includes a song configuration file 28, the song configuration file 28 includes a plurality of rhythm templates 34, a song structure setting 38, a score template 46, a score template generator 60, and a music generator 62. As used herein, the term score refers to a music score.

[0016] The bus 20 may operatively couple the processor 14, the input / output module 18, and the volatile memory 16 to the non-volatile memory 24. Although the song configuration file 28, the song structure settings 38, the score templates 46, the score template generator 60, and the music generator 62 are depicted as being hosted (i.e., executed) on one computing device 12, it will be understood that the song configuration file 28, the song structure settings 38, the score templates 46, the score template generator 60, and the music generator 62 may alternatively be hosted across multiple computing devices to which the computing device 12 is communicatively coupled via the network 22.

[0017] As one example of such another computing device, a client computing device 64 may be provided that is operatively coupled to computing device 12. In some examples, network 22 may take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, or a combination thereof, and may include the Internet.

[0018] The computing device 12 includes a processor 14 and a non-volatile memory 24, which is configured to store a song configuration file 28, a song structure setting 38, a score template 46, a score template generator 60, and a music generator 62 in the non-volatile memory 16. The non-volatile memory 24 is a memory that retains instruction storage data even in the absence of externally applied power, such as flash memory, a hard disk, a read-only memory (ROM), an electrically erasable programmable memory (EEPROM), etc. The instructions include one or more programs (including the music generation program 26, which includes the score template generator 60, the music generator 62) and data used by such programs sufficient to perform the operations described herein. In response to execution by the processor 14, the instructions cause the processor 14 to execute the music generation program 26.

[0019] Processor 14 is a microprocessor that includes one or more of a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SOC), a field programmable gate array (FPGA), a logic circuit, or other suitable type of microprocessor configured to perform the functions described herein. System 10 also includes volatile memory 16, such as random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), etc., which temporarily stores data only when power is applied during the execution of a program.

[0020] Reference Figure 2, depicts an example of a song configuration file 28 accessed by a music generation program 26 to generate an output music file 84. The song structure 30 includes: a first song structure 30a, which includes a first solo part; a second song structure 30b, which includes a sequence of a first solo part, a first chorus part, a second solo part, and a third solo part; and a third song structure, which includes a third solo part, a second chorus part, a first solo part, and a second solo part. The song structure template 32 includes a solo part 32a and a chorus part 32b. The solo part 32a includes a first solo part 32aa that specifies a chord progression sequence defined by a numerical sequence 1-3-2-1, a second solo part 32ab that specifies a chord progression sequence defined by a numerical sequence 1-2-1-1, and a third solo part 32ac that specifies a chord progression sequence defined by a numerical sequence 1-1. The chorus part 32b includes a first chorus part 32ba, a second chorus part 32bb, a third chorus part 32bc, and a fourth chorus part 32bd. The rhythm template 34 includes a first rhythm template 34a, a second rhythm template 34b, a third rhythm template 34c, a fourth rhythm template 34d, and a fifth rhythm template 34e. The chord progression template 36 includes a first chord progression template 36a, a second chord progression template 36b, a third chord progression template 36c, a fourth chord progression template 36d, and a fifth chord progression template 36e. It should be understood that the number of rhythm templates 34, chord progression templates 36, solo parts 32a, chorus parts 32b, and song structures 30 is not necessarily limited to the number of reference numerals 34a, 34b, 34c, and 34d. Figure 2 The numbers described can be less or more than Figure 2 The number described in .

[0021] Return to reference Figure 1, when executed by the processor 14, the music generation program 26 is configured to receive a user input 82 of lyrics, identify a plurality of syllables in a set of lyrics, determine a syllable pattern 56 in the identified plurality of syllables, match a selected rhythm template from a plurality of rhythm templates 34 to the syllable pattern 56, generate a melody based on the selected rhythm template, generate a music file 84 encoding the melody and the lyrics, and output the music file 84 encoding the melody and the lyrics. Thus, each of the plurality of song parts 50a-h is matched to one of the plurality of rhythm templates. For example, the output music file 84 may be a melody sheet with lyrics, a MIDI file with chord progressions, or an audio file. For example, the output music file 84 may be output as a graphical output 72 on a display 66 of a client computing device 64, or as an audio output 80 on an audio reproduction device 78 of the client computing device 64. The audio reproduction device 78 may be, for example, a pair of speakers. The user input 82 of lyrics may be input, for example, via the user input 70 on the user interface 68 on the client computing device 64 , or via the user voice input 76 on the microphone 74 of the client computing device 64 .

[0022] The user input 82 of lyrics may include song structure settings 38 or lyrics settings 42. Figure 1 and Figure 3 , the song structure setting 38 may include a lyric setting 42, a song structure portion 40 indicating the order of the plurality of song portions 50a-h, and an instrument portion 44. The song structure portion 40 and the instrument portion 44 may be generated by the music generation program 26 or specified by the user input 82. The lyric setting 42 includes one or more song portions 42a-c, each corresponding to a list of one or more lyric segments or lyric phrases or lyric strings. Figure 3 In the example of , the song section 42a-c includes a solo song section 42a, a chorus song section 42b, and a bridge song section 42c. In the solo song section 42a, there are a first solo lyrics paragraph 42aa and a second solo lyrics paragraph 42ab, each of which represents a change, so that the first solo lyrics paragraph 42aa and the second solo lyrics paragraph 42ab have the same number of lyrics phrases. In the chorus song section 42b, there are a first chorus lyrics paragraph 42ba, a second chorus lyrics paragraph 42bb, and a third chorus lyrics paragraph 42bc, each of which represents a change, so that the first chorus lyrics paragraph 42ba, the second chorus lyrics paragraph 42bb, and the third chorus lyrics paragraph 42bc have the same number of lyrics phrases. In the bridge song section 42c, there is only one bridge lyrics paragraph 42ca.

[0023] The order of the changes can determine the actual order in which the changes appear in the final song. The number of changes can match the number of appearances of the song parts in the song structure part 40.

[0024] The instrumental section 44 specifies the instrumental music to accompany each song part within the song. In this example, the instrumental music is identified by a number representing a unique segment of the instrumental music. The song structure section 40 specifies the actual structure of the song. In this example, the song structure section 40 includes a sequence starting with a chorus section, followed by a chorus section, another chorus section, another chorus section, a bridge, and ending with a third chorus section.

[0025] Return to reference Figure 1 The score template generator 60 that populates the score template 46 based on the user's lyrics input 82 includes a lyrics parser 60a, a rhythm template selector 60b, a chord progression template selector 60c, a repetition constructor 60d, and a melody processing selector 60e. The score template 46 specifies the lyrics, all chord progressions, rhythm, and melody post-processing methods for the song. The score template 46 can be serialized as a JSON file.

[0026] Reference Figure 1 and Figure 4 , the score template 46 specifies the lyrics 50 and the song parts 50a-h within the lyrics 50, wherein the phonemes or syllables of each song part 50a-h are identified. In other words, the score template 46 specifies the rhythm template 34a-e and the chord progression template 36a-e, which are assigned to each syllable pattern 56. The syllable pattern 56 may include a solo pattern 56a-d and a chorus part pattern 56e-h corresponding to the corresponding song part 50a-h, and the corresponding song part 50a-h is also called a lyric paragraph. Other syllable patterns 56 may include, for example, a bridge, an introduction, and a solo, and may contain one or more lines of lyrics. The lyrics analyzer 60a parses the user input 82 of the lyrics to obtain phonemes. The text-to-speech service can be called to obtain normalized lyrics with phonemes grouped by syllables. Each song part 50a-h may contain one or more lyrics phrases, each lyrics phrase representing one or more lines of lyrics.

[0027] For example, when the Mandarin Chinese lyrics are processed by the lyrics parser 60a, pinyin can be used as the phoneme format to obtain phonemes. Phonemes can be grouped by syllables by identifying compound words in the lyrics. For example, in the Mandarin Chinese lyrics, the lyrics "Jīntiān tiānqìzhēnhǎo" are parsed into syllable groups "Jīntiān", "tiānqì", and "zhēn hǎo", so that there are three identified syllable groups, each with two syllables. In the English lyrics, the lyrics "How are you today" are parsed into four syllable groups "How", "are", "you", "today", so that there are three identified syllable groups each with one syllable and one identified syllable group with two syllables.

[0028] In response to parsing the lyrics, the rhythm template selector 60b is configured to assign a rhythm template to each song part of the lyrics, and the chord progression template selector 60c is configured to assign a chord progression template to each song part of the lyrics. Based on how the rhythm template selector 60b ranks the rhythm templates, the rhythm template selector 60b selects a rhythm template for each song part. Similarly, based on how the chord progression template selector 60c ranks the chord progression templates, the chord progression template selector 60c selects a chord progression template for each song part.

[0029] exist Figure 4 In the example of FIG. 5 , the first rhythm template 34 a and the first progression template 36 a are assigned to the first solo pattern 56 a corresponding to the first song portion 50 a, which includes one or more lyric phrases. Figure 4 Not shown, but a song portion may be represented by multiple rhythm templates and / or multiple chord progression templates such that each rhythm template-chord progression template pair corresponds to a lyric phrase of song portion 50a-h.

[0030] The rhythm template selector 60b can evaluate a first condition to ensure that the number of notes in the rhythm template is not equal to or less than the number of syllables in the syllable pattern of the song portion. The rhythm template selector 60b can evaluate a second condition to ensure that the minimum number of syllables supported by the rhythm template is equal to or greater than the number of syllables in the song portion. The rhythm template selector 60b can evaluate a third condition to ensure that there are no breaks in polysyllabic English words (e.g., "hello") or compound Chinese words (e.g., "nǐhǎo") when the rhythm template matches the song portion. The rhythm template selector 60b can evaluate a fourth condition to determine how many words need to be deleted or added in the song portion (e.g., add one word or delete two words) in order for the rhythm template to match the song portion.

[0031] For each song part, the rhythm template selector 60b of the music generation program 26 can rank the rhythm templates 34a-e based on how the first condition, the second condition, the third condition and / or the fourth condition are satisfied by the rhythm template for the song part to determine a rating reflecting the degree of match between the syllable pattern and each of the multiple rhythm templates 34a-e. The rhythm template selector 60b selects the rhythm template 34a-e with the best rating for each song part.

[0032] The rhythm template selector 60b can use the lyrics alignment template 54 of the score template 46 to map the syllable sequence in the lyrics to the melody and the monophonic note sequence. The lyrics matching template 54 supports the mapping of a certain range of syllables (e.g., 10 to 15 syllables). The lyrics matching template 54 matches the syllables of the lyrics to the melody and the monophonic note sequence in the corresponding rhythm template 34. The lyrics matching template 54 can be initialized manually or automatically generated. The lyrics alignment template 54 can be provided in a format that indicates how to map syllables to notes. Figure 4 In the example of , the first lyrics template 54a indicates "XXXXX", "XX-XX", "X-XX", where the 'X' letter indicates that the system maps the syllables of the lyrics to notes, and the '-' character indicates that the system does not map the syllables of the lyrics to notes. The second line "XX-XX" of the first lyrics template 54a maps the first syllable to the first note, the second syllable to the second note, the third syllable to the fourth note, and the fourth syllable to the fifth note. When the final score is finally parsed, the second syllable will cover two notes: the second note and the third note. In the second lyrics template 54b, the second line "XXX-XX" maps the first syllable to the first note, the second syllable to the second note, the third syllable to the third note, the fourth syllable to the fifth note, and the fifth syllable to the sixth note.

[0033] Reference Figure 1 , the repetition constructor 60D is configured to generate a song structure 58 based on the selected rhythm template 34a-e and the chord progression template 36a-e of the selected syllable pattern 56. The song structure 58 indicates how the song part patterns 56a-h of each song part 50a-h are serialized or sorted in the final output music file 84. The song part patterns 56a-h are sorted and serialized based on the repetition structure list 52 that specifies the allowed rhythm repetition types. For example, the 'AAAA' repetition type indicates that four solo parts with the same rhythm can be linked together, and the 'ABAB' repetition type indicates that two solo parts with two respectively different rhythms alternate in the sequence. Lyric phrases that share the same rhythm template and chord progression template can be configured to share the same pattern, so that the number of possible melody repetitions is maximized.

[0034] Reference Figure 4 In the example of , the generated song structure 58 includes a first solo pattern 56a (V1) repeated four times (according to the 'AAAA' repetition type), followed by a first chorus pattern 56e (C1), followed by a sequence of the first solo pattern 56a (V1) to the first solo pattern 56a (V1) to the second solo pattern 56b (V2) to the first solo pattern 56a (V1) (according to the 'AABA' repetition type), followed by the second chorus pattern 56f (C2), followed by the first solo pattern 56a (V1) to the second solo pattern 56b (V2) to the third solo pattern 56c (V3) The sequence of the pattern 56c (V3) (according to the 'ABCC' repetition type), followed by the third chorus pattern 56g (C3), followed by the first solo pattern 56a (V1) to the second solo pattern 56b (V2) to the third solo pattern 56c (V3) to the second solo pattern 56b (V2) (according to the 'ABCB' repetition type), followed by the fourth harmony pattern 56h (C4), followed by the sequence of the first solo pattern 56a (V1) to the second solo pattern 56b (V2) to the first solo pattern 56a (V1) to the fourth solo pattern 56d (V4) (according to the 'ABAC' repetition type). As shown below with reference to Figure 8 As described in more detail, the linking of solos can be regulated by prohibiting a prescribed list of rhythm pairs from being linked to each other. For example, if the prescribed prohibited rhythm pair list includes a second rhythm template 34b-third rhythm template 34c pair, then the third solo pattern 56c containing the third rhythm template 34c and the fourth solo pattern 56d containing the second rhythm template 34b can be prohibited from being serialized together in the generated song structure 58.

[0035] It should be understood that the post-processing methods in the melody post-processing template 48, the syllable patterns 56, the song parts 50a-h, the lyrics alignment template 54, and the number of repetition types in the repetition structure list 52 are not necessarily limited to the referenced Figure 4 The numbers described can be less or more than those in Figure 4 The number depicted in .

[0036] Reference Figure 1 , a melody processing selector 60e selects a melody post-processing step, which is performed after the song structure 58 is generated and before the music file 84 is generated. Melody post-processing can improve the melody quality. The selection of the post-processing step can be performed by a rule-based algorithm, or by the user. Figure 4For example, the score template 46 may include a melody post-processing template 48 that specifies post-processing methods implemented to process the last note in the syllable pattern 56. These post-processing methods may include a pitch-to-root process 48a that changes the pitch to the nearest root note of the current chord, a re-tune-to-chord process 48b that changes the pitch to the nearest chord note of the current chord, an extended pitch-to-root process 48c that extends the note offset to the extent it can reach before the end of the pattern and changes the pitch to the nearest root note of the current chord, an extended pitch-to-chord process 48d that extends the note offset to the extent it can reach before the end of the pattern and changes the pitch to the nearest chord note of the current chord, a drift pitch-to-root process 48e that extends the note offset to the extent it can reach before the second beat of the next pattern and changes the pitch to the nearest root note of the current chord, and / or a moderate extension process 48f that moderately extends the duration of the last note to a quarter note (1 beat) to avoid a sudden stop (which may cause duration drift). The final post-processing method may be set as a default post-processing method applied to each lyric phrase to compensate for flawed or peculiar rhythmic template designs that may cause the singing to stop abruptly.

[0037] Then, the filled score template 46 is input into the music generator 62 to generate a music file 84 encoding the melody and lyrics, applying the melody post-processing method selected by the melody processing selector 60e. The melody post-processing method can be performed by a tone generator, which can be, for example, a conditional multi-mode variation automatic encoder (MVAE) model. For each song part, the music generator 62 can select and splice all syllable patterns 56 in the score template 46 to generate, for example, a melody score with lyrics, a MIDI file with chord progressions, or an audio file. The length of the music file 84 can depend on the pattern length and the number of unique patterns for each song part. For example, if the song structure includes an 'ABAB' repetition type, the splicing pattern 'AB' instead of 'ABAB' is processed by the music generator 62 to generate pitch. In order to handle the incompleteness in the process of generating the MIDI file, the music generator 62 can fill the first chord into an extra measure at the beginning of the song without merging any pick-up measures. For example, after pitch generation, the generated music file 84 may be segmented into 4-bar or 5-bar segments.

[0038] The output music file 84 may be in a format that carries information about the melody, rhythmic pattern, etc. of the song. One example is the MIDI format, which may carry music information about pitch, start timing, stop timing, loudness (attack speed), etc. The MIDI data may be multi-track, and each track may have an instrument type associated with it, such as piano, bass guitar, strings, and drums. In this way, the melody may be encoded in one track of a multi-track MIDI file, the rhythm may be another track of the MIDI file, and so on. The MIDI file may be played by a playback program that may assign a synthesized and / or sampled electronic instrument to play each track, thereby generating an audio file of the song. In one example, the MIDI file may have a general MIDI format, so that similar sounding instruments are assigned to predetermined MIDI instrument codes.

[0039] Figure 5 A flow chart of an exemplary method 100 for generating music from lyrics input is shown. The following description of the method 100 is based on the above description and is incorporated herein by reference. Figures 1 to 4 It is understood that method 100 may also be performed in other contexts using other suitable hardware and software components.

[0040] When user input of lyrics for a song portion is received, method 100 may begin at step 102. After step 102, a song structure is selected for the song portion specified by the user input of lyrics at step 104. After step 104, at step 106, the music generation program generates the remaining song portions and selects a song structure for the remaining song portions.

[0041] Alternatively, when a user or machine inputs lyrics for all song parts, method 100 may start at step 106. At step 108, based on the input lyrics, a song structure is selected for all song parts, the lyrics are analyzed to obtain phonemes, and the phonemes are grouped by syllables.

[0042] The method 100 continues to step 110, where a rhythm template and a chord progression template are matched for each song part. At step 112, a song structure with rhythm repetition is generated. At step 114, a melody post-processing method is selected. At step 116, the populated score template is input into a music generator, and pitch generation is performed based on the rhythm and chord progression specified in the score template. At step 118, rhythm repetition is applied according to the score template. At step 120, the melody post-processing method selected in the score template is executed.

[0043] At step 122, a chord score is generated, and at step 124, an accompaniment is generated. Simultaneously with steps 122 and 124, at step 126, a melody score is generated. At step 128, the melody score is aligned with the lyrics. At step 130, a melody score with lyrics is generated. At step 132, a vocal is generated. At step 134, a mixed audio is generated based on the generated chord score and melody score.

[0044] Figure 6 A flow chart of an example method 200 for assigning a rhythm template to a song portion and generating a rhythmically repeated song structure is shown. The following description of the method 200 is based on the above description and is incorporated herein by reference. Figures 1 to 4 It is provided with the software and hardware components shown in FIG. It will be appreciated that the method 200 may also be performed in other contexts using other suitable hardware and software components.

[0045] Method 200 includes three main steps: step 202, where all rhythms that meet the requirements for each lyric phrase are ranked, step 204, where all possible rhythm sequences for each song part are searched and ranked, and step 206, where inter-part rhythm ranking is performed.

[0046] Step 202, wherein for all rhythm ratings that meet the requirements of each lyric phrase, the following steps may be included and may be performed by a rhythm template selector. In step 202a, the phonemes of the lyrics are aligned with the rhythm of the rhythm template by the lyrics matching template. In step 202b, for each lyric group, the left interval (the start interval between the start of the last lyric in the previous lyric group and the start of the first lyric in the current lyric group), the right interval (the duration of the last lyric), and all the start intervals in the current lyric group are calculated.

[0047] Reference Figure 7 , shows a process for determining the left interval and right interval for an exemplary lyric phrase. The lyric phrase "I'm literally so on cloud nine" is aligned with syllables A to I of the rhythm template. The lyric packet "literally" is aligned with syllables BCDE of the rhythm template. Here, the left interval is the duration time interval between the start of note A and the start of note B, and the right interval is the duration of note E.

[0048] Reference Figure 8, describes an example of a process for rating a rhythm template when there are multiple variations for a given lyric phrase in a song section. In this example, there are three variations A2, A3, A4 for the first lyric phrase A1 of the first solo pattern 56a, and there are three variations B2, B3, B4 for the second lyric phrase b1 of the second solo pattern 56b. In this case, lyric phrases that share the same position are used together to rate the rhythm template. Figure 8 In the example of , the first lyric phrase A1 (main phrase) is used together with the second lyric phrase B1 (main phrase), the first lyric phrase A2 (first variation) is used together with the second lyric phrase B2 (first variation), the first lyric phrase A3 (second variation) is used together with the second lyric phrase B3 (second variation), and the first lyric phrase A4 (third variation) is used together with the second lyric phrase B4 (third variation) to assign ratings to two lyric phrases in adjacent solo parts to be linked together in the rhythm sequence. Here, the lyric phrase pair A1-B1, the lyric phrase pair A2-B2, the lyric phrase pair A3-B3, and the lyric phrase pair A4-B4 are rated with respect to the first rhythm template 34a, the second rhythm template 34b, the third rhythm template 34c, the fourth rhythm template 34d, and the fifth rhythm template 34e.

[0049] return Figure 6 In step 202c, the minimum interval between the left interval and the right interval is determined. For example, if the left interval is smaller than the right interval, the left interval is determined as the minimum interval. If the right interval is smaller than the left interval, the right interval is determined as the minimum interval.

[0050] At step 202d, for each start interval greater than the minimum value, the rating is increased by start interval / minimum interval. In pseudo code, this rating increase would be represented by the following equation: rating = rating + g / m, where g is each start interval and m is the minimum interval. At step 202e, for any rest or musical bar with no notes, in each start interval, the rating is increased by the product of the rest duration and the rest weight. In pseudo code, this rating increase would be represented by the following equation: rating = rating + rest_duration * rest_weight.

[0051] Step 204, wherein all possible rhythm sequences for each song part are searched and rated, may include the following steps and may be performed by a repetition constructor. In step 204a, the types of rhythm repetitions allowed are defined. For example, for two lyric phrases, AA and AB rhythm repetitions may be allowed. For four lyric phrases, AAAA, AABA, ABAB, AABB, ABCB, ABAC, AABC and ABCC rhythm repetitions may be allowed. In step 204b, for each repetition type, an accurate rhythm sequence that matches the repetition type is searched. For example, for the rhythm repetition ABAB, the rhythm sequences [rhythm 1, rhythm 3, rhythm 1, rhythm 3] and [rhythm 1, rhythm 4, rhythm 1, rhythm 4] may be selected.

[0052] In step 204c, all rhythm sequences that are not "joinable" or not suitable for linking together in a rhythm sequence are filtered out. For example, if the next rhythm template has a defect (pickup notes) and the current rhythm template does not have enough space at the end to accommodate the defect, the two rhythms are not "joinable". In step 204d, a total rating is given for each "joinable" rhythm sequence by adding the rating of each rhythm template multiplied by the weight, where the weight is the number of repeated rhythms in this sequence. For example, for the rhythm sequence [rhythm 1, rhythm 4, rhythm 1, rhythm 4], the total rating will be (0.1+0.2+0.3+0.6)×2=2.4. The weighting mechanism makes it more likely that method 200 selects a more repeated rhythm sequence due to a lower weight. In 204e, the "joinable" rhythm sequence with the lowest rating is selected as the rhythm sequence for the song part.

[0053] Step 206, in which inter-segment rhythm rating is performed, evaluates the "jointability" between the rhythm sequences selected in step 204, and may include the following steps, which may be performed by the repetition constructor. In step 206a, all rhythm sequences that are not "jointable" between song parts are filtered out. In step 206b, for all "jointable" rhythm sequences, a total rating is given by adding up all ratings in the rhythm sequence. In step 206c, the "jointable" rhythm sequence with the lowest rating is selected as the rhythm sequence between song parts.

[0054] The above-described system and method generate lyrics-aligned melodies from lyrics, which can reduce the user's barrier to music creation. The generated melody adapts to various song structures including solo parts and chorus parts, as well as rhythm and song part repetitions. Lyrics are aligned with melodies by drawing from different rhythm templates and chord progression templates, thereby increasing the diversity of possible melodic works. By reducing the barrier to music creation, user engagement can be increased on social platforms that host music generation programs. Amateurs and professional musicians alike can benefit from using easy-to-use music creation tools to create songs from lyrics.

[0055] In some embodiments, the methods and processes described herein may be associated with a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer application or service, an application programming interface (API), a library, and / or other computer program products.

[0056] Fig. 9 A non-limiting embodiment of a computing system 300 is schematically shown that can implement one or more of the above methods and processes. The computing system 300 is shown in simplified form. The computing system 300 can respectively embody the above described and in Figure 1 The music generating computing device 12 or client computing device 64 shown in FIG. The computing system 300 can take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0057] The computing system 300 includes a logic processor 302, a volatile memory 304, and a non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or Fig. 9 Other components not shown.

[0058] Logical processor 302 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve desired results.

[0059] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The various components of the logical processor may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In this case, it will be understood that these virtualized aspects run on different physical logical processors of various different machines.

[0060] The non-volatile storage device 306 includes one or more physical devices configured to store instructions that can be executed by a logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 306 can be transformed—for example, by changing the state to change to hold different data.

[0061] The non-volatile storage device 306 may include a removable and / or built-in physical device. The non-volatile storage device 306 may include an optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.), or other mass storage device technology. The non-volatile storage device 306 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It will be appreciated that the non-volatile storage device 306 is configured to retain instructions even when power to the non-volatile storage device 306 is cut off.

[0062] The volatile memory 304 may include a physical device that includes random access memory. The volatile memory 304 is typically used by the logical processor 302 to temporarily store information during the processing of software instructions. It will be appreciated that the volatile memory 304 typically does not continue to store instructions when power to the volatile memory 304 is removed.

[0063] Aspects of the logic processor 302, volatile memory 304, and non-volatile storage device 306 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0064] The terms "module," "program," and "engine" may be used to describe an aspect of a computing system 300 that is typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves a transformation process that specifically configures the processor to perform that function. Thus, a module, program, or engine may be instantiated via a logical processor 302 executing instructions held by a non-volatile storage device 306 using portions of volatile memory 304. It will be appreciated that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, and the like. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, and the like. The terms "module," "program," and "engine" may include individual executable files, data files, libraries, drivers, scripts, database records, and the like, or groups thereof.

[0065] When included, the display subsystem 308 can be used to present a visual representation of the data held by the non-volatile storage device 306. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data held by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 308 can also be transformed to visually represent the changes in the underlying data. The display subsystem 308 can include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logical processor 302, the volatile memory 304, and / or the non-volatile storage device 306 in a shared housing, or such a display device can be a peripheral display device.

[0066] When included, the input subsystem 310 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0067] When included, the communication subsystem 312 can be configured to communicatively couple the various computing devices described herein to each other, as well as to communicatively couple to other devices. The communication subsystem 312 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network (e.g., an HDMI connected via Wi-Fi). In some embodiments, the communication subsystem can allow the computing system 300 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0068] The following paragraphs provide additional support for the claims of the present application. One aspect provides a music generation system, including a processor and a memory and a music generation program, the memory being operatively coupled to the processor and storing a rhythm template database including a plurality of rhythm templates; the music generation program being stored in the memory and executed by the processor to be configured to receive user input of lyrics; identify a plurality of syllables in the lyrics; determine a syllable pattern in the identified plurality of syllables; match the syllable pattern with a selected rhythm template in the plurality of rhythm templates; generate a melody based on the selected rhythm template; generate a music file encoding the melody and lyrics; and output a music file encoding the melody and lyrics. In this aspect, additionally or alternatively, the output music file is a melody score with lyrics, a MIDI file with chord progressions, or an audio file. In this aspect, additionally or alternatively, the lyrics include a plurality of song parts; the selected rhythm template matches the syllable pattern of one of the plurality of song parts; and each of the plurality of song parts matches one of the plurality of rhythm templates. In this aspect, additionally or alternatively, the user input of lyrics includes a song structure setting, which includes a song structure; and the melody is generated based on the song structure indicating the order of the plurality of song parts. In this aspect, additionally or alternatively, each of the plurality of song parts is matched with one of the plurality of rhythm templates according to a list of allowed rhythm repetition types. In this aspect, additionally or alternatively, each of the plurality of song parts is matched with one of the plurality of rhythm templates based on a list of rhythm pairs that are prohibited from being linked to each other. In this aspect, additionally or alternatively, in order to match the syllable pattern with the selected rhythm template, the degree of match between the syllable pattern and each of the plurality of rhythm templates is determined based on the number of syllables in the syllable pattern and the minimum number of syllables supported by each of the plurality of rhythm templates. In this aspect, additionally or alternatively, in order to match the syllable pattern with the selected rhythm template, the degree of match between the syllable pattern and each of the plurality of rhythm templates is determined based on the number of syllable groups that need to be added to or subtracted from the determined syllable pattern to match the selected rhythm pattern. In this aspect, additionally or alternatively, the memory is configured to further store a chord progression database including a plurality of chord progressions; the syllable pattern is matched with a selected chord progression from the plurality of chord progressions; and the melody is generated based on the selected rhythm template and the selected chord progression. In this aspect, additionally or alternatively, the melody is generated by performing pitch generation based on the selected rhythm template and the selected chord progression.

[0069] Another aspect provides a music generation method, comprising the following steps: receiving user input of lyrics; identifying multiple syllables in the lyrics; determining a syllable pattern in the identified multiple syllables; matching the syllable pattern with a selected rhythm template in the multiple rhythm templates; generating a melody based on the selected rhythm template; generating a music file encoding the melody and lyrics; and outputting the music file encoding the melody and lyrics. In this aspect, additionally or alternatively, the outputted music file is a melody score with lyrics, a MIDI file with chord progressions, or an audio file. In this aspect, additionally or alternatively, the lyrics include multiple song parts; the selected rhythm template matches the syllable pattern of one of the multiple song parts; and each of the multiple song parts matches one of the multiple rhythm templates. In this aspect, additionally or alternatively, the user input of the lyrics includes a song structure setting, which includes a song structure; and the melody is generated based on the song structure indicating the order of the multiple song parts. In this aspect, additionally or alternatively, each of the multiple song parts matches one of the multiple rhythm templates according to a list of allowed rhythm repetition types. In this aspect, additionally or alternatively, each of the plurality of song parts is polarity matched with one of the plurality of rhythm templates based on a list of rhythm pairs that are prohibited from being linked to each other. In this aspect, additionally or alternatively, in order to match the syllable pattern with the selected rhythm template, a degree of match between the syllable pattern and each of the plurality of rhythm templates is determined based on the number of syllable groups that need to be added to or subtracted from the determined syllable pattern to match the selected rhythm pattern. In this aspect, additionally or alternatively, the syllable pattern is matched with a selected chord progression from a plurality of chord progressions; and the melody is generated based on the selected rhythm template and the selected chord progression. In this aspect, additionally or alternatively, the melody is generated by performing pitch generation based on the selected rhythm template and the selected chord progression.

[0070] On the other hand, a music generation system is provided, including a processor and a memory, which is operatively coupled to the processor and stores a rhythm template database including multiple rhythm templates and a chord progression database including multiple chord progressions; an audio reproduction device, which is operatively coupled to the memory and the processor; and a music generation program, which is stored in the memory and executed by the processor to be configured to receive user input of lyrics; identify multiple syllables in the lyrics; determine a syllable pattern in the identified multiple syllables; match the syllable pattern with a selected rhythm template in the multiple rhythm templates; match the syllable pattern with a selected chord and chord progression in the multiple chord progressions; and generate a melody based on the selected rhythm template and the selected chord progression; generate an audio file or MIDI file encoding the melody and lyrics; and output the audio file or MIDI file encoding the melody and lyrics on the audio reproduction device.

[0071] It will be appreciated that as used herein, "and / or" refers to a logical "OR" operation, and thus A and / or B has the following truth table.

[0072] A B A and / or T T T T F T F T T F F F

[0073] It will be appreciated that the configuration and / or method described herein is exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above-mentioned processing may be changed.

[0074] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

[0075] To the extent the terms "includes," "including," "having," "containing" and variations thereof are used herein, these terms are intended to be inclusive in a manner similar to the term "comprising" as an open transitional word and do not exclude any additional or other elements.

Claims

1. A music generation system, comprising: a processor and a memory operatively coupled to the processor and storing a rhythm template database including a plurality of rhythm templates; as well as a music generation program stored in the memory and executed by the processor to be configured to: receiving user input of lyrics; Identifying a plurality of syllables in the lyrics; determining a syllable pattern among the identified plurality of syllables; matching the syllable pattern with a selected rhythm template from among the plurality of rhythm templates; generating a melody based on the selected rhythm template; generating a music file encoding the melody and the lyrics; as well as The music file in which the melody and the lyrics are encoded is output.

2. The music generating system according to claim 1, wherein the outputted music file is a melody score with lyrics, a MIDI file with chord progression, or an audio file.

3. The music generation system according to claim 1, wherein The lyrics include a plurality of song parts; The selected rhythm template is matched to the syllable pattern of one of the plurality of song parts; and Each of the plurality of song parts is matched to one of the plurality of rhythm templates.

4. The music generation system according to claim 3, wherein The user input of lyrics includes a song structure setting, the song structure setting includes a song structure; and The melody is generated based on the song structure indicating the order of the plurality of song parts.

5. The music generation system of claim 3, wherein each of the plurality of song parts is matched to one of the plurality of rhythm templates based on a list of allowed rhythm repetition types.

6. The music generating system according to claim 3, wherein each of the plurality of song parts is matched with one of the plurality of rhythm templates based on a list of rhythm pairs prohibited from being linked to each other.

7. A music generation system according to claim 1, wherein, in order to match the syllable pattern with the selected rhythm template, the degree of matching between the syllable pattern and each of the multiple rhythm templates is determined based on the number of syllables in the syllable pattern and the minimum number of syllables supported by each of the multiple rhythm templates.

8. A music generation system according to claim 1, wherein, in order to match the syllable pattern with the selected rhythm template, the degree of matching between the syllable pattern and each of the multiple rhythm templates is determined based on the number of syllable groups that need to be added to or subtracted from the determined syllable pattern to match the selected rhythm pattern.

9. The music generation system according to claim 1, wherein The memory is configured to further store a chord progression database including a plurality of chord progressions; matching the syllable pattern to a selected chord progression from among the plurality of chord progressions; and The melody is generated based on the selected rhythm template and the selected chord progression.

10. The music generating system according to claim 1, wherein the melody is generated by performing pitch generation based on the selected rhythm template and the selected chord progression.

11. A music generation method, comprising the following steps: receiving user input of lyrics; Identifying a plurality of syllables in the lyrics; determining a syllable pattern among the identified plurality of syllables; matching the syllable pattern with a selected rhythm template from among the plurality of rhythm templates; generating a melody based on the selected rhythm template; generating a music file encoding the melody and the lyrics; as well as The music file in which the melody and the lyrics are encoded is output.

12. The music generating method according to claim 11, wherein the outputted music file is a melody score with lyrics, a MIDI file with chord progression, or an audio file.

13. The music generating method according to claim 11, wherein The lyrics include a plurality of song parts; The selected rhythm template is matched to the syllable pattern of one of the plurality of song parts; and Each of the plurality of song parts is matched to one of the plurality of rhythm templates.

14. The music generating method according to claim 13, wherein The user input of lyrics includes a song structure setting, the song structure setting includes a song structure; and The melody is generated based on the song structure indicating the order of the plurality of song parts.

15. The music generation method according to claim 13, wherein each of the plurality of song parts is matched with one of the plurality of rhythm templates based on a list of allowed rhythm repetition types. 16 . The music generating method according to claim 13 , wherein each of the plurality of song parts is matched with one of the plurality of rhythm templates based on a list of rhythm pairs prohibited from being linked to each other.

17. A music generation method according to claim 11, wherein in order to match the syllable pattern with the selected rhythm template, the degree of matching between the syllable pattern and each of the multiple rhythm templates is determined based on the number of syllable groups that need to be added to or subtracted from the determined syllable pattern to match the selected rhythm pattern.

18. The music generating method according to claim 11, wherein The syllable pattern is matched to a selected chord progression among a plurality of chord progressions; and The melody is generated based on the selected rhythm template and the selected chord progression.

19. The music generating method according to claim 11, wherein the melody is generated by performing pitch generation based on the selected rhythm template and the selected chord progression.

20. A music generation system, comprising: a processor and a memory operatively coupled to the processor and storing a rhythm template database including a plurality of rhythm templates and a chord progression database including a plurality of chord progressions; an audio reproduction device operatively coupled to the memory and the processor; as well as a music generation program stored in the memory and executed by the processor to be configured to: receiving user input of lyrics; Identifying a plurality of syllables in the lyrics; determining a syllable pattern among the identified plurality of syllables; matching the syllable pattern with a selected rhythm template from among the plurality of rhythm templates; matching the syllable pattern to a selected chord progression from among the plurality of chord progressions; generating a melody based on the selected rhythm template and the selected chord progression; Generate an audio file or a MIDI file encoding the melody and the lyrics; as well as The audio file or the MIDI file encoding the melody and the lyrics is output on the audio reproducing device.

Images