Method and apparatus for generating music, and device and storage medium

Abstract

A method and apparatus for generating music, and a device and a storage medium. The method provided herein comprises: on the basis of input reference audio content, determining reference melody information; on the basis of the reference melody information and target text content, constructing an input sequence, wherein the target text content is used for describing music to be generated; providing the input sequence to a target model to generate an audio feature representation, wherein the audio feature representation comprises a plurality of audio tokens; and on the basis of the audio feature representation, generating target music matching the target text content.

Description

Method, device, equipment and storage medium for generating music

[0001] The present application claims priority to the Chinese patent application No. 202411252180.2, filed on September 6, 2024, entitled “Method, device, equipment and storage medium for generating music”, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, device, equipment and computer readable storage medium for generating music. BACKGROUND

[0003] With the development of the Internet and computer technology, generative artificial intelligence technology has developed rapidly and is widely used in various fields. As a common media form that people encounter in their daily life, music generation technology has also become the focus of attention. SUMMARY

[0004] In a first aspect of the present disclosure, a method for generating music is provided. The method comprises: determining reference melody information based on input reference audio content; constructing an input sequence based on the reference melody information and target text content, the target text content being used to describe music to be generated; providing the input sequence to a target model to generate an audio feature representation, the audio feature representation comprising a plurality of audio tokens; and generating target music matching the target text content based on the audio feature representation.

[0005] In a second aspect of the present disclosure, a device for generating music is provided. The device comprises: an information determination module configured to determine reference melody information based on input reference audio content; a sequence construction module configured to construct an input sequence based on the reference melody information and target text content, the target text content being used to describe music to be generated; a feature generation module configured to provide the input sequence to a target model to generate an audio feature representation, the audio feature representation comprising a plurality of audio tokens; and a music generation module configured to generate target music matching the target text content based on the audio feature representation.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. The instructions, when executed by the at least one processor, cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer-executable instructions that are executable by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0009] It should be understood that nothing in the Summary is to be construed as a limitation on the scope of the embodiments of the present disclosure or the claims. Other features, aspects, and advantages of the present disclosure will become apparent from the following description, which is given for the purpose of illustration, but is not intended to be limiting, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects, and advantages of embodiments of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0011] FIG. 1 shows a schematic diagram of an example environment in which embodiments according to the present disclosure can be implemented;

[0012] FIG. 2 shows a flowchart of an example process of generating music according to some embodiments of the present disclosure;

[0013] FIGS. 3A-3G show an example process of generating music according to some embodiments of the present disclosure;

[0014] FIG. 4 shows a schematic block diagram of an example music generating device according to some embodiments of the present disclosure; and

[0015] FIG. 5 shows a block diagram of an electronic device capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and should not be construed as limiting the scope of protection of the present disclosure.

[0017] It should be noted that the titles of any sections / sub-sections provided herein are not limiting. Various embodiments are described throughout this document and any type of embodiment can be included under any section / sub-section. Furthermore, embodiments described in any section / sub-section can be combined with any other embodiments described in the same section / sub-section and / or different section / sub-section in any manner.

[0018] In the description of embodiments of the disclosure, the term "includes" and its conjugates are open-ended, that is, "includes but is not limited to". The term "based on" is understood as "based at least in part on". The term "one embodiment" or "the embodiment" is understood as "at least one embodiment". The term "some embodiments" is understood as "at least some embodiments". The following can also include other explicit and implicit definitions. The terms "first", "second", etc. can refer to different or the same objects. The following can also include other explicit and implicit definitions.

[0019] Embodiments of the present disclosure can involve data of users, acquisition and / or use of data, etc. These aspects all comply with the corresponding laws and regulations and relevant provisions. In embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms. Accordingly, in the implementation of various embodiments of the present disclosure, the type of data or information that can be involved, the scope of use, the use scenario, etc. should be notified to the user and authorized by the user in a proper manner according to the relevant laws and regulations. The specific notification and / or authorization mode can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this regard.

[0020] In the present specification and embodiments, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. The user refuses to process personal information other than the necessary information required for the basic function, which does not affect the user's use of the basic function.

[0021] In the present specification and embodiments, if model training and inference are involved, the data involved (including but not limited to data itself, acquisition and / or use of data) are in compliance with the requirements of relevant laws and regulations and provisions.

[0022] According to the traditional scheme, additional artificially made humming training data is required, and the melody cannot be continued according to the user input, and it does not have the ability to additionally specify lyrics and sing, and can only generate accompaniment music. For example, some traditional schemes cannot continue the melody according to the user input, and some traditional schemes do not have the ability to additionally specify lyrics and sing. In addition, some traditional schemes can only generate accompaniment music that does not contain vocals.

[0023] Embodiments of the present disclosure propose a scheme for generating music. According to the scheme, reference melody information can be determined based on input reference audio content; an input sequence can be constructed based on the reference melody information and target text content, the target text content being used to describe music to be generated; the input sequence can be provided to a target model to generate an audio feature representation, the audio feature representation including a plurality of audio tokens; and target music matching the target text content can be generated based on the audio feature representation.

[0024] In this way, embodiments of the present disclosure can automatically extend a melody sequence according to reference audio content, and convert the melody sequence into final target music according to input text description (e.g., lyrics content). Thus, embodiments of the present disclosure can complete music creation based on a reference melody (e.g., a melody hummed by a user, or a sound / melody made by a user in other ways), for example, thereby improving the flexibility of music generation.

[0025] Various example implementations of the scheme are described in further detail below in conjunction with the accompanying drawings.

[0026] Example Environment

[0027] FIG. 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in FIG. 1, the example environment 100 can include an electronic device 110 and a target model 120.

[0028] In this example environment 100, the electronic device 110 completes a melody continuation and conversion to lyrics target music task based on invoking the target model 120. The electronic device 110 is at least configured to output received input content as corresponding target music.

[0029] In some embodiments, the electronic device 110 can establish a communication connection with the target model 120. That is, the electronic device 110 can invoke a local or remote target model 120. To obtain input content from the electronic device 110, and convert the input content into corresponding target music.

[0030] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a tablet computer, a laptop computer, a notebook computer, a netbook computer, a smartbook, a tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combinations of the aforementioned and the like, including accessories and peripherals for these devices, or any combinations thereof. In some embodiments, the electronic device 110 can also support any type of interface to a user (such as "wearable" circuitry, etc.).

[0031] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0032] Example process

[0033] FIG. 2 illustrates a flowchart of an example process 200 of generating music, in accordance with some embodiments of the present disclosure. The process 200 can be implemented at the electronic device 110. The process 200 is described below with reference to FIG. 1.

[0034] As shown in FIG. 2, at block 210, the electronic device 110 determines reference melody information based on an inputted reference audio content.

[0035] In some embodiments, the reference audio content can include any suitable audio content provided or specified by a user. In some example scenarios, the reference audio content may, for example, include a user audio captured via an audio capturing device, e.g., a piece of content hummed by a user. Alternatively, the reference audio content can also include a melodic audio content generated by a user in other ways, e.g., a sound generated by tapping an object, a sound emitted by pressing a certain button, etc.

[0036] In some embodiments, the electronic device 110 can determine reference melody information of the reference audio content. In some embodiments, the reference melody information can be expressed in a suitable form, examples of which can include, but are not limited to, a note sequence, pitch information, vocal score information, etc.

[0037] In some instances, the electronic device 110 may, for example, determine a note sequence of the reference audio content using a note extractor. Further, the electronic device 110 can determine the reference melody information based on the determined note sequence.

[0038] FIG. 3A illustrates an example process 300A of determining a note sequence. As shown in FIG. 3A, in block 310, the electronic device 110 can obtain a reference audio content input by a user. In block 311, the electronic device 110 extracts a pitch sequence of the vocal of the reference audio content by a pitch extractor (e.g., 110 Hz, 200 Hz,...). In block 312, the electronic device 110 obtains a pitch sequence corresponding to the pitch sequence of the reference audio content. In block 313, the electronic device 110 converts the obtained pitch to a corresponding note based on a pitch-to-note conversion logic. In block 314, the electronic device 110 can obtain the reference melody information, i.e., a note sequence (e.g., G, A,...). In some embodiments, the note sequence is a relative value, but can also preserve the relative melody information between different notes to some extent.

[0039] In some other embodiments, the electronic device 110 can determine the pitch information of the reference audio content by the pitch extractor. Further, the electronic device 110 can also determine the reference melody information based on the pitch information.

[0040] FIG. 3B illustrates an example process 300B of determining a pitch. Referring to FIG. 3B, in block 320, the electronic device 110 can obtain a reference audio content input by a user. In block 322, the electronic device 110 determines the pitch information of the reference audio content by a pitch extractor. In block 324, the electronic device 110 obtains the pitch information of the reference audio content. In this way, the electronic device 110 can simplify the process of determining the reference melody information.

[0041] In some other embodiments, the electronic device 110 can convert the reference audio content to a score sequence. Further, the electronic device 110 can generate the reference melody information based on the score sequence.

[0042] As an example, the electronic device 110 can convert the reference audio content to a score sequence by a transposition model. Further, the electronic device 110 can encode the score sequence into score information and encode into a text string in a specific format, e.g., REMI (revamped MIDI) encoding. In this way, the electronic device 110 can write a measure of score into a text string, including BPM (Beat Per Minute), start time of a note, duration, pitch sequence, and chord, etc., to determine the reference melody information.

[0043] With reference to FIG. 2, in block 220, the electronic device 110 constructs an input sequence based on the reference melody information and target text content, which is used to describe the music to be generated.

[0044] In some embodiments, such target text content can include, for example, lyrics information of the music to be generated, style information of the music to be generated, and beat information of the music to be generated, to expand more rich musical scores.

[0045] FIG. 3C illustrates an example process 300C of determining target music. Referring to FIG. 3C, in block 330, the electronic device 110 can identify the input of the user based on the speech recognition module to obtain target text content (e.g., lyrics information of the music to be generated). In block 331, the electronic device 110 can extract reference melody information corresponding to the content hummed by the user based on the pitch-name extractor, for example, a sequence of names 332.

[0046] In some embodiments, the above-mentioned target text content can be segmented into target text sequences based on word segmentation processing or other manners.

[0047] FIG. 3D illustrates an example process 300D of processing of the target model. Referring to FIG. 3D, in block 340, the electronic device 110 receives the target text sequence (e.g., word 1, word 2, and word 3…) and the sequence of names (e.g., sequence 1, sequence 2, and sequence 3…) to construct an input sequence of the target model.

[0048] Continuing to refer to FIG. 2, in block 230, the electronic device 110 provides the input sequence to the target model to generate an audio feature representation including a plurality of audio tokens.

[0049] Referring back to FIG. 3D, in block 333, the electronic device 110 can receive the input sequence of the target model. In some embodiments, the electronic device 110 can supplement the sequence of names in the input sequence based on the target model or other generation model to obtain complete melody corresponding to the music to be generated, i.e., target melody information (hereinafter referred to as melody sequence). Further, the electronic device 110 can encode the melody sequence and the target text sequence based on the target model.

[0050] In block 334, the electronic device 110 can obtain a plurality of audio feature representations (e.g., token 1, token 2, token 3, and token 4…) corresponding to the encoding result, same as block 334 in process 300C. It can be understood that the plurality of audio tokens are encoded by the target model based on the target text sequence, and the melody sequence is encoded into a plurality of audio tokens matching the target text sequence, same as the audio feature 334 shown in process 300C.

[0051] Continuing to refer to FIG. 2, in block 240, the electronic device 110 generates target music matching the target text content based on the audio feature representation.

[0052] Referring back to FIG. 3D, in block 335, the electronic device 110 can decode the plurality of audio tokens into target music 336 that matches the target text content based on the pre-trained audio decoder, similar to blocks 335-336 in process 300C.

[0053] In this way, embodiments of the present disclosure are able to extend a melody sequence according to a reference audio content and convert the melody sequence into target music that matches target text content (e.g., lyrics content) according to an input text description, thereby enabling music content creation based on a user-provided reference melody.

[0054] In some embodiments, the target model mentioned above can include both an audio encoder-decoder and an autoregressive model. In some embodiments, the main component of such an autoregressive model is a transducer, the main function of which is to generate a token sequence, and the core idea is to train a sequence prediction model, i.e., the output of the next time step can be predicted according to the input of the previous time step, and this cycle is repeated to generate sequence data.

[0055] FIG. 3E illustrates an example process 300E of audio processing by an audio encoder and decoder in a target model. Referring to FIG. 3E, in block 350, the electronic device 110 can receive target audio. In block 352, the electronic device 110 can process the target audio based on an audio encoder. In block 354, the electronic device can obtain a set of audio tokens, i.e., a token sequence, corresponding to the target audio, such a token sequence is usually represented by integers (e.g., 1, 2, 3). In block 356, the electronic device 110 can further process the obtained token sequence based on an audio decoder to obtain generated audio.

[0056] For ease of understanding, the training process of the target model will be described below.

[0057] FIG. 3F illustrates an example process 300F of constructing a training data set for a target model. Referring to FIG. 3F, in block 360, the electronic device 110 can obtain audio samples, e.g., a large-scale audio data set. As described above, the audio samples (including but not limited to their acquisition and / or use) are all in compliance with the requirements of relevant laws, regulations and provisions. In some embodiments, such audio samples may, for example, include a set of songs of different styles, etc. In block 362, the electronic device 110 can encode the obtained audio samples based on an audio encoder to obtain corresponding training audio feature representations, e.g., a training audio feature representation of an accompaniment.

[0058] In block 364, the electronic device 110 can identify a vocal track based on a vocal-accompaniment separation model and obtain corresponding training melody information, e.g., a training note sequence, based on a note extractor.

[0059] In block 366, the electronic device 110 can identify the corresponding training text content in the audio sample based on the speech recognition model, such as the lyric information of the music to be generated, the style information of the music to be generated, and the beat information of the music to be generated, and the like.

[0060] Further, the electronic device 110 can construct a training dataset based on the training audio feature representation, the training melody information, and the training text content, to train the target model.

[0061] FIG. 3G illustrates an example process 300G of training the target model. Referring to FIG. 3G, in block 370, the electronic device 110 can construct a training sequence based on a target segment (e.g., audio encoding 0, …t-1) of the training audio feature representation, the training text content (e.g., lyric information), and the training melody information (e.g., training note sequence). In some embodiments, such a target segment corresponds to a preset time period of the audio sample. As an example, such a target segment of the training audio feature representation can be, for example, a segment from the 0th frame of the audio feature representation to the tth frame of the audio feature representation.

[0062] As an example, the electronic device 110 can concatenate the target segment of the training audio feature representation, the training melody information, and the training text content to construct a training sequence, which can be represented as [token1, token2..., G, F..., word1, word2...], for example.

[0063] In block 372, the electronic device 110 can provide the training sequence to the target model to be trained to generate a first audio feature representation corresponding to the target time period, such as generating a predicted tth frame of the audio feature representation.

[0064] In block 374, the electronic device 110 can determine a training loss based on a comparison between the first audio feature representation (e.g., the predicted tth frame of the audio feature representation) and a second audio feature representation (e.g., the true tth frame of the audio feature representation) in the training audio feature representation. In some embodiments, such a training loss can be determined based on a loss function (e.g., a cross-entropy loss function) or other manners, for example. Further, the electronic device 110 can train the target model based on the training loss.

[0065] In this way, embodiments of the present disclosure can automatically expand the melody sequence according to the reference audio content, and convert the melody sequence to the final target music according to the input text description (e.g., lyric content). Thus, embodiments of the present disclosure can complete the creation of music based on the reference melody (e.g., a melody hummed by a user, or a sound / melody created by a user in other ways), for example, thereby improving the flexibility of music generation.

[0066] Example apparatus and device

[0067] Embodiments of the present disclosure also provide a corresponding apparatus for implementing the above method or process. FIG. 4 shows a schematic structural block diagram of an example music generation apparatus 400 according to certain embodiments of the present disclosure. The apparatus 400 can be implemented as or included in the electronic device 110. Various modules / components in the apparatus 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0068] As shown in FIG. 4, the apparatus 400 includes an information determination module 410 configured to determine reference melody information based on input reference audio content; a sequence construction module 420 configured to construct an input sequence based on the reference melody information and target text content, the target text content being used to describe music to be generated; a feature generation module 430 configured to provide the input sequence to a target model to generate an audio feature representation, the audio feature representation including a plurality of audio tokens; and a music generation module 440 configured to generate target music matching the target text content based on the audio feature representation.

[0069] In some embodiments, the information determination module 410 is further configured to determine a note sequence of the reference audio content by using a note extractor; and determine the reference melody information based on the note sequence.

[0070] In some embodiments, the information determination module 410 is further configured to determine fundamental frequency pitch information of the reference audio content by using a fundamental frequency extractor; and determine the reference melody information based on the fundamental frequency pitch information.

[0071] In some embodiments, the information determination module 410 is further configured to convert the reference audio content into a score sequence; and generate the reference melody information based on the score sequence.

[0072] In some embodiments, the target model is configured to generate target melody information corresponding to the music to be generated based on the reference melody information; and generate the plurality of audio tokens based on the target text content and the target melody information.

[0073] In some embodiments, the target text content indicates at least one of: lyric information of the music to be generated, style information of the music to be generated, and beat information of the music to be generated.

[0074] In some embodiments, the target model is trained based on a training data set, the training data set being constructed by a process of: determining training audio feature representation of an audio sample; determining training melody information based on a vocal track of the audio sample; determining training text content corresponding to the audio sample; and constructing the training data set based on the training audio feature representation, the training melody information, and the training text content.

[0075] In some embodiments, the target model is trained based on a process comprising: constructing a training sequence based on target segments of training audio feature representations, training textual content, and training melody information, the target segments corresponding to a preset time period of the audio samples; providing the training sequence to the target model to generate first audio feature representations corresponding to the target time period; determining a training loss based on a comparison between the first audio feature representations and second audio feature representations in the training audio feature representations, the second audio feature representations corresponding to the target time period; and training the target model based on the training loss.

[0076] In some embodiments, the music generation module 440 is further configured to process the audio feature representations to generate target music matching the target textual content, with a pre-trained audio decoder.

[0077] In some embodiments, the target model is an autoregressive model.

[0078] The modules included in the apparatus 400 can be implemented utilizing a variety of means, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, e.g., machine executable instructions stored on a storage medium. In addition to or alternatively, some or all of the modules in the apparatus 400 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include Field- programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0079] FIG. 5 shows a block diagram of an electronic device 500 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 500 illustrated in FIG. 5 is exemplary only and should not be taken as limiting the functionality and scope of the embodiments described herein. The electronic device 500 illustrated in FIG. 5 can be used to implement the electronic device 110 of FIG. 1.

[0080] As shown in FIG. 5, the electronic device 500 is in the form of a general electronic device. Components of the electronic device 500 can include, but are not limited to, one or more processing units or processors 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processor 510 can be a real or virtual processor and is capable of performing various processes according to programs stored in the memory 520. In a multi-processor system, multiple processors perform computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0081] Electronic device 500 typically includes a plurality of computer storage media. Such media can be any available media that is accessible by electronic device 500 and includes both volatile and nonvolatile media, removable and non-removable media. Memory 520 can be volatile (such as register, cache, RAM), non-volatile (such as ROM, EEPROM, flash memory), or some combination of the two. Storage 530 can be removable or non-removable media, and can include machine-readable media, such as flash drives, magnetic disks, or any other medium that can be used to store information and / or data and that can be accessed by electronic device 500.

[0082] Electronic device 500 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5, a disk drive and a disk controller to access a removable, non-volatile magnetic disk (e.g., a "floppy disk") and a disk drive and a disk controller to access a removable, non-volatile optical disk (e.g., a CD-ROM) can be provided. In these instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. Memory 520 can include computer program product 525 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0083] Communication unit 540 enables communications with other electronic devices over a communication medium. Additionally, the functionality of the components of electronic device 500 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating with one another over a communication connection. As such, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes.

[0084] Input device 550 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. Output device 560 can be one or more output devices, such as a display, a speaker, a printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through communication unit 540, as needed, one or more devices that enable a user to interact with electronic device 500, or any devices (e.g., a network card, a modem, etc.) that enable electronic device 500 to communicate with one or more other electronic devices. Such communication can be enabled by an input / output (I / O) interface (not shown).

[0085] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0086] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0087] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0088] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0089] The computer program product of the present disclosure can have a signal including said computer program. This signal can be electronic, electromagnetic, optical, or any other suitable type of signal. Such a signal can be provided through a communication connection, such as electrical wiring, optical fiber, cellular phone infrastructure, or computer bus or network. Such a signal can also be provided as a computer readable (storage) medium, for example a compact disk, a DVD, a Blu-ray disk, a memory stick, a memory card, or a USB stick. The computer program product of the present disclosure can have a signal including said computer program. This signal can be electronic, electromagnetic, optical, or any other suitable type of signal. Such a signal can be provided through a communication connection, such as electrical wiring, optical fiber, cellular phone infrastructure, or computer bus or network. Such a signal can also be provided as a computer readable (storage) medium, for example a compact disk, a DVD, a Blu-ray disk, a memory stick, a memory card, or a USB stick.

[0090] Having described several implementations of the present disclosure, it will be clear to those of ordinary skill in the art that many modifications can be made to the described implementations without departing from the scope and spirit of the described implementations. In some instances, well-known steps have not been described in detail in order to avoid obscuring the present disclosure. No limitation is intended to the details of the described implementations, which can also include techniques other than those explicitly described to provide the features and benefits described in this disclosure. Further, each of the implementations discussed above can be used independently or in combination with other implementations discussed above.

Claims

1. A method for generating music, comprising: determining reference melody information based on input reference audio content; constructing an input sequence based on the reference melody information and target text content, the target text content being used to describe music to be generated; providing the input sequence to a target model to generate an audio feature representation, the audio feature representation comprising a plurality of audio tokens; and generating target music matching the target text content based on the audio feature representation.

2. The method of claim 1, wherein determining reference melody information based on input reference audio content comprises: determining a note sequence of the reference audio content using a note extractor; and determining the reference melody information based on the note sequence.

3. The method of any one of claims 1-2, wherein determining reference melody information based on input reference audio content comprises: determining fundamental frequency pitch information of the reference audio content using a fundamental frequency extractor; and determining the reference melody information based on the fundamental frequency pitch information.

4. The method of any one of claims 1-3, wherein determining reference melody information based on input reference audio content comprises: converting the reference audio content to a score sequence; and generating the reference melody information based on the score sequence.

5. The method of any one of claims 1-4, wherein the target model is configured to: generate target melody information corresponding to the music to be generated based on the reference melody information; and generate the plurality of audio tokens based on the target text content and the target melody information.

6. The method of any one of claims 1-5, wherein the target text content indicates at least one of: lyric information of the music to be generated, style information of the music to be generated, tempo information of the music to be generated.

7. The method of any one of claims 1-6, wherein the target model is trained based on a training data set, the training data set being constructed by: determining training audio feature representations of audio samples; determining training melody information based on vocal tracks of the audio samples; determining training text content corresponding to the audio samples; and constructing the training data set based on the training audio feature representations, the training melody information, and the training text content.

8. The method of claim 7, wherein the target model is trained based on: constructing a training sequence based on target segments of the training audio feature representations, the training text content, and the training melody information, the target segments corresponding to a preset time period of the audio samples; providing the training sequence to the target model to generate a first audio feature representation corresponding to a target time period; determining a training loss based on a comparison of the first audio feature representation and a second audio feature representation of the training audio feature representations, the second audio feature representation corresponding to the target time period; and training the target model based on the training loss. ​ ​ ​ ​ ​ 9. The method of any one of claims 1-8, wherein generating target music matching the target textual content based on the audio feature representation comprises: processing the audio feature representation with a pre-trained audio decoder to generate target music matching the target textual content.

10. The method of any one of claims 1-9, wherein the target model is an autoregressive model.

11. An apparatus for generating music, comprising: an information determination module configured to determine reference melody information based on input reference audio content; a sequence construction module configured to construct an input sequence based on the reference melody information and target textual content, the target textual content being used to describe music to be generated; a feature generation module configured to provide the input sequence to a target model to generate an audio feature representation, the audio feature representation comprising a plurality of audio tokens; and a music generation module configured to generate target music matching the target textual content based on the audio feature representation.

12. An electronic device, comprising: at least one processor; and at least one memory that is coupled to the at least one processor and stores instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, cause the electronic device to perform the method of any one of claims 1-10.

13. A computer-readable storage medium having computer-executable instructions stored therein, the computer-executable instructions being executable by a processor to implement the method of any one of claims 1-10.

14. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method of any one of claims 1-10. ​

Images