Method and apparatus for generating music, device, and storage medium

Abstract

A method and apparatus for generating music, a device, and a storage medium. The method comprises: acquiring inputted reference audio content (510); on the basis of the reference audio content, determining a first set of audio features (520); on the basis of the first set of audio features and music description information, constructing a target input sequence (530); providing the target input sequence to a music generation model, so as to generate a second set of audio features (540); and, on the basis of the second set of audio features, generating target music content (550), thereby improving the generation quality of the music content.

Description

Method, device, equipment and storage medium for generating music

[0001] The present application claims priority to the Chinese patent application No. 202411252217.1, filed on September 6, 2024, entitled “Method, device, equipment and storage medium for generating music”, the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Example embodiments of the present disclosure generally relate to the field of computer, and in particular, to a method, device, equipment and computer readable storage medium for generating music. BACKGROUND

[0003] With the rapid development of generative technology, it has good performance to generate content such as text, pictures, videos, code, etc. using models. However, the application of generative technology in the field of music generation is relatively lagging behind, and the length, quality and controllability of the generated music still need to be improved. SUMMARY

[0004] In a first aspect of the present disclosure, a method for generating music is provided. The method comprises: obtaining input reference audio content; determining a first set of audio features based on the reference audio content; constructing a target input sequence based on the first set of audio features and music description information, the music description information being used to describe the music content to be generated; providing the target input sequence to a music generation model to generate a second set of audio features; and generating target music content based on the second set of audio features.

[0005] In a second aspect of the present disclosure, a device for generating music is provided. The device comprises: an obtaining module configured to obtain input reference audio content; a determining module configured to determine a first set of audio features based on the reference audio content; a constructing module configured to construct a target input sequence based on the first set of audio features and music description information, the music description information being used to describe the music content to be generated; a providing module configured to provide the target input sequence to a music generation model to generate a second set of audio features; and a generating module configured to generate target music content based on the second set of audio features.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The medium has stored thereon a computer program which, when executed by a processor, implements the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to the first aspect of the present disclosure.

[0009] It should be understood that all statements herein made regarding the exemplary embodiments of the present disclosure are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future. It is therefore intended that the present disclosure embraces all known and future equivalents of the exemplary embodiments described herein. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, aspects, and advantages of various embodiments of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numbers represent like elements throughout. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating aspects of the present disclosure.

[0011] FIG. 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG. 2 shows a schematic diagram of an architecture for generating music, according to some embodiments of the present disclosure;

[0013] FIG. 3 shows a schematic diagram of an architecture for generating music, according to some embodiments of the present disclosure;

[0014] FIG. 4 shows a schematic diagram of an architecture for generating music, according to some embodiments of the present disclosure;

[0015] FIG. 5 shows a flowchart of a process for generating music, according to some embodiments of the present disclosure;

[0016] FIG. 6 shows a schematic structural block diagram of an apparatus for generating music, according to some embodiments of the present disclosure; and

[0017] FIG. 7 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be more thoroughly and completely understood. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and should not be construed as limiting the scope of the present disclosure.

[0019] It should be noted that the headings provided in any section / subsection herein are not limitations. Various embodiments are described throughout this document and any type of embodiment can be included under any section / subsection. Furthermore, embodiments described in any section / subsection can be combined with any other embodiment described in the same section / subsection and / or a different section / subsection in any manner.

[0020] In the description of embodiments of the disclosure, the term "includes" and its conjugates are open-ended, meaning "including but not limited to". The term "based on" is intended to mean "based, at least in part, on" which is not to be construed in an exclusive sense. The term "one embodiment" or "an embodiment" is intended to mean "at least one embodiment". The term "some embodiments" is intended to mean "at least some embodiments". Other explicitly and implicitly recited definitions can also be found below.

[0021] Data of users, acquisition and / or use of data, etc. can be involved in embodiments of the disclosure. These aspects all comply with corresponding laws and regulations and relevant provisions. In embodiments of the disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are performed on the premise that the user is aware of and confirms. Accordingly, in implementing embodiments of the disclosure, the type of data or information that can be involved, the scope of use, the use scenario, etc. should be notified to the user and the authorization of the user should be obtained according to relevant laws and regulations through appropriate means. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the disclosure is not limited in this regard.

[0022] In the description and embodiments of the present specification, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. Users refuse to process personal information other than the necessary information required for basic functions, which will not affect the user's use of basic functions.

[0023] In the description and embodiments of the present specification, if model training and inference are involved, the data involved (including but not limited to data itself, acquisition and / or use of data) complies with the requirements of relevant laws and regulations.

[0024] As used herein, the term “model” can learn the relationship between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes input and provides a corresponding output by using multiple layers of processing units. Neural network models are an example of models based on deep learning. In this document, “model” can also be referred to as “machine learning model”, “learning model”, “machine learning network” or “learning network”, which are used interchangeably herein.

[0025] A “neural network” is a machine learning network based on deep learning. Neural networks are capable of processing input and providing a corresponding output, which typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence, so that the output of a previous layer is provided as input to a subsequent layer, with the input layer receiving the input to the neural network and the output of the output layer as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes input from the previous layer.

[0026] Generally, machine learning can include three stages, namely a training stage, a testing stage and an application stage (also referred to as an inference stage). In the training stage, a given model can be trained using a large amount of training data, iteratively updating parameters until the model can obtain consistent inferences from the training data that meet the expected target. Through training, the model can be considered to have learned the relationship (also referred to as the mapping from input to output) between input and output from the training data. The parameters of the trained model are determined. In the testing stage, test input is applied to the trained model to test whether the model can provide correct output, thereby determining the performance of the model. In the application stage, the model can be used to process actual input based on the parameters obtained by training to determine the corresponding output.

[0027] As briefly discussed above, with the rapid development of generative technology, the use of models to generate content such as text, pictures, videos, code, etc. has good performance. With the application of generative technology in the field of music generation, some application products have also been produced, such as generating music from lyrics, generating music from text and melody, generating music from text and control instructions, generating music from pure text, etc. However, these traditional application products for music generation are generally slow in music generation, the generated music is short in length, poor in quality and cannot contain vocals, and the controllability needs to be improved.

[0028] Embodiments of the present disclosure provide a scheme for generating music. According to various embodiments of the present disclosure, input reference audio content is obtained; a first set of audio features is determined based on the reference audio content; a target input sequence is constructed based on the first set of audio features and music description information, the music description information being used to describe music content to be generated; the target input sequence is provided to a music generation model to generate a second set of audio features; and target music content is generated based on the second set of audio features.

[0029] In this way, embodiments of the present disclosure can perform controllable music content generation according to input information, thereby improving the quality of generated music content and improving the flexibility of the generation process.

[0030] The audio content, timbre information, lyrics content, etc. mentioned in the present disclosure are all used in the case of authorization.

[0031] Example environment

[0032] FIG. 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In the environment 100 of FIG. 1, it is desirable to train and use a music generation model 130 configured for multiple application environments. For example, the music generation model 130 can be configured to generate music using input lyrics. Also for example, the music generation model 130 can also be configured to continue writing input partial lyrics and generate music based on the continued lyrics. Also for example, the music generation model can also be configured to generate music using input audio, video or image, etc. multi-modal information.

[0033] As shown in FIG. 1, the environment 100 includes a training sample set 110, a model training system 120, and a model application system 140. The upper part of FIG. 1 shows the process of the model training stage, and the lower part shows the process of the model application stage. Before training, the parameters of the music generation model 130 can have initial values, or can have pre-trained parameters obtained through a pre-training process.

[0034] At the model training stage, the music generation model 130 can be trained based on a training sample set 110 including a plurality of training samples 111, and by utilizing a model training system 120. Here, each training sample 111 can involve a tuple format. For example, a training sample 111 can include a model input 112 referring to a music generation task and a model output 113. The model input 112 referring to the music generation task can include, for example, a set of sample audio features and sample description information for describing sample music content, and the model output 113 can include, for example, sample music content. The training sample 111 including the model input 112 and the model output 113 can be used to train the music generation model 130. Specifically, a training process can be iteratively performed by utilizing a large number of training samples. The music generation model 130 can be trained via forward propagation and backward propagation, and parameters of the music generation model 130 can be updated and adjusted during the training process.

[0035] After the training is completed, the music generation model 130` can be obtained. At this time, the parameters of the music generation model 130` have been updated, and based on the updated parameters, the music generation model 130` can be used to implement the music generation task at the model application stage. In some embodiments, the music generation model 130 can be a language model based on a transformer architecture.

[0036] At the model application stage, the music generation model 130`(the music generation model 130` at this time has trained parameters) can be used to perform the corresponding task by the model application system 140. For example, a model input 141 referring to a music generation task can be received, and a corresponding model output 142 can be output, which can include target music content corresponding to the music generation task.

[0037] In FIG. 1, the model training system 120 and the model application system 140 can include any computing system with computing capability, such as various computing devices / systems, terminal devices, servers, etc. The terminal device can involve any type of mobile terminal, fixed terminal, or portable terminal including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. The server includes, but is not limited to, a mainframe, an edge computing node, a computing device in a cloud environment, etc.

[0038] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0039] Example Interaction

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings. FIG. 2 shows a schematic diagram of an architecture 200 for generating music, according to some embodiments of the present disclosure. For ease of discussion, the architecture 200 will be discussed with reference to the environment 100 of FIG. 1.

[0041] In embodiments of the present disclosure, the model training system 120 can obtain sample music content 201.

[0042] Here, the model training system 120 can obtain various types of musical pieces or songs as the sample music content 201 from, for example, a music database. The sample music content 201 can be formed by whole musical pieces or songs, or can be formed by musical piece segments or song segments. After obtaining the musical pieces, musical piece segments, songs, or song segments, the obtained musical pieces, musical piece segments, songs, or song segments can also be filtered to improve the quality of the sample music content 201.

[0043] Illustratively, the model training system 120 can obtain a whole song from the music data. Then, the model training system 120 can perform slicing processing on the whole song. In order to maintain integrity, the paragraph boundaries of the song can be determined as the slicing positions. Then, the song segments obtained by slicing can be filtered based on parameters such as clarity, completeness, semantics, etc., and each of the filtered song segments can be used as the sample music content 201.

[0044] In embodiments of the present disclosure, the model training system 120 can determine a set of sample audio features 202 and sample description information 203 based on the sample music content 201.

[0045] Alternatively or additionally, the model training system 120 can perform encoding processing on the sample music content 201 to obtain a set of sample audio features 202. Illustratively, as shown in FIG. 2, the model training system 120 can perform encoding processing on the sample music content 201 using an audio marker 220 to generate a set of sample audio features 202.

[0046] As to the encoding manner of the sample music content 201, in one example, the sample music content 201 can be encoded as a set of sample music tokens. The set of sample music tokens can relatively well preserve the timbre characteristics of the sample music content 201, and can accurately prompt the music generation model 130 of the timbre characteristics of the sample music content 201. In another example, the sample music content 201 can be encoded as a set of sample music embedding vectors. The set of sample music embedding vectors can relatively well preserve the melody characteristics of the sample music content 201, and can accurately prompt the music generation model of the melody characteristics of the sample music content 201. It can be understood that the above encoding manners are only exemplary, and in specific implementation, the encoding manner of the sample music content 201 can be selected and configured according to actual needs.

[0047] The sample description information 203 is used to describe the sample music content. Alternatively or additionally, the sample description information 203 can include lyrics content 204. Alternatively or additionally, the sample description information 203 can be determined based on at least one of another sample music content, sample video content, sample description text, and a set of sample labels 205.

[0048] The two sample music contents are different music contents, and different dimensions of music characteristics can be prompted to the music generation model 130 through the two sample music contents respectively. For example, the style of music can be prompted through the sample music content 201, and the timbre characteristics can be prompted through the other sample music content, so that the music generation model 130 can generate predicted music content 207 similar to the sample music content 201 in style and similar to the other sample music content in timbre.

[0049] The sample video content can be used to instruct the music generation model 130 to generate a soundtrack or a background music for the sample video content. The sample description text can be used to describe the sample music content 201 or the other sample music content, for example, the sample description text can be used to describe the style, singing form, singing method, timbre characteristics, and the like of the sample music content.

[0050] The set of sample labels 205 can include one or more labels, and each label can label the music characteristics of the sample music content 201 from different dimensions respectively. For example, the set of sample labels 205 can include labels for labeling the style, scene, timbre, and the like of music characteristics.

[0051] In some embodiments, the model training system 120 can provide the obtained sample music content 201 to the music analysis model 210, and generate the sample description information 203 by utilizing the music analysis model 210. For example, the music analysis model 210 can be utilized to identify the lyrics of the sample music content, and obtain the lyrics content 204. The music analysis model 210 can also be utilized to analyze the music characteristics of the sample music content 201, such as the style, the scene, the timbre, etc., and obtain a set of sample tags 205. Of course, in actual applications, the sample description information 203 can also be obtained in other manners, for example, a set of sample tags 205 input by a user can be obtained.

[0052] In embodiments of the present disclosure, the model training system 120 constructs the training input sequence based on the set of sample music features 202 and the sample description information 203.

[0053] Alternatively or additionally, the model training system 120 can perform encoding processing on the sample description information 203, and obtain sample description features. Then, the training input sequence is constructed based on the sample music features 202 and the sample description features.

[0054] For example, the sample description information 203 can include the lyrics content 204, the set of sample tags 205, sample description text, etc. The model training system 120 can encode the information in multiple dimensions into multiple sets of sample description tokens corresponding to the information in multiple dimensions respectively. The model training system 120 can also encode the multiple sets of sample description tokens into a set of sample description embedding vectors. Then, the training input sequence is constructed based on, for example, the set of sample music embedding vectors or the set of sample music tokens, and the set of sample description embedding vectors. The training input sequence can be used as the model input 112 of the music generation model 130.

[0055] In some embodiments, the model training system 120 can generate a plurality of lyrics tokens corresponding to the lyrics content 204, each of which indicates phoneme information and pitch information corresponding to a respective lyrics part. Then, the model training system 120 can determine a set of sample description tokens based on the plurality of lyrics tokens. For example, the model training system 120 can convert the lyrics content 204 into pronunciation information, which can include phoneme information and pitch information. The model training system 120 can encode the phoneme information and the pitch information corresponding to each lyrics part (e.g., a pronunciation unit) in the lyrics content 204 into a lyrics token respectively. Then, the plurality of lyrics tokens can be combined into a set of sample description tokens in the order of the lyrics parts in the lyrics content 204.

[0056] In some embodiments, the model training system 120 can insert, based on structural information of the lyrics content 204, a plurality of structure tokens representing the structural information into the plurality of lyrics tokens to determine the set of sample description tokens. Here, the structural information is used to indicate the section structure of the lyrics content 204. For example, the model training system 120 can insert section tokens (Section Tokens) that label the sections of a song, such as intro, verse, chorus, bridge, etc. into the plurality of lyrics tokens. It can be appreciated that the above-mentioned structure tokens are only exemplary. In actual applications, the section structure of a song can be various, and the corresponding structural information can not be the same, and the structure tokens representing the various structural information can not be the same.

[0057] In some embodiments, the model training system 120 can insert, based on singer information of the lyrics content 204, a plurality of singer tokens representing the singer information into the plurality of lyrics tokens to determine the set of sample description tokens. For example, a song can include multiple singers, and a singer token (Singer Token) can be added before a set of lyrics tokens corresponding to each line of lyrics in the lyrics content 204.

[0058] In embodiments of the present disclosure, the model training system 120 can provide the training input sequence to the music generation model 130 to generate at least one set of predicted audio features. The model training system 120 can generate at least one predicted music content 207 based on the at least one set of predicted audio features. Then, the model training system 120 can train the music generation model 130 based on the at least one predicted music content 207. Here, the at least one predicted music content 207 can include one or more predicted music contents 207.

[0059] For example, the model training system 120 can utilize the music generation model 130 to generate a set of predicted audio features 206 based on the training input sequence. The decoder 230 can be utilized to generate a predicted music content 207 based on the set of predicted audio features 206. Then, the model training system 120 can update the parameters of the music generation model 130 based on the difference between the sample music content 201 and the predicted music content 207.

[0060] In some embodiments, the model training system 120 can determine an evaluation of the at least one predicted music content 207, and train the music generation model 130 based on the evaluation of the at least one predicted music content 207. For example, the model training system 120 can utilize an evaluation model to evaluate the at least one predicted music content 207 to obtain the evaluation of the at least one predicted music content 207. Here, the evaluation can include evaluation results of qualitative evaluation and / or quantitative evaluation, such as evaluation grades, scores, rankings, etc.

[0061] In some embodiments, the evaluation of the predicted music content 207 can be determined based on at least one of the following: similarity between the predicted music content 207 and the sample music content 201, matching degree between the predicted music content 207 and the sample description information 203, and melody rationality of the predicted music content 207.

[0062] Here, the similarity between the predicted music content 207 and the sample music content can include similarity between the predicted music content 207 and the sample music content as a whole, or similarity between one or more dimensions of music features between the predicted music content 207 and the sample music content.

[0063] Here, the matching degree between the predicted music content 207 and the sample description information can include matching degree between the predicted music content 207 and any one or more of the sample description information 203, such as the lyric content 204, the set of sample tags 205, sample video content, sample description text, etc. For example, the matching degree between the style of the lyrics of the predicted music content 207 and the lyric content 204 can be determined, the matching degree between the predicted music content 207 and the sample tags of the style, scene, timbre, etc. in the set of sample tags 205 can be determined, and so on.

[0064] Here, the melody rationality of the predicted music content 207 can include matching degree between the melody of the predicted music content 207 and the melody evaluation criteria in music theory or music technology. For example, the matching degree between the melody of the predicted music content 207 and the principles of music harmony, aesthetic principles, cultural background, traditional music style melodies (such as jazz melodies, classical music melodies), etc. can be determined.

[0065] Here, the evaluation of the predicted music content 207 can also include artificial evaluation, such as feedback information of the predicted music content 207 can be obtained.

[0066] After determining the evaluation of the predicted music content 207 as described above, the model training system 120 can update the parameters of the music generation model 130 based on the evaluation of the predicted music content 207, in order to achieve the purpose of training the music generation model. In one example, the model training system 120 can update the parameters of the music generation model 130 based on the similarity between the predicted music content 207 and the sample music content 201, so as to improve the similarity between the predicted music content 207 generated by the music generation model 130 and the sample music content 201. In another example, the model training system 120 can update the parameters of the music generation model 130 based on the matching degree between the predicted music content 207 and the sample description information 203, so as to improve the matching degree between the predicted music content 207 and the sample description information 203. In yet another example, the model training system 120 can determine a comprehensive evaluation of the predicted music content 207 based on multiple evaluations, and update the parameters of the music generation model 130 based on the comprehensive evaluation, so as to improve the performance of the music generation model 130 in multiple dimensions.

[0067] It can be understood that the specific way in which the model training system 120 trains the music generation model 130 based on the predicted music content 207 also depends, to some extent, on the application manner of the music generation model 130.

[0068] In one example application scenario, the music generation model 130 can be provided with the lyrics content 204, and the music generation model 130 is used to continue writing lyrics based on the lyrics content 204, and then generate a song based on the continued lyrics. In this application scenario, the model training system 120 can determine the matching degree between the predicted music content 207 and the lyrics content 204, and update the parameters of the music generation model 130 based on the matching degree between the predicted music content 207 and the lyrics content 204, so as to improve the matching degree between the predicted music content 207 generated by the music generation model 130 and the lyrics content 204.

[0069] In another example application scenario, the music generation model 130 can be provided with the sample music content 201 and another sample music content, and the music generation model 130 is used to generate music with a timbre similar to the sample music content 201 and a melody similar to the other sample music content. In this application scenario, the model training system 120 can determine the similarity of the timbre between the predicted music content 207 and the sample music content 201, and also determine the similarity of the melody between the predicted music content 207 and the other sample music content. Then, the parameters of the music generation model 130 can be updated based on the similarity of the timbre and the similarity of the melody.

[0070] In order to fully illustrate the training process of the music generation model 130, the training process of the music generation model 130 will be exemplarily described again in combination with FIG. 3. FIG. 3 shows a schematic diagram of an architecture 300 for generating music according to some embodiments of the present disclosure. For ease of discussion, reference will be made to the environment 100 of FIG. 1.

[0071] It should be noted that the architecture 300 shown in FIG. 3 is similar to the architecture 200 shown in FIG. 2, and the difference between the architecture 300 shown in FIG. 3 and the architecture 200 shown in FIG. 2 mainly lies in the number of predicted music contents generated based on the training input sequence, and the training method of the music generation model based thereon. For the same or similar processes between the architecture 300 and the architecture 200, no further detailed introduction will be made here, and reference can be made to the foregoing description of the architecture 200. The difference between the architecture 300 and the architecture 200 will be described in detail below.

[0072] In embodiments of the present disclosure, the model training system 120 provides the music generation model 130 with a training input sequence, and the music generation model 130 is used to generate a set of predicted audio features 206 and a set of predicted audio features 306, respectively.

[0073] In embodiments of the present disclosure, the model training system 120 can utilize the decoder 230 to generate the predicted music content 207 (sometimes also referred to herein as “first predicted music content”) based on the set of predicted audio features 206, and can also utilize the decoder 330 to generate the predicted music content 307 (sometimes also referred to herein as “second predicted music content”) based on the set of predicted audio features 306. Here, the decoder 230 and the decoder 330 can be the same decoder, or can be different decoders.

[0074] In embodiments of the present disclosure, the model training system 120 can determine an evaluation of the predicted music content 207 (sometimes also referred to herein as “first evaluation”) and an evaluation of the predicted music content 307 (sometimes also referred to herein as “second evaluation”). Illustratively, the model training system 120 can utilize the evaluation model 340 to determine the evaluation of the predicted music content 207 and the evaluation of the predicted music content 307.

[0075] In embodiments of the present disclosure, the model training system 120 can compare the evaluation of the predicted music content 207 and the evaluation of the predicted music content 307. If it is determined that the evaluation of the predicted music content 207 is superior to the evaluation of the predicted music content 307, the model training system 120 can adjust the parameters of the music generation model 130 such that the probability of the generation of the predicted music content 207 is higher than the probability of the generation of the predicted music content 307.

[0076] It should be understood that the above process of training the music generation model 130 based on the evaluation of the predicted music content is merely exemplary. In actual applications, the evaluation criteria of the predicted music content can be flexibly selected according to actual needs, and the process of training the music generation model 130 based on the evaluation of the predicted music content can be correspondingly selected.

[0077] The application process (also referred to as the inference process) of the music generation model 130’ will be explained below in conjunction with FIG. 4. FIG. 4 shows a schematic diagram of an architecture 400 for generating music according to some embodiments of the present disclosure. For ease of discussion, the architecture 400 will be discussed with reference to the environment 100 of FIG. 1.

[0078] In embodiments of the present disclosure, the model application system 140 can obtain an input reference audio content 401. A user can input the reference audio content 401 to the model application system 140 using a music input device. The model application system 140 can also obtain the reference audio content 401 from a local or other electronic device. Illustratively, a song or a song segment can be obtained as the reference audio content 401. In some embodiments, the reference audio content 401 is an optional input item, which may, for example, also be omitted in the generation process.

[0079] In embodiments of the present disclosure, the model application system 140 can determine the first set of audio features 402 based on the reference audio content 401.

[0080] Alternatively or additionally, the model application system 140 can perform an encoding process on the reference audio content 401 to obtain the first set of audio features 402. Exemplarily, as shown in FIG. 4, the model application system 140 can encode the reference audio content 401 into the first set of audio features 402 by utilizing an audio marker 410.

[0081] Regarding the encoding manner of the reference audio content 401, in one example, the reference audio content 401 can be encoded into a set of reference audio tokens. The set of reference audio tokens can relatively well preserve the timbre characteristics of the reference audio content 401, and can accurately prompt the music generation model 130' of the timbre characteristics of the reference audio content 401 through the set of reference audio tokens.

[0082] In another example, the reference audio content 401 can be encoded into a set of reference audio embedding vectors. The set of reference audio embedding vectors can relatively well preserve the melody characteristics of the reference audio content 401, and can accurately prompt the music generation model of the melody characteristics of the reference audio content 401 through the set of reference audio embedding vectors. It can be understood that the above-mentioned encoding manner is only exemplary, and in specific implementation, the encoding manner of the reference audio content 401 can be selected and configured according to actual needs.

[0083] In embodiments of the present disclosure, the model application system 140 constructs a target input sequence based on the first set of audio features and the music description information.

[0084] The music description information 403 is used to describe the music content to be generated. Alternatively or additionally, the music description information 403 can include lyrics content 404. Alternatively or additionally, the music description information 403 can be determined based on at least one of the following: reference audio content 406, reference video content, reference description text, and a set of music tags 405.

[0085] The two reference audio contents 401, 406 are different audio contents, and different dimensions of music characteristics can be prompted to the music generation model 130' through the two reference audio contents 401, 406 respectively. Exemplarily, the style of the music can be prompted through the reference audio content 401 (also referred to as the “first reference audio content” in the present document), and the timbre characteristics can be prompted through the reference audio content 406 (also referred to as the “second reference audio content” in the present document), so that the music generation model 130' can generate target music content 408 similar to the reference audio content 401 in style and similar to the reference audio content 406 in timbre.

[0086] The music generation model 130` can be referred to generate a voice-over or a music score for the reference video content. The reference description text can describe the music content to be generated in natural language, for example. The music content to be generated can be described in terms of style, singing form, singing method, timbre characteristics, etc. by the reference description text, for example.

[0087] The set of music tags 405 can include one or more tags, each of which can label the music characteristics of the music content to be generated from different dimensions. For example, the set of music tags 405 can include tags for labeling the style, scene, timbre, etc. of the music characteristics.

[0088] In some embodiments, the model application system 140 can provide the reference audio content 401 to the music analysis model to generate the music description information. Illustratively, the lyrics of the reference audio content 401 can be identified by the music analysis model to obtain the lyrics content 404. The music characteristics of the reference audio content 401, such as style, scene, timbre, etc. can also be analyzed by the music analysis model to obtain the set of music tags 405.

[0089] In some embodiments, the model application system 140 can determine the music description information based on the input reference description data. Illustratively, an additional control signal can be input to the model application system 140, which indicates the music characteristics of the music content to be generated, such as style, scene, timbre, etc. The model application system 140 can determine the music description information based on the additional control signal.

[0090] In some embodiments, the model application system 140 can determine a set of description information tokens based on the music description information 403. Then, the target input sequence can be constructed based on the first set of audio features 402 and the set of description information tokens.

[0091] In some embodiments, the model application system 140 can also determine a set of description information embedding vectors based on the music description information 403. Then, the target input sequence can be constructed based on the first set of audio features 402 and the set of description information embedding vectors.

[0092] Illustratively, as shown in FIG. 4, the music description information 203 can include lyrics content 404, a set of music tags 405, reference audio content 406, reference description text, etc. The multiple dimensions of description information can be encoded into multiple sets of description information tokens by the model application system 140. The model application system 140 can also encode the multiple sets of description information tokens into a set of description information embedding vectors. Then, the target input sequence can be constructed based on the first set of audio features 402 and the set of description information embedding vectors.

[0093] In some embodiments, the model application system 140 can generate a plurality of lyric tokens corresponding to the lyric content 404, each lyric token indicating phoneme information and pitch information corresponding to a respective lyric portion. Subsequently, a set of description information tokens can be determined based on the plurality of lyric tokens. Illustratively, the model application system 140 can convert the lyric content 404 into pronunciation content, which can include phoneme information and pitch information. The model application system 140 can encode the phoneme information and the pitch information corresponding to each lyric portion (e.g., a pronunciation unit) in the lyric content 404 into a lyric token, respectively. The plurality of lyric tokens can then be combined into a set of description information tokens in the order of the lyric portions in the lyric content 404.

[0094] In some embodiments, the model application system 140 can insert a plurality of structure tokens representing structure information of the lyric content 404 into the plurality of lyric tokens based on the structure information to determine the set of description information tokens. The structure information here is used to indicate the paragraph structure of the lyric content 404. Illustratively, the model application system 140 can insert paragraph label tokens (Section Tokens) annotating the prelude, verse, chorus, bridge, etc. of the song into the plurality of lyric tokens. It can be appreciated that the structure tokens described above are merely illustrative. In actual applications, the paragraph structure of a song can be various, and the corresponding structure information can not be the same, and the structure tokens representing the various structure information can not be the same.

[0095] In some embodiments, the model application system 140 can insert a plurality of singer tokens representing singer information of the lyric content 404 into the plurality of lyric tokens based on the singer information to determine the set of description information tokens. For example, a song can include multiple singers, and a singer token can be added before a set of lyric tokens corresponding to each lyric in the lyric content 404.

[0096] In embodiments of the present disclosure, the model application system 140 provides the target input sequence to the music generation model 130' to generate the second set of audio features 407.

[0097] The way in which the music generation model 130` generates the second audio features 407 based on the target input sequence can vary depending on the type of machine learning model employed by the music generation model 130`. The following illustrates the process of the music generation model 130` generating the second audio features 407 with reference to an example. The second set of audio features 407 can include a plurality of audio features, such as audio feature 1, audio feature 2, audio feature 3, …, audio feature N-1, audio feature N, and so on. The model application system 140 provides the target input sequence to the music generation model 130`, and uses the music generation model 130` to generate the audio feature 1 based on the target input sequence. Then, the music generation model 130` is used to generate the audio feature 2 based on the audio feature 1, and the music generation model 130` is used to generate the audio feature 3 based on the audio feature 2. In the case where the second set of audio features 407 includes N audio features, the music generation model 130` needs to perform the generation process of the audio features for N times to generate the second set of audio features 407.

[0098] In an embodiment of the present disclosure, the model application system 140 generates the target music content 408 based on the second set of audio features 407. Illustratively, as shown in FIG. 4, the model application system 140 can use the decoder 420 to perform a decoding operation on the second set of audio features 407 to obtain the target music content 408.

[0099] The target music content 408 can include a piece of music, a piece of music segment, a song, or a song segment. It can be understood that the specific content of the target music content 408 depends mainly on the provided reference audio content 401 and the music description information 403. For example, the music description information 403 can include textual description information for describing the lyrics in the audio content to be generated. At this time, the music generation model 130` can be used to write the lyrics corresponding to the textual description information, and generate a song based on the written lyrics. For another example, the music description information 403 can include the lyrics content 404, and the music generation model 130` can be used to continue writing the complete lyrics based on the lyrics content 404, and generate a song based on the continued lyrics. For yet another example, the music description information 403 can include the reference audio content 406, and the target music content 408 can be generated to have a similar timbre to the reference audio content 401 and a similar melody to the reference audio content 406. For still another example, the music description information 403 can include a reference video content, and the target music content 408 can be generated to be a music piece for the reference video content.

[0100] In summary, the embodiments of the present disclosure can control the generation process of the music content according to appropriate types of input information, thereby improving the flexibility of the music generation process. Thus, the embodiments of the present disclosure can generate music content that is more in line with expectations, and improve the quality of the generated music content.

[0101] Example processes, apparatuses, and devices

[0102] FIG. 5 illustrates a flowchart of a process 500 of generating music, according to some embodiments of the present disclosure. The process 500 can be implemented in the model training system 120 or the model application system 140.

[0103] At block 510, the model application system 140 obtains input reference audio content.

[0104] At block 520, the model application system 140 determines a first set of audio features based on the reference audio content.

[0105] At block 530, the model application system 140 constructs a target input sequence based on the first set of audio features and music description information, the music description information being used to describe music content to be generated.

[0106] At block 540, the model application system 140 provides the target input sequence to a music generation model to generate a second set of audio features.

[0107] At block 540, the model application system 140 generates target music content based on the second set of audio features.

[0108] In some embodiments, the process 500 further includes: providing the reference audio content to a music analysis model to generate the music description information; or determining the music description information based on input reference description data.

[0109] In some embodiments, constructing the target input sequence based on the first set of audio features and the music description information includes: determining a set of description information tokens based on the music description information; and constructing the target input sequence based on the first set of audio features and the set of description information tokens.

[0110] In some embodiments, the music description information includes lyric content, and determining the set of description information tokens based on the music description information includes: generating a plurality of lyric tokens corresponding to the lyric content, each lyric token indicating phoneme information and pitch information corresponding to a respective lyric portion; and determining the set of description information tokens based on the plurality of lyric tokens.

[0111] In some embodiments, determining the set of description information tokens based on the plurality of lyric tokens includes: inserting a plurality of structure tokens for representing structure information of the lyric content in the plurality of lyric tokens based on the structure information to determine the set of description information tokens.

[0112] In some embodiments, determining the set of description information tokens based on the plurality of lyric tokens includes: inserting a plurality of singing tokens for representing singer information of the lyric content in the plurality of lyric tokens based on the singer information to determine the set of description information tokens.

[0113] In some embodiments, the reference audio content is a first reference audio content, and the music description information is further determined based on at least one of the following: a second reference audio content; a reference video content; a reference description text; a set of music tags.

[0114] In some embodiments, the music generation model is trained based on the following process: the model training system 120 obtains sample music content; determines a set of sample audio features and sample description information based on the sample music content, the sample description information being used to describe the sample music content; constructs a training input sequence based on the set of sample music features and the sample description information; and trains the music generation model using the training input sequence.

[0115] In some embodiments, training the music generation model using the training input sequence comprises: providing the training input sequence to the music generation model to generate at least one set of predicted audio features; generating at least one predicted music content based on the at least one set of predicted audio features; and training the music generation model based on an evaluation of the at least one predicted music content.

[0116] In some embodiments, the at least one predicted music content comprises a first predicted music content and a second predicted music content generated based on the training input sequence, and training the music generation model based on the evaluation of the at least one predicted music content comprises: in response to a first evaluation of the first predicted music content being better than a second evaluation of the second predicted music content, adjusting parameters of the music generation model such that a generation probability of the first predicted music content is higher than that of the second predicted music content.

[0117] In some embodiments, the evaluation is determined based on at least one of the following: a similarity of the predicted music content to the sample music content, a matching degree of the predicted music content to the sample description information, a melodic rationality of the predicted music content.

[0118] Embodiments of the present disclosure also provide a corresponding apparatus for implementing the above-described methods or processes. FIG. 6 shows a schematic structural block diagram of an apparatus 600 for generating music, according to some embodiments of the present disclosure. The apparatus 600 can be implemented as or included in the model training system 120 or the model application system 140. Various modules / components in the apparatus 600 can be implemented by hardware, software, firmware, or any combination thereof.

[0119] As shown in FIG. 6, the apparatus 600 includes an obtaining module 610, a determining module 620, a constructing module 630, a providing module 640, and a generating module 650. The obtaining module 610 is configured to obtain an input reference audio content. The determining module 620 is configured to determine a first set of audio features based on the reference audio content. The constructing module 630 is configured to construct a target input sequence based on the first set of audio features and music description information, the music description information being used to describe music content to be generated. The providing module 640 is configured to provide the target input sequence to a music generation model to generate a second set of audio features. The generating module 650 is configured to generate target music content based on the second set of audio features.

[0120] In some embodiments, the obtaining module 610 is further configured to: provide the reference audio content to a music analysis model to generate the music description information; or determine the music description information based on input reference description data.

[0121] In some embodiments, the constructing module 630 is further configured to: determine a set of description information tokens based on the music description information; and construct the target input sequence based on the first set of audio features and the set of description information tokens.

[0122] In some embodiments, the music description information includes lyric content, and the constructing module 630 is further configured to: generate a plurality of lyric tokens corresponding to the lyric content, each lyric token indicating phoneme information and pitch information corresponding to a respective lyric part; and determine the set of description information tokens based on the plurality of lyric tokens.

[0123] In some embodiments, the constructing module 630 is further configured to: insert a plurality of structure tokens used to represent structure information of the lyric content into the plurality of lyric tokens based on the structure information to determine the set of description information tokens.

[0124] In some embodiments, the constructing module 630 is further configured to: insert a plurality of singing tokens used to represent singer information of the lyric content into the plurality of lyric tokens based on the singer information to determine the set of description information tokens.

[0125] In some embodiments, the reference audio content is a first reference audio content, and the music description information is further determined based on at least one of: a second reference audio content; a reference video content; a reference description text; a set of music tags.

[0126] In some embodiments, the apparatus 600 can further include a training module, and the music generation model is trained by the training module based on the following process: obtaining sample music content; determining a set of sample audio features and sample description information based on the sample music content, the sample description information being used to describe the sample music content; constructing a training input sequence based on the set of sample music features and the sample description information; and training the music generation model using the training input sequence.

[0127] In some embodiments, the training module is further configured to: provide the training input sequence to the music generation model to generate at least one set of predicted audio features; generate at least one predicted music content based on the at least one set of predicted audio features; and train the music generation model based on an evaluation of the at least one predicted music content.

[0128] In some embodiments, the at least one predicted music content includes a first predicted music content and a second predicted music content generated based on the training input sequence, and the training module is further configured to: in response to a first evaluation of the first predicted music content being better than a second evaluation of the second predicted music content, adjust a parameter of the music generation model such that a generation probability of the first predicted music content is higher than the second predicted music content.

[0129] In some embodiments, the evaluation is determined based on at least one of: a similarity of the predicted music content to the sample music content, a matching degree of the predicted music content to the sample description information, a melody reasonableness of the predicted music content.

[0130] The units and / or modules included in the apparatus 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, e.g., machine executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units and / or modules in the apparatus 600 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, example types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0131] It should be understood that one or more steps in the above methods can be performed by an appropriate electronic device or combination of electronic devices. Such an electronic device or combination of electronic devices may, for example, include the model training system 120 or the model application system 140 in FIG. 1.

[0132] FIG. 7 illustrates a block diagram of an electronic device 700 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 700 illustrated in FIG. 7 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 700 illustrated in FIG. 7 can be implemented as the model training system 120 or the model application system 140 of FIG. 1, and / or the apparatus 600 of FIG. 6.

[0133] As illustrated in FIG. 7, the electronic device 700 is in the form of a general electronic device. Components of the electronic device 700 can include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 can be a real or virtual processor and is capable of executing various processing in accordance with programs stored in the memory 720. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel to improve the processing power of the electronic device 700.

[0134] The electronic device 700 typically includes a plurality of computer storage media. Such media can be any available media that is accessible by the electronic device 700 and includes both volatile and non-volatile media, removable and non-removable media. The memory 720 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 can be a removable or non-removable media and can include a machine-readable medium, such as a flash drive, a magnetic disk drive, or any other medium that can be used to store information and / or data and that can be accessed by the electronic device 700.

[0135] The electronic device 700 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7, a disk drive and a disk drive interface can be provided for reading from or writing to a removable, non- volatile magnetic disk (e.g., a "hard disk that can be used for storing software and / or data) and an optical disk drive and an optical disk drive interface can be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM or other optical medium). In these instances, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 720 can include a computer program product 725 having one or more program modules configured to carry out the various methods or actions of the various embodiments of the present disclosure.

[0136] The communication unit 740 enables communication through the communication medium with other electronic devices. Additionally, the functionality of the components of the electronic device 700 can be implemented in a single computing cluster or a plurality of computer machines capable of communicating over a communication connection. As such, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node in the networking environment.

[0137] The input device 750 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 760 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc., through the communication unit 740, as needed, with one or more devices that enable a user to interact with the electronic device 700, or with any devices (e.g., a network card, a modem, etc.) that enable the electronic device 700 to communicate with one or more other electronic devices. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0138] According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0139] Various aspects of the disclosure are now described with reference to the drawings. In general, the drawings described below are diagrammatic and schematic representations of actual or conceptual structures and processes, and are not limiting of the scope of the present disclosure. In the drawings, the same reference numerals are used to represent similar or like items.

[0140] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including a manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0141] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0142] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various implementations of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions (s). In some implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or acts or combinations of special-purpose hardware and computer instructions.

[0143] implementations. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The herein disclosed implementations are meant to be illustrative and not limiting, of the scope or spirit of the implementations. It will be apparent to those reasonably skilled in the art that varying and / or modified implementations can be apparent at ome time to some embodiments. As such, although the subject matter has been described in language specific to structural features, methodological acts, it is to be understood that the subject matter defined herein is not necessarily limited to these described examples. Any and all equivalents arising from a reasonable application of the essentials, spirit, and scope of the following claims should be covered thereby.

Claims

1. A method for generating music, comprising: obtaining input reference audio content; determining a first set of audio features based on the reference audio content; constructing a target input sequence based on the first set of audio features and music description information, the music description information being used to describe music content to be generated; providing the target input sequence to a music generation model to generate a second set of audio features; and generating target music content based on the second set of audio features.

2. The method of claim 1, further comprising: providing the reference audio content to a music analysis model to generate the music description information; or determining the music description information based on input reference description data.

3. The method of claim 1, wherein constructing the target input sequence based on the first set of audio features and the music description information comprises: determining a set of description information tokens based on the music description information; and constructing the target input sequence based on the first set of audio features and the set of description information tokens.

4. The method of claim 3, wherein the music description information comprises lyric content, and determining the set of description information tokens based on the music description information comprises: generating a plurality of lyric tokens corresponding to the lyric content, each lyric token indicating phoneme information and pitch information corresponding to a respective lyric portion; and determining the set of description information tokens based on the plurality of lyric tokens.

5. The method of claim 4, wherein determining the set of description information tokens based on the plurality of lyric tokens comprises: inserting a plurality of structure tokens in the plurality of lyric tokens based on structure information of the lyric content to determine the set of description information tokens.

6. The method of claim 4, wherein determining the set of description information tokens based on the plurality of lyric tokens comprises: inserting a plurality of singing tokens in the plurality of lyric tokens based on singer information of the lyric content to determine the set of description information tokens.

7. The method of claim 4, wherein the reference audio content is first reference audio content, and the music description information is further determined based on at least one of: second reference audio content; reference video content; reference description text; a set of music tags.

8. The method of claim 1, wherein the music generation model is trained based on the following process: obtaining sample music content; determining a set of sample audio features and sample description information based on the sample music content, the sample description information being used to describe the sample music content; constructing a training input sequence based on the set of sample music features and the sample description information; and training the music generation model using the training input sequence.

9. The method of claim 8, wherein training the music generation model using the training input sequence comprises: providing the training input sequence to the music generation model to generate at least one set of predicted audio features; generating at least one predicted music content based on the at least one set of predicted audio features; and ​ ​ ​ ​ ​ train the music generation model based on the evaluation of the at least one predicted music content. 10.The method of claim 9, wherein the at least one predicted music content comprises a first predicted music content and a second predicted music content generated based on the training input sequence, and training the music generation model based on the evaluation of the at least one predicted music content comprises: in response to a first evaluation of the first predicted music content being better than a second evaluation of the second predicted music content, adjusting parameters of the music generation model such that a generation probability of the first predicted music content is higher than the second predicted music content. 11.The method of claim 9, wherein the evaluation is determined based on at least one of the following information: a similarity of the predicted music content to the sample music content, a matching degree of the predicted music content to the sample description information, a melody reasonableness of the predicted music content. 12.An apparatus for generating music, comprising: an obtaining module configured to obtain an input reference audio content; a determining module configured to determine a first set of audio features based on the reference audio content; a constructing module configured to construct a target input sequence based on the first set of audio features and music description information, the music description information being used to describe music content to be generated; a providing module configured to provide the target input sequence to a music generation model to generate a second set of audio features; and a generating module configured to generate target music content based on the second set of audio features. 13.An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1-11. 14.A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-11. 15.A computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the method according to any one of claims 1-11. ​ ​

Images