Music generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a music generation method and device, equipment and a storage medium. The method comprises the following steps: acquiring input reference audio content; determining a first set of audio features based on the reference audio content; based on the first group of audio features and music description information, constructing a target input sequence, the music description information being used for describing to-be-generated music content; providing the target input sequence to a music generation model to generate a second set of audio features; and generating the target music content based on the second group of audio features. In this way, the generation quality of the music content can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for generating music. Background Technology

[0002] With the rapid development of generative technologies, the generation of content such as text, images, videos, and code using models has already shown good performance. However, the application of generative technologies in the field of music generation is relatively lagging behind, and the duration, quality, and controllability of the generated music still need improvement. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for generating music is provided. The method includes: acquiring input reference audio content; determining a first set of audio features based on the reference audio content; constructing a target input sequence based on the first set of audio features and music description information, wherein the music description information describes the music content to be generated; providing the target input sequence to a music generation model to generate a second set of audio features; and generating the target music content based on the second set of audio features.

[0004] In a second aspect of this disclosure, an apparatus for generating music is provided. The apparatus includes: an acquisition module configured to acquire input reference audio content; a determination module configured to determine a first set of audio features based on the reference audio content; a construction module configured to construct a target input sequence based on the first set of audio features and music description information, wherein the music description information describes the music content to be generated; a providing module configured to provide the target input sequence to a music generation model to generate a second set of audio features; and a generation module configured to generate the target music content based on the second set of audio features.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. A computer program is stored on the medium, which, when executed by a processor, implements the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0008] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram of an architecture for generating music according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of an architecture for generating music according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram of an architecture for generating music according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A flowchart illustrating a process for generating music according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A schematic structural block diagram of an apparatus for generating music according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device that can implement one or more embodiments of the present disclosure is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0020] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0021] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0022] In this specification and the embodiments, if the training and inference of the model are involved, the data involved (including but not limited to the data itself, the acquisition and / or use of the data) comply with the requirements of relevant laws and regulations.

[0023] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0024] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0025] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameters until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameters of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameters to determine the corresponding output.

[0026] As briefly discussed above, with the rapid development of generative technologies, model-based generation of content such as text, images, videos, and code has already shown good performance. The application of generative technologies in music generation has also led to the development of several applications, such as generating music from lyrics, text and melody, text and control commands, and plain text. However, these traditional music generation applications are typically slow, produce short, low-quality music, and often lack vocals, with room for improvement in controllability.

[0027] Embodiments of this disclosure provide a scheme for generating music. According to various embodiments of this disclosure, the following steps are taken: obtaining input reference audio content; determining a first set of audio features based on the reference audio content; constructing a target input sequence based on the first set of audio features and music description information, wherein the music description information describes the music content to be generated; providing the target input sequence to a music generation model to generate a second set of audio features; and generating the target music content based on the second set of audio features.

[0028] In this way, embodiments of the present disclosure can perform controllable music content generation based on input information, thereby improving the quality of the generated music content and increasing the flexibility of the generation process.

[0029] The audio content, timbre information, lyrics, etc. mentioned in this disclosure are used under license.

[0030] Example Environment

[0031] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. Figure 1 In environment 100, it is desired to train and use a music generation model 130 configured for various application environments. For example, the music generation model 130 may be configured to generate music using input lyrics. Alternatively, it may be configured to continue writing parts of the input lyrics and generate music based on the continued lyrics. Yet another example is that the music generation model may be configured to generate music using multimodal information such as input audio, video, or images.

[0032] like Figure 1 As shown, environment 100 includes training sample set 110, model training system 120 and model application system 140. Figure 1 The upper part shows the process of the model training phase, and the lower part shows the process of the model application phase. Before training, the parameters of the music generation model 130 can have initial values, or they can have pre-trained parameters obtained through a pre-training process.

[0033] During the model training phase, a music generation model 130 can be trained using a training sample set 110 comprising multiple training samples 111 and a model training system 120. Each training sample 111 can involve a binary format. For example, a training sample 111 may include a model input 112 and a model output 113 representing the music generation task. The model input 112 representing the music generation task may include, for example, a set of sample audio features and sample description information describing the sample music content, while the model output 113 may include, for example, the sample music content. The training samples 111, including the model input 112 and model output 113, can be used to train the music generation model 130. Specifically, the training process can be performed iteratively using a large number of training samples. The music generation model 130 can be trained via forward and backward propagation, during which the parameters of the music generation model 130 can be updated and adjusted.

[0034] After training is complete, a music generation model 130' is obtained. At this point, the parameters of the music generation model 130' have been updated, and based on the updated parameters, the music generation model 130' can be used to implement the music generation task during the model application phase. In some embodiments, the music generation model 130 can be a language model based on the transformer architecture.

[0035] During the model application phase, the model application system 140 utilizes the music generation model 130' (which has trained parameters) to perform the corresponding task. For example, it can receive a model input 141 representing the music generation task and output a corresponding model output 142, which may include the target music content corresponding to the music generation task.

[0036] exist Figure 1 In this context, the model training system 120 and the model application system 140 can include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. Terminal devices can involve any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0037] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0038] Example Interaction

[0039] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure. Figure 2 A schematic diagram of an architecture 200 for generating music according to some embodiments of the present disclosure is shown. For ease of discussion, reference will be made to... Figure 1 Environment 100 is discussed in relation to architecture 200.

[0040] In embodiments of this disclosure, the model training system 120 can acquire sample music content 201.

[0041] Here, the model training system 120 can obtain various types of music or songs as sample music content 201 from, for example, a music database. The sample music content 201 can be formed from a whole piece of music or song, or from a piece of music or a fragment of a song. After obtaining the music, piece of music, song, or fragment of a song, the obtained music, piece of music, song, or fragment of a song can be filtered to improve the quality of the sample music content 201.

[0042] For example, the model training system 120 can acquire the entire song from music data. Then, the model training system 120 can slice the entire song. To maintain integrity, the segment boundaries of the song can be determined as slice locations. Subsequently, the sliced ​​song fragments can be filtered based on parameters such as clarity, completeness, and semantics, and each filtered song fragment is used as sample music content 201.

[0043] In embodiments of this disclosure, the model training system 120 may determine a set of sample audio features 202 and sample description information 203 based on sample music content 201.

[0044] Alternatively or additionally, the model training system 120 may perform encoding processing on the sample music content 201 to obtain a set of sample audio features 202. For example, as... Figure 2 As shown, the model training system 120 can use the audio tagger 220 to perform encoding processing on the sample music content 201 to generate a set of sample audio features 202.

[0045] Regarding the encoding method of sample music content 201, in one example, sample music content 201 can be encoded as a set of sample music tokens. This set of sample music tokens can relatively well preserve the timbre characteristics of sample music content 201, and through this set of sample music tokens, the music generation model 130 can accurately indicate the timbre characteristics of sample music content 201. In another example, sample music content 201 can be encoded as a set of sample music embedding vectors. This set of sample music embedding vectors can relatively well preserve the melodic characteristics of sample music content 201, and through this set of sample music embedding vectors, the music generation model can accurately indicate the melodic characteristics of sample music content 201. It is understood that the above encoding methods are only exemplary; in specific implementations, the encoding method of sample music content 201 can be selected and configured according to actual needs.

[0046] The sample description information 203 here is used to describe the sample music content. Alternatively or additionally, the sample description information 203 may include lyrics 204. Alternatively or additionally, the sample description information 203 here may also be determined based on at least one of the following: another sample music content, sample video content, sample description text, and a set of sample tags 205.

[0047] The two sample music contents here are different music contents, and can be used to prompt different dimensions of music characteristics to the music generation model 130. For example, the style of the music can be prompted by the sample music content 201, and the timbre can be prompted by the other sample music content, so that the music generation model 130 can generate predicted music content 207 with a style similar to the sample music content 201 and a timbre similar to the other sample music content.

[0048] Here, the music generation model 130 can be used to generate voiceover or background music for the sample video content. The sample description text can be used to describe the sample music content 201 or another sample music content. For example, the sample description text can be used to describe the style, singing style, singing method, timbre characteristics, and other features of the sample music content.

[0049] The set of sample labels 205 here may include one or more labels, each of which can annotate the musical characteristics of the sample music content 201 from different dimensions. For example, the set of sample labels 205 may include labels for annotating musical characteristics such as style, scene, and timbre.

[0050] In some embodiments, the model training system 120 may provide the obtained sample music content 201 to the music analysis model 210, and use the music analysis model 210 to generate sample description information 203. For example, the music analysis model 210 may be used to identify the lyrics of the sample music content to obtain lyrics content 204. The music analysis model 210 may also be used to analyze the style, scene, timbre, and other musical characteristics of the sample music content 201 to obtain a set of sample labels 205. Of course, in practical applications, the sample description information 203 can also be obtained through other means, such as obtaining a set of sample labels 205 input by the user.

[0051] In embodiments of this disclosure, the model training system 120 constructs a training input sequence based on a set of sample music features 202 and sample description information 203.

[0052] Alternatively or additionally, the model training system 120 may perform encoding processing on the sample description information 203 to obtain sample description features. Then, a training input sequence is constructed based on the sample music features 202 and the sample description features.

[0053] For example, the sample description information 203 may include information from multiple dimensions such as lyrics content 204, a set of sample labels 205, and sample description text. This information from multiple dimensions can be encoded into multiple sets of sample description tokens corresponding to each dimension by the model training system 120. The model training system 120 may also encode these multiple sets of sample description tokens into a set of sample description embedding vectors. Then, a training input sequence is constructed based on, for example, a set of sample music embedding vectors or a set of sample music tokens, and the set of sample description embedding vectors. This training input sequence can be used as the model input 112 of the music generation model 130.

[0054] In some embodiments, the model training system 120 may generate multiple lyric tokens corresponding to the lyrics content 204, each lyric token indicating phoneme information and pitch information corresponding to a corresponding lyric part. Then, the model training system 120 may determine a set of sample description tokens based on the multiple lyric tokens. Exemplarily, the model training system 120 may convert the lyrics content 204 into pronunciation information, which may include phoneme information and pitch information. The model training system 120 may encode the phoneme information and pitch information corresponding to each lyric part (e.g., a pronunciation unit) in the lyrics content 204 into lyric tokens. Then, the multiple lyric tokens may be combined in the order of the lyric parts in the lyrics content 204 to form a set of sample description tokens.

[0055] In some embodiments, the model training system 120 may insert multiple structure tokens representing structural information into multiple lyric tokens based on the structural information of the lyrics content 204 to determine a set of sample description tokens. The structural information here is used to indicate the paragraph structure of the lyrics content 204. For example, the model training system 120 may insert section tag tokens that label song paragraphs such as the intro, verse, chorus, and bridge into the multiple lyric tokens. It is understood that the above-mentioned structure tokens are merely exemplary. In practical applications, there may be various paragraph structures in a song, and the corresponding structural information may not be the same, and the structure tokens representing each piece of structural information may also be different.

[0056] In some embodiments, the model training system 120 may, based on the singer information in the lyrics content 204, insert multiple singing tokens representing singer information into multiple lyric tokens to determine a set of sample description tokens. For example, a song may include multiple singers, and a singer token may be added before a set of lyric tokens corresponding to each line of lyrics in the lyrics content 204.

[0057] In embodiments of this disclosure, model training system 120 may provide training input sequences to music generation model 130 to generate at least one set of predicted audio features. Model training system 120 may generate at least one predicted music content 207 based on the at least one set of predicted audio features. Subsequently, model training system 120 may train music generation model 130 based on at least one predicted music content 207. The at least one predicted music content 207 may include one or more predicted music content sets 207.

[0058] For example, the model training system 120 can use the music generation model 130 to generate a set of predicted audio features 206 based on the training input sequence. The decoder 230 can then use this set of predicted audio features 206 to generate predicted music content 207. Subsequently, the model training system 120 can update the parameters of the music generation model 130 based on the differences between the sample music content 201 and the predicted music content 207.

[0059] In some embodiments, the model training system 120 may determine the evaluation of the at least one predicted music content 207, and train the music generation model 130 based on the evaluation of the at least one predicted music content 207. Exemplarily, the model training system 120 may use an evaluation model to evaluate the at least one predicted music content 207 and obtain an evaluation of the at least one predicted music content 207. This evaluation may include qualitative and / or quantitative evaluation results, such as evaluation grades, scores, rankings, etc.

[0060] In some embodiments, the evaluation of the predicted music content 207 may be determined based on at least one of the following: the similarity between the predicted music content 207 and the sample music content 201, the matching degree between the predicted music content 207 and the sample description information 203, and the melodic rationality of the predicted music content 207.

[0061] The predicted similarity between music content 207 and sample music content may include the predicted similarity between music content 207 and the overall sample music content, or the predicted similarity between music content 207 and sample music content in one or more dimensions of music features.

[0062] The matching degree between the predicted music content 207 and the sample description information can include the matching degree between the predicted music content and any one or more of the sample description information 203, such as lyrics 204, a set of sample labels 205, sample video content, and sample description text. For example, the matching degree between the lyrics of the predicted music content 207 and the style of the lyrics 204 can be determined, and the matching degree between the predicted music content 207 and the sample labels such as style, scene, and timbre in the set of sample labels 205 can be determined, etc.

[0063] The melodic rationality of the predicted musical content 207 here may include the degree of matching between the melody of the predicted musical content 207 and the melody evaluation criteria in music theory or music technology. For example, it can be determined how well the melody of the predicted musical content 207 matches the principles of musical harmony, aesthetic principles, cultural background, and melodies of traditional musical styles (such as jazz melodies, classical melodies, etc.).

[0064] The evaluation of the predicted music content 207 may also include human evaluation, such as obtaining feedback information on the predicted music content 207.

[0065] After determining the evaluation of the predicted music content 207, the model training system 120 can update the parameters of the music generation model 130 based on the evaluation of the predicted music content 207 to achieve the purpose of training the music generation model. In one example, the model training system 120 can update the parameters of the music generation model 130 based on the similarity between the predicted music content 207 and the sample music content, thereby improving the similarity between the predicted music content 207 generated by the music generation model 130 and the sample music content 201. In another example, the model training system 120 can update the parameters of the music generation model 130 based on the matching degree between the predicted music content 207 and the sample description information, thereby improving the matching degree between the predicted music content 207 and the sample description information 203. In yet another example, the model training system 120 can determine the comprehensive evaluation of the predicted music content 207 based on multiple evaluations, and update the parameters of the music generation model 130 based on the comprehensive evaluation, thereby improving the performance of the music generation model 130 in multiple dimensions.

[0066] It is understandable that the specific way the model training system 120 trains the music generation model 130 based on the evaluation of the predicted music content 207 also depends to some extent on the application method of the music generation model 130.

[0067] In an exemplary application scenario, lyrics 204 can be provided to the music generation model 130. The music generation model 130 then continues writing lyrics based on the lyrics 204, and subsequently generates a song based on the continued lyrics. In this application scenario, the model training system 120 can determine the matching degree between the predicted music content 207 and the lyrics 204, and update the parameters of the music generation model 130 based on the matching degree between the predicted music content 207 and the lyrics 204, so as to improve the matching degree between the predicted music content 207 generated by the music generation model 130 and the lyrics 204 in subsequent applications.

[0068] In another exemplary application scenario, sample music content 201 and another sample music content can be provided to the music generation model 130. The music generation model 130 then generates music with a timbre similar to that of the sample music content 201 and a melody similar to that of the other sample music content. In this application scenario, the model training system 120 can determine the timbre similarity between the predicted music content 207 and the sample music content 201, and also determine the melody similarity between the predicted music content 207 and the other sample music content. Subsequently, the parameters of the music generation model 130 can be updated based on the timbre similarity and melody similarity.

[0069] To fully illustrate the training process of music generation model 130, the following will combine... Figure 3 The training process of the music generation model 130 will be illustrated again by way of example. Figure 3 A schematic diagram of an architecture 300 for generating music according to some embodiments of the present disclosure is shown. For ease of discussion, reference will be made to... Figure 1 The environment 100 will be discussed.

[0070] It should be noted that, Figure 3 The architecture 300 shown is... Figure 2 The architecture shown in 200 is similar. Figure 3 The architecture 300 shown is... Figure 2 The main differences between the architectures 200 shown lie in the number of predicted music contents generated based on the training input sequences, and the different training methods used to train the music generation model. Processes that are the same or similar to those in architecture 300 and architecture 200 will not be described in detail here; please refer to the aforementioned introduction to architecture 200. The differences between architecture 300 and architecture 200 will be explained in detail below.

[0071] In embodiments of this disclosure, the model training system 120 provides a training input sequence to the music generation model 130, which can use the music generation model 130 to generate a set of predicted audio features 206 and a set of predicted audio features 306, respectively.

[0072] In embodiments of this disclosure, the model training system 120 can use decoder 230 to generate predicted music content 207 (sometimes referred to herein as "first predicted music content") based on the set of predicted audio features 206, and can also use decoder 330 to generate predicted music content 307 (sometimes referred to herein as "second predicted music content") based on the set of predicted audio features 306. Here, decoder 230 and decoder 330 can be the same decoder or different decoders.

[0073] In embodiments of this disclosure, model training system 120 may determine an evaluation of predicted music content 207 (sometimes referred to herein as a “first evaluation”) and an evaluation of predicted music content 307 (sometimes referred to herein as a “second evaluation”). Exemplarily, model training system 120 may utilize evaluation model 340 to determine the evaluation of predicted music content 207 and the evaluation of predicted music content 307.

[0074] In embodiments of this disclosure, the model training system 120 can compare the evaluation of predicted music content 207 with the evaluation of predicted music content 307. If it is determined that the evaluation of predicted music content 207 is better than the evaluation of predicted music content 307, the parameters of the music generation model 130 can be adjusted so that the generation probability of predicted music content 207 is higher than the generation probability of predicted music content 307.

[0075] It should be understood that the above process of training the music generation model 130 based on the evaluation of predicted music content is merely exemplary. In practical applications, the evaluation criteria for the predicted music content can be flexibly selected and configured according to actual needs, and the corresponding process of training the music generation model 130 based on the evaluation of predicted music content can be selected.

[0076] The following will combine Figure 4 The application process (also known as the reasoning process) of the music generation model 130 is explained. Figure 4 A schematic diagram of an architecture 400 for generating music according to some embodiments of the present disclosure is shown. For ease of discussion, reference will be made to... Figure 1 Environment 100 is discussed in relation to architecture 400.

[0077] In embodiments of this disclosure, the model application system 140 may acquire input reference audio content 401. A user may input the reference audio content 401 into the model application system 140 using a music input device. The model application system 140 may also acquire the reference audio content 401 from a local device or other electronic device. Exemplarily, a song or song segment may be acquired as the reference audio content 401. In some embodiments, the reference audio content 401 is an optional input item, which may, for example, be omitted during the generation process.

[0078] In embodiments of this disclosure, the model application system 140 may determine a first set of audio features 402 based on reference audio content 401.

[0079] Alternatively or additionally, the model application system 140 may perform encoding processing on the reference audio content 401 to obtain a first set of audio features 402. For example, as... Figure 4 As shown, the model application system 140 can use the audio tagger 410 to encode the reference audio content 401 into a first audio feature 402.

[0080] Regarding the encoding method of the reference audio content 401, in one example, the reference audio content 401 can be encoded as a set of reference audio tokens. This set of reference audio tokens can relatively well preserve the timbre characteristics of the reference audio content 401, and through this set of reference audio tokens, the music generation model 130 can accurately indicate the timbre characteristics of the reference audio content 401.

[0081] In another example, the reference audio content 401 can be encoded as a set of reference audio embedding vectors. This set of reference audio embedding vectors can preserve the melodic characteristics of the reference audio content 401 relatively well, and can accurately indicate the melodic characteristics of the reference audio content 401 to the music generation model. It is understood that the above encoding method is only exemplary; in specific implementations, the encoding method of the reference audio content 401 can be selected and configured according to actual needs.

[0082] In embodiments of this disclosure, the model application system 140 constructs a target input sequence based on a first set of audio features and music description information.

[0083] The music description information 403 here is used to describe the music content to be generated. Alternatively or additionally, the music description information 403 may include lyrics 404. Alternatively or additionally, the music description information 403 here may also be determined based on at least one of the following: reference audio content 406, reference video content, reference descriptive text, and a set of music tags 405.

[0084] The two reference audio contents 401 and 406 here are different audio contents, which can be used to prompt different dimensions of music characteristics to the music generation model 130'. For example, the reference audio content 401 (sometimes referred to herein as "first reference audio content") can be used to prompt the style of the music, and the reference audio content 406 (sometimes referred to herein as "second reference audio content") can be used to prompt the timbre characteristics, so that the music generation model 130' can generate target music content 408 with a style similar to the reference audio content 401 and a timbre similar to the reference audio content 406.

[0085] Here, the music generation model 130' can be instructed to generate voice-over or background music for the reference video content. The reference description text can use, for example, natural language to describe the music content to be generated. For example, the reference description text can describe the style, singing style, singing method, timbre characteristics, and other features of the music content to be generated.

[0086] A set of music tags 405 here may include one or more tags, each tag can annotate the musical characteristics of the music content to be generated from different dimensions. For example, this set of music tags 405 may include tags for annotating musical characteristics such as style, scene, and timbre.

[0087] In some embodiments, the model application system 140 may provide reference audio content 401 to the music analysis model to generate music description information. For example, the music analysis model may be used to identify the lyrics of the reference audio content 401 to obtain lyrics content 404. The music analysis model may also be used to analyze the style, scene, timbre, and other musical characteristics of the reference audio content 401 to obtain a set of music tags 405.

[0088] In some embodiments, the model application system 140 can determine music description information based on input reference description data. For example, additional control signals can be input to the model application system 140 to indicate musical characteristics such as style, scene, and timbre of the music content to be generated. The model application system 140 can determine music description information based on the additional control signals.

[0089] In some embodiments, the model application system 140 may determine a set of description information tokens based on the music description information 403. Then, a target input sequence may be constructed based on the first set of audio features 402 and the set of description information tokens.

[0090] In some embodiments, the model application system 140 may also determine a set of description information embedding vectors based on the music description information 403. Then, a target input sequence can be constructed based on the first set of audio features 402 and the set of description information embedding vectors.

[0091] For example, such as Figure 4 As shown, the music description information 203 may include multiple dimensions of description information such as lyrics 404, a set of music tags 405, reference audio content 406, and reference description text. The model application system 140 can encode these multiple dimensions of description information into multiple sets of description information tokens. The model application system 140 can also encode these multiple sets of description information tokens into a set of description information embedding vectors. Then, the target input sequence is constructed based on the first set of audio features 402 and this set of description information embedding vectors.

[0092] In some embodiments, the model application system 140 can generate multiple lyric tokens corresponding to the lyrics content 404, each lyric token indicating phoneme information and pitch information corresponding to a corresponding lyric part. Then, a set of description information tokens can be determined based on the multiple lyric tokens. For example, the model application system 140 can convert the lyrics content 404 into pronunciation content, which may include phoneme information and pitch information. The model application system 140 can encode the phoneme information and pitch information corresponding to each lyric part (e.g., a pronunciation unit) in the lyrics content 404 into lyric tokens. Then, the multiple lyric tokens can be combined in the order of the lyric parts in the lyrics content 404 to form a set of description information tokens.

[0093] In some embodiments, the model application system 140 may, based on the structural information of the lyrics content 404, insert multiple structure tokens representing the structural information into multiple lyrics tokens to determine a set of descriptive information tokens. The structural information here is used to indicate the paragraph structure of the lyrics content 404. For example, the model application system 140 may insert section tag tokens (marking song paragraphs such as the intro, verse, chorus, and bridge) into multiple lyrics tokens. It is understood that the above-mentioned structure tokens are merely exemplary. In practical applications, there may be various paragraph structures in a song, and the corresponding structural information may not be the same; the structure tokens representing each piece of structural information may also be different.

[0094] In some embodiments, the model application system 140 may, based on the singer information in the lyrics content 404, insert multiple singing tokens representing singer information into multiple lyric tokens to determine a set of descriptive information tokens. For example, a song may include multiple singers, and a singer token may be added before the set of lyric tokens corresponding to each line of lyrics in the lyrics content 404.

[0095] In embodiments of this disclosure, the model application system 140 provides a target input sequence to the music generation model 130' to generate a second set of audio features 407.

[0096] The method by which the music generation model 130' generates the second audio feature 407 based on the target input sequence can vary depending on the type of machine learning model used by the music generation model 130'. The following example illustrates the process of the music generation model 130' generating the second audio feature 407. The second set of audio features 407 may include multiple audio features, such as audio feature 1, audio feature 2, audio feature 3, ..., audio feature N-1, audio feature N, etc. The model application system 140 provides the target input sequence to the music generation model 130', and uses the music generation model 130' to generate audio feature 1 based on the target input sequence. Then, the music generation model 130' generates audio feature 2 based on audio feature 1, and audio feature 3 based on audio feature 2. When the second set of audio features 407 includes N audio features, the music generation model 130' needs to perform the audio feature generation process N times to generate the second set of audio features 407.

[0097] In embodiments of this disclosure, the model application system 140 generates target music content 408 based on a second set of audio features 407. For example, as... Figure 4 As shown, the model application system 140 can use the decoder 420 to perform a decoding operation on the second set of audio features 407 to obtain the target audio content 408.

[0098] The target music content 408 here may include a piece of music, a fragment of music, a song, or a fragment of a song. It is understood that the specific content of the target music content 408 mainly depends on the provided reference audio content 401 and music description information 403. For example, music description information 403 may include textual description information used to describe the lyrics in the audio content to be generated. In this case, the music generation model 130' can be used to write lyrics corresponding to the textual description information, and a song can be generated based on the written lyrics. Alternatively, music description information 403 may include lyrics content 404, and the music generation model 130' can be used to continue writing complete lyrics based on the lyrics content 404, and a song can be generated based on the continued lyrics. Yet another example is that music description information 403 may include reference audio content 406, which can generate target music content 408 with a timbre and melody similar to reference audio content 401 and reference audio content 406. Still another example is that music description information 403 may include reference video content, which can generate target music content 408 as background music for the reference video content.

[0099] In summary, the embodiments of this disclosure can control the music content generation process based on appropriate types of input information, thereby improving the flexibility of the music generation process. Consequently, the embodiments of this disclosure can generate music content that better meets expectations and improve the quality of the generated music content.

[0100] Example processes, apparatus and equipment

[0101] Figure 5 A flowchart of a music generation process 500 according to some embodiments of the present disclosure is shown. Process 500 can be implemented in model training system 120 or model application system 140.

[0102] In box 510, the model application system 140 acquires the input reference audio content.

[0103] In box 520, the model application system 140 determines the first set of audio features based on the reference audio content.

[0104] In box 530, the model application system 140 constructs a target input sequence based on the first set of audio features and music description information, whereby the music description information is used to describe the music content to be generated.

[0105] In box 540, the model application system 140 provides the target input sequence to the music generation model to generate a second set of audio features.

[0106] In box 540, the model application system 140 generates target music content based on the second set of audio features.

[0107] In some embodiments, process 500 further includes: providing reference audio content to a music analysis model to generate music description information; or determining music description information based on input reference description data.

[0108] In some embodiments, constructing a target input sequence based on a first set of audio features and music description information includes: determining a set of description information tokens based on the music description information; and constructing a target input sequence based on the first set of audio features and the set of description information tokens.

[0109] In some embodiments, the music description information includes lyrics content, and determining a set of description information tokens based on the music description information includes: generating multiple lyrics tokens corresponding to the lyrics content, each lyrics token indicating phoneme information and pitch information corresponding to a corresponding lyric part; and determining a set of description information tokens based on the multiple lyrics tokens.

[0110] In some embodiments, determining a set of descriptive information tokens based on multiple lyric tokens includes: inserting multiple structural tokens representing structural information into the multiple lyric tokens based on structural information of the lyric content to determine a set of descriptive information tokens.

[0111] In some embodiments, determining a set of descriptive information tokens based on multiple lyric tokens includes: inserting multiple performance tokens representing the performance information into the multiple lyric tokens to determine a set of descriptive information tokens based on the performer information of the lyrics content.

[0112] In some embodiments, the reference audio content is a first reference audio content, and the music description information is also determined based on at least one of the following: a second reference audio content; reference video content; reference description text; and a set of music tags.

[0113] In some embodiments, the music generation model is trained based on the following process: the model training system 120 acquires sample music content; determines a set of sample audio features and sample description information based on the sample music content, the sample description information being used to describe the sample music content; constructs a training input sequence based on the set of sample music features and sample description information; and trains the music generation model using the training input sequence.

[0114] In some embodiments, training a music generation model using a training input sequence includes: providing a training input sequence to the music generation model to generate at least one set of predicted audio features; generating at least one predicted music content based on the at least one set of predicted audio features; and training the music generation model based on an evaluation of the at least one predicted music content.

[0115] In some embodiments, at least one predicted music content includes a first predicted music content and a second predicted music content generated based on a training input sequence, and training a music generation model based on the evaluation of at least one predicted music content includes: adjusting the parameters of the music generation model in response to a first evaluation of the first predicted music content being better than a second evaluation of the second predicted music content, such that the generation probability of the first predicted music content is higher than that of the second predicted music content.

[0116] In some embodiments, the evaluation is determined based on at least one of the following: the similarity between the predicted music content and the sample music content, the matching degree between the predicted music content and the sample description information, and the melodic rationality of the predicted music content.

[0117] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6A schematic structural block diagram of an apparatus 600 for generating music according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in a model training system 120 or a model application system 140. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0118] like Figure 6 As shown, the device 600 includes an acquisition module 610, a determination module 620, a construction module 630, a provision module 640, and a generation module 650. The acquisition module 610 is configured to acquire input reference audio content. The determination module 620 is configured to determine a first set of audio features based on the reference audio content. The construction module 630 is configured to construct a target input sequence based on the first set of audio features and music description information, where the music description information describes the music content to be generated. The provision module 640 is configured to provide the target input sequence to a music generation model to generate a second set of audio features. The generation module 650 is configured to generate the target music content based on the second set of audio features.

[0119] In some embodiments, the obtaining module 610 is further configured to: provide reference audio content to the music analysis model to generate music description information; or determine music description information based on the input reference description data.

[0120] In some embodiments, the construction module 630 is further configured to: determine a set of description information tokens based on music description information; and construct a target input sequence based on a first set of audio features and a set of description information tokens.

[0121] In some embodiments, the music description information includes lyrics content, and the construction module 630 is further configured to: generate multiple lyric tokens corresponding to the lyrics content, each lyric token indicating phoneme information and pitch information corresponding to the corresponding lyric part; and determine a set of description information tokens based on the multiple lyric tokens.

[0122] In some embodiments, the construction module 630 is further configured to: insert multiple structure tokens representing structure information into multiple lyric tokens based on the structure information of the lyrics content, in order to determine a set of descriptive information tokens.

[0123] In some embodiments, the construction module 630 is further configured to: insert multiple singing tokens representing singing information into multiple lyric tokens based on the singer information of the lyrics content, in order to determine a set of descriptive information tokens.

[0124] In some embodiments, the reference audio content is a first reference audio content, and the music description information is also determined based on at least one of the following: a second reference audio content; reference video content; reference description text; and a set of music tags.

[0125] In some embodiments, the apparatus 600 may further include a training module, which trains the music generation model based on the following process: acquiring sample music content; determining a set of sample audio features and sample description information based on the sample music content, wherein the sample description information is used to describe the sample music content; constructing a training input sequence based on the set of sample music features and sample description information; and training the music generation model using the training input sequence.

[0126] In some embodiments, the training module is further configured to: provide a training input sequence to the music generation model to generate at least one set of predicted audio features; generate at least one predicted music content based on the at least one set of predicted audio features; and train the music generation model based on the evaluation of at least one predicted music content.

[0127] In some embodiments, at least one predicted music content includes a first predicted music content and a second predicted music content generated based on a training input sequence, and the training module is further configured to: adjust the parameters of the music generation model in response to a first evaluation of the first predicted music content being superior to a second evaluation of the second predicted music content, such that the generation probability of the first predicted music content is higher than that of the second predicted music content.

[0128] In some embodiments, the evaluation is determined based on at least one of the following: the similarity between the predicted music content and the sample music content, the matching degree between the predicted music content and the sample description information, and the melodic rationality of the predicted music content.

[0129] The units and / or modules included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0130] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may include, for example, […]. Figure 1 The model training system 120 or the model application system 140 in the middle.

[0131] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The illustrated electronic device 700 can be implemented as Figure 1 The model training system 120 or the model application system 140, and / or Figure 6 Device 600.

[0132] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0133] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0134] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0135] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0136] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0137] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0138] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0140] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some, as newer, implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0142] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating music, comprising: obtaining input reference audio content; determining a first set of audio features based on the reference audio content; constructing a target input sequence based on the first set of audio features and music description information, the music description information being used to describe music content to be generated; providing the target input sequence to a music generation model to generate a second set of audio features; and generating target music content based on the second set of audio features.

2. The method of claim 1, further comprising: providing the reference audio content to a music analysis model to generate the music description information; or determining the music description information based on input reference description data.

3. The method of claim 1, wherein constructing the target input sequence based on the first set of audio features and the music description information comprises: determining a set of description information tokens based on the music description information; and constructing the target input sequence based on the first set of audio features and the set of description information tokens.

4. The method of claim 3, wherein the music description information comprises lyric content, and determining the set of description information tokens based on the music description information comprises: generating a plurality of lyric tokens corresponding to the lyric content, each lyric token indicating phoneme information and pitch information corresponding to a respective lyric portion; and determining the set of description information tokens based on the plurality of lyric tokens.

5. The method of claim 4, wherein determining the set of description information tokens based on the plurality of lyric tokens comprises: inserting a plurality of structure tokens in the plurality of lyric tokens based on structure information of the lyric content to determine the set of description information tokens.

6. The method of claim 4, wherein determining the set of description information tokens based on the plurality of lyric tokens comprises: inserting a plurality of singing tokens in the plurality of lyric tokens based on singer information of the lyric content to determine the set of description information tokens.

7. The method of claim 4, wherein the reference audio content is first reference audio content, and the music description information is further determined based on at least one of: second reference audio content; reference video content; reference description text; a set of music tags.

8. The method of claim 1, wherein the music generation model is trained based on the following process: obtaining sample music content; determining a set of sample audio features and sample description information based on the sample music content, the sample description information being used to describe the sample music content; constructing a training input sequence based on the set of sample music features and the sample description information; and training the music generation model using the training input sequence.

9. The method of claim 8, wherein training the music generation model using the training input sequence comprises: providing the training input sequence to the music generation model to generate at least one set of predicted audio features; generating at least one predicted music content based on the at least one set of predicted audio features; and ​ ​ ​ ​ ​ train the music generation model based on the evaluation of the at least one predicted music content. 10.The method of claim 9, wherein the at least one predicted music content comprises a first predicted music content and a second predicted music content generated based on the training input sequence, and training the music generation model based on the evaluation of the at least one predicted music content comprises: in response to a first evaluation of the first predicted music content being better than a second evaluation of the second predicted music content, adjusting parameters of the music generation model such that a generation probability of the first predicted music content is higher than the second predicted music content. 11.The method of claim 9, wherein the evaluation is determined based on at least one of the following information: a similarity of the predicted music content to the sample music content, a matching degree of the predicted music content to the sample description information, a melody reasonableness of the predicted music content. 12.An apparatus for generating music, comprising: an obtaining module configured to obtain an input reference audio content; a determining module configured to determine a first set of audio features based on the reference audio content; a constructing module configured to construct a target input sequence based on the first set of audio features and music description information, the music description information being used to describe music content to be generated; a providing module configured to provide the target input sequence to a music generation model to generate a second set of audio features; and a generating module configured to generate target music content based on the second set of audio features. 13.An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1-11. 14.A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-11. 15.A computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the method according to any one of claims 1-11. ​ ​

Images