Music generation method and device, electronic equipment and storage medium

Abstract

The invention provides a music generation method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a lyric text to be converted and a reference audio; mapping the reference audio to a potential space from an original audio space to obtain a first audio feature; based on a diffusion model acting on the potential space, performing multi-step noise adding processing on the first audio feature to obtain a noise feature; under the condition that preconditions are given, multi-step denoising processing is carried out on the noise features based on a diffusion model, and second audio features are obtained; wherein the precondition is related to the total duration of the lyric text and the reference audio; and mapping the second audio feature from the potential space to the original audio space to obtain a music work for singing the lyric text. The user input in the music generation process is simplified, and the music generation efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a music generation method, device, electronic device and storage medium. Background Art

[0002] Nowadays, music generation has become an important application in the field of AIGC (Artificial Intelligence Generated Content). Music generation converts lyrics text entered by users into natural and pleasant music waveforms, thus creating music works for users.

[0003] The related art requires the user to input detailed information when generating music. For example, the user is required to input the lyrics text, the duration and pitch of each word, etc. Since the user input is too complicated, the music generation efficiency is low. Summary of the invention

[0004] The present disclosure provides a music generation method, device, electronic device and storage medium. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a music generation method, comprising:

[0006] Get the lyrics text and reference audio to be converted;

[0007] Mapping the reference audio from the original audio space to the latent space to obtain a first audio feature;

[0008] Based on the diffusion model acting on the latent space, performing a multi-step noise addition process on the first audio feature to obtain a noise feature;

[0009] Under a given precondition, performing a multi-step denoising process on the noise feature based on the diffusion model to obtain a second audio feature; wherein the precondition is related to the total duration of the reference audio and the lyrics text;

[0010] The second audio feature is mapped from the latent space back to the original audio space to obtain a musical work for singing the lyrics text.

[0011] In some embodiments, the method further comprises:

[0012] Acquire the phoneme features of the lyrics text;

[0013] Vectorizing the total duration of the reference audio to obtain a duration feature;

[0014] The phoneme feature and the duration feature are concatenated, and the concatenated feature is input as the precondition into the reverse denoising process of the diffusion model.

[0015] In some other embodiments, obtaining the phoneme features of the lyrics text includes:

[0016] Performing phonetic conversion on the lyrics text to obtain a phoneme sequence;

[0017] Vectorizing the phoneme sequence to obtain initial phoneme features;

[0018] Based on the initial phoneme features, context information of the lyrics text is obtained; wherein the context information is used to describe potential semantic and emotional information in the lyrics text;

[0019] The context information is integrated into the initial phoneme feature to obtain a final phoneme feature.

[0020] In other embodiments, the precondition is related to the lyrics text, the total duration of the reference audio, and the music type;

[0021] The music type is used to indicate at least one of the music style, vocal timbre or accompaniment intensity of the reference audio.

[0022] In some other embodiments, the method further comprises:

[0023] Acquire music type features of the reference audio;

[0024] The step of concatenating the phoneme feature and the duration feature, and inputting the concatenated feature as the precondition into the reverse denoising process of the diffusion model includes:

[0025] The phoneme feature, the duration feature and the music type feature are concatenated, and the concatenated features are input as the preconditions into the reverse denoising process of the diffusion model.

[0026] In some other embodiments, the step of inputting the concatenated features as the precondition into the reverse denoising process of the diffusion model includes:

[0027] Based on the cross-attention mechanism, the preconditions are input as keys and values ​​into the reverse denoising process of the diffusion model.

[0028] In some other embodiments, the reference audio is an audio signal having a specific sampling rate;

[0029] Mapping the reference audio from the original audio space to the latent space to obtain the first audio feature includes:

[0030] An encoder based on a variational autoencoder maps the audio signal from the original audio space to the latent space to obtain the first audio feature.

[0031] In some other embodiments, mapping the second audio feature from the latent space back to the original audio space to obtain a musical work for singing the lyrics text includes:

[0032] Based on the decoder of the variational autoencoder, the second audio feature is mapped from the latent space back to the original audio space to obtain a musical work for singing the lyrics text; wherein the musical work includes vocals and accompaniment.

[0033] According to a second aspect of an embodiment of the present disclosure, there is provided a music generating device, comprising:

[0034] An acquisition module is configured to acquire lyrics text and reference audio to be converted;

[0035] A first encoding module is configured to map the reference audio from an original audio space to a latent space to obtain a first audio feature;

[0036] A generating module is configured to perform a multi-step noise addition process on the first audio feature based on a diffusion model acting on the latent space to obtain a noise feature; and, under a given precondition, perform a multi-step denoising process on the noise feature based on the diffusion model to obtain a second audio feature; wherein the precondition is related to the total duration of the reference audio and the lyrics text;

[0037] The first decoding module is configured to map the second audio feature from the latent space back to the original audio space to obtain a musical work for singing the lyrics text.

[0038] In some embodiments, the apparatus further comprises:

[0039] A second encoding module is configured to obtain the phoneme features of the lyrics text;

[0040] A third encoding module is configured to vectorize the total duration of the reference audio to obtain a duration feature;

[0041] The processing module is configured to perform feature concatenation on the phoneme feature and the duration feature, and input the concatenated feature as the precondition into the reverse denoising process of the diffusion model.

[0042] In some other embodiments, the second encoding module is configured to:

[0043] Performing phonetic conversion on the lyrics text to obtain a phoneme sequence;

[0044] Vectorizing the phoneme sequence to obtain initial phoneme features;

[0045] Based on the initial phoneme features, context information of the lyrics text is obtained; wherein the context information is used to describe potential semantic and emotional information in the lyrics text;

[0046] The context information is integrated into the initial phoneme feature to obtain a final phoneme feature.

[0047] In other embodiments, the precondition is related to the lyrics text, the total duration of the reference audio and the music type; wherein the music type is used to indicate at least one of the musical style, vocal timbre or accompaniment intensity of the reference audio.

[0048] In some other embodiments, the device further comprises:

[0049] A fourth encoding module is configured to obtain a music type feature of the reference audio;

[0050] The processing module is configured as follows:

[0051] The phoneme feature, the duration feature and the music type feature are concatenated, and the concatenated features are input as the preconditions into the reverse denoising process of the diffusion model.

[0052] In some other embodiments, the processing module is configured to:

[0053] Based on the cross-attention mechanism, the preconditions are input as keys and values ​​into the reverse denoising process of the diffusion model.

[0054] In some other embodiments, the reference audio is an audio signal having a specific sampling rate;

[0055] The first encoding module is configured as follows:

[0056] An encoder based on a variational autoencoder maps the audio signal from the original audio space to the latent space to obtain the first audio feature.

[0057] In some other embodiments, the first decoding module is configured to:

[0058] Based on the decoder of the variational autoencoder, the second audio feature is mapped from the latent space back to the original audio space to obtain a musical work for singing the lyrics text; wherein the musical work includes vocals and accompaniment.

[0059] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0060] one or more processors;

[0061] A memory for storing program codes executable by the processor;

[0062] Among them, the processor is configured to execute the program code to implement the music generation method described in the first aspect above.

[0063] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, and when a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to execute the music generation method described in the first aspect above.

[0064] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor of an electronic device, implements the music generation method described in the first aspect above.

[0065] The disclosed embodiment provides a new music generation solution, which simplifies the user input in the music generation process. The user only needs to input the lyrics, without the need for the user to input the duration and pitch of each word in the lyrics. Since the user input is simplified, it not only lowers the threshold for users to create music by themselves, but also improves the efficiency of music generation.

[0066] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0068] Figure 1 It is a schematic diagram of the overall framework of a music generation system according to an exemplary embodiment.

[0069] Figure 2 It is a schematic diagram of an implementation environment of a music generation method according to an exemplary embodiment.

[0070] Figure 3 The present invention is a flowchart of a music generating method according to an exemplary embodiment.

[0071] Figure 4 It is a schematic diagram of the overall framework of another music generation system according to an exemplary embodiment.

[0072] Figure 5 is a flowchart of another music generating method according to an exemplary embodiment.

[0073] Figure 6 It is a schematic diagram showing a lyrics analysis process according to an exemplary embodiment.

[0074] Figure 7 It is a block diagram of a music generating device according to an exemplary embodiment.

[0075] Figure 8 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0076] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0077] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0078] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0079] Some nouns or abbreviations involved in the embodiments of the present disclosure are first introduced below.

[0080] AIGC: Use artificial intelligence technology to generate content with certain creativity and quality, such as text, images, music and video. That is, the core idea of ​​AIGC technology is to generate relevant content based on the input conditions or guidance through training models and learning from a large amount of data. For example, by inputting keywords, descriptions or samples, content matching them is generated.

[0081] G2P (Grapheme-to-Phoneme): After inputting text, the pinyin or phoneme annotation of the text can be output using G2P technology. Usually, G2P technology adopts a rule-based dictionary mapping method.

[0082] VAE (Variational Autoencoder): A generative model with wide applications in the field of AIGC. The structure of VAE consists of two main parts: encoder and decoder. The encoder converts the real sample into an ideal data distribution, which is then passed to the decoder to generate new samples.

[0083] With the continuous iteration of hardware computing power such as graphics cards and the continuous in-depth research on deep learning technology, music generation methods are also constantly evolving. Among them, the embodiments of the present disclosure provide an end-to-end, low-latency music generation system that can simultaneously generate vocals and accompaniment. Figure 1 It is a schematic diagram of the overall framework of a music generation system according to an exemplary embodiment.

[0084] See also Figure 1 ,According to the functional division, the process of music generation can be divided into two parts: front-end lyrics analysis and back-end waveform synthesis. Among them, the front-end lyrics analysis is used to provide lyrics analysis function for the input lyrics text, and the back-end waveform synthesis part is used to convert the lyrics analysis results into music waveforms.

[0085] It should be noted that the application scenarios of music generation technology are wide-ranging, covering multiple fields such as music creation, film and television soundtracks, game sound effects, music editing, and singing conversion. As users' demand for personalized content increases, music generation technology will provide more convenient music creation tools for more non-professional creators. In addition, with the emergence of more efficient algorithms and further improvements in computing hardware, music generation technology will tend to have higher naturalness and emotional expression capabilities. In addition, combined with technologies such as virtual reality (VR) and augmented reality (AR), music generation technology can bring users a more immersive music experience.

[0086] Figure 2 It is a schematic diagram of an implementation environment of a music generation method according to an exemplary embodiment.

[0087] See also Figure 2 , the implementation environment includes: a terminal 201 and a server 202. The terminal 201 is an electronic device used by a user. In the embodiment of the present disclosure, a target application is installed on the terminal 201. Exemplarily, the target application is an instant messaging application, such as a short video application, which is not limited in the present disclosure.

[0088] In some embodiments, the music generation method is jointly executed by the terminal 201 and the server 202. For example, the music generation process is: the terminal 201 obtains the lyrics text and reference audio input by the user based on the target application, and then sends the lyrics text and reference audio to the server 202, and then the server 202 generates music and feeds back the generated music work to the terminal 201, so that the terminal 201 presents the generated music work to the user.

[0089] In other embodiments, the terminal 201 is a type of device such as a smart phone, a desktop computer, or a laptop computer. Figure 2 The terminal 201 is only used as a smart phone for illustration. In addition, those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminals can be only a few, or the above devices can be dozens or hundreds, or more. The embodiment of the present disclosure does not limit the number and type of terminals.

[0090] In other embodiments, the server 202 provides background services for the target application. In addition, the server 202 is connected to the terminal 201 via a wireless network or a wired network. In addition, the server involved in the embodiment of the present disclosure may also include other functional servers to provide more comprehensive and diversified services.

[0091] The music generation solution provided by the embodiment of the present disclosure is introduced in detail through the following embodiments.

[0092] Figure 3 is a flowchart of a music generation method according to an exemplary embodiment. Figure 3 As shown, the music generation method is applied to electronic devices, such as Figure 2 The terminal 201 and the server 202 shown are jointly executed. The music generation method includes the following steps:

[0093] In 301 , the electronic device obtains lyrics text to be converted and reference audio.

[0094] For LTS (Lyrics-to-Sing) tasks, the disclosed embodiments are based on LDM (Latent Diffusi-on Model) and can simultaneously generate harmonious and high-fidelity vocals and accompaniment.

[0095] Figure 4 FIG. 1 is a schematic diagram showing the overall framework of another music generation system according to an exemplary embodiment. Figure 4 As shown, LDM mainly consists of three parts: conditional encoder, diffusion model ( Figure 3The U-Net Transformer is used as an example in the paper), and the generative model for audio compression and waveform recovery ( Figure 3 Take VAE as an example).

[0096] In some embodiments, the lyrics text entered by the user is further input Figure 4 The text encoder in is fed into the VAE (a conditional encoder), while the reference audio is fed into the VAE’s encoder.

[0097] In the LTS task, lyrics are used as pronunciation conditions, while vocal timbre, musical style, and accompaniment are considered details built around the pronunciation information. In addition to the lyrics, if conditions related to these details are not provided, the ability of LDM to generate these details will be affected by the level of randomness, and the introduction of conditions related to these details can reduce this dependence and keep the generation ability of LDM stable. Based on this, the embodiment of the present disclosure will also obtain reference audio, and then use the reference audio as a singing prompt to guide music generation. Exemplarily, the reference audio is input by the user and is an audio signal with a specific sampling rate. For example, the reference audio is a high-fidelity audio signal with a sampling rate of 44.1kHz, which is not limited in the present disclosure.

[0098] In 302, the electronic device maps the reference audio from the original audio space to the latent space to obtain a first audio feature.

[0099] Taking the audio signal with a monophonic reference audio and a sampling rate of 44.1kHz as an example, the embodiment of the present disclosure uses the VAE encoder to map (also called compress) the audio signal from the high-dimensional original audio space to the low-dimensional latent space (LS) to obtain the first audio feature, that is, Figure 4 Feature Z0 shown in .

[0100] Exemplarily, the VAE is composed of an encoder and a decoder composed of fully convolutional layers, allowing encoding and decoding of sequences of arbitrary length. In addition, the embodiment of the present disclosure adopts the Snake activation function to improve the quality of audio reconstruction at a high compression ratio. In addition, in the process of training the VAE, the training objectives include reconstruction loss, adversarial loss and Gaussian constraint loss, which are not limited in the present disclosure.

[0101] In summary, the VAE encoder approximates the complex probability distribution of the input data through variational inference, thereby completing the data modeling. Among them, the VAE encoder outputs the parameters (mean and variance) of the probability distribution in the latent space, rather than just a single encoding. This process enables the VAE decoder to generate diverse outputs from the latent space, enhancing the VAE's ability to generate new data.

[0102] In 303, the electronic device performs a multi-step noise addition process on the first audio feature based on a diffusion model acting on the latent space to obtain a noise feature; and, under a given precondition, performs a multi-step denoising process on the noise feature based on the diffusion model to obtain a second audio feature; wherein the precondition is related to the total duration of the reference audio and the lyrics text.

[0103] The diffusion model usually includes two processes, namely, the forward diffusion process (also called the forward noise addition process or the forward process) and the reverse diffusion process (also called the reverse denoising process, the sampling process or the reverse process).

[0104] It should be noted that, taking the example that both the forward process and the reverse process include N time steps, the noise adding process lasts N times, and the noise removal process also lasts N times, where the value of N is a positive integer.

[0105] For the forward process, see Figure 4 , given the original sample Z0, gradually add Gaussian noise to the original sample Z0, for example, add a small Gaussian noise each time. The noise addition process lasts N times, also known as N time steps. In this process, a series of noisy frequency features Z1, Z2, ..., Z n-1 , Z n , …, Z N , to achieve the purpose of destroying the sample. Among them, after the Nth time step, Z N That is, after performing multi-step noise addition processing on the first audio feature Z0, the noise feature Z is obtained. N .

[0106] For the reverse process, given the precondition In the case of N , by gradually denoising it, we can gradually get Z N-1 , Z N-2 , …, Z n , Z n-1 , …, Z2, Z1, and finally obtain the music prior, that is, the second audio feature.

[0107] In the embodiment of the present disclosure, for the conditional mechanism, the conditional encoder will concatenate the preconditions into a set of feature vectors And feed it into the reverse process of the diffusion model. For example, Figure 4 As shown, the precondition is related to the encoding result of the lyrics text by the text encoder and the total duration of the reference audio, as described below for details.

[0108] In 304, the electronic device maps the second audio feature from the latent space back to the original audio space to obtain a musical work for singing the lyrics text.

[0109] In some embodiments, Figure 4 As shown, the decoder of the VAE in the embodiment of the present disclosure maps the second audio feature from the latent space back to the original audio space, thereby obtaining a musical work for singing the lyrics text. That is, the scheme uses the decoder of the VAE to process the audio compression representation generated by the diffusion model to restore it to a high-fidelity waveform.

[0110] Among them, since the VAE decoder has a strong reconstruction capability, it can effectively map the audio compression representation generated by the diffusion model from the low-dimensional latent space back to the high-dimensional original audio space, generating a waveform signal close to the real audio. This process not only ensures the high fidelity of the generated audio, but also effectively retains the details and rich sound effect information in the reference audio, improving the quality of the generated music. In addition, the generated music includes both vocals and accompaniment, that is, LDM can generate a complete music work containing vocals and accompaniment, which improves the quality of the generated music.

[0111] In summary, the disclosed embodiments provide a new music generation solution, which simplifies the user input in the music generation process. The user only needs to input the lyrics, without the need for the user to input the duration and pitch of each word in the lyrics. Since the user input is simplified, it not only lowers the threshold for users to create music by themselves, but also improves the efficiency of music generation.

[0112] Above Figure 3 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 5 is a flowchart of another music generation method according to an exemplary embodiment. Figure 5 As shown, the music generation method is applied to electronic devices, such as Figure 2 The terminal 201 and the server 202 shown are jointly executed. The music generation method includes the following steps:

[0113] In 501, the electronic device obtains lyrics text to be converted and reference audio, inputs the lyrics text into a text encoder, and inputs the reference audio into an encoder of a VAE.

[0114] This step can refer to the aforementioned 301-302 and will not be repeated here.

[0115] In 502, the electronic device maps the reference audio from the original audio space to the latent space based on the VAE encoder to obtain the first audio feature.

[0116] This step may refer to the aforementioned 302 and will not be described in detail here.

[0117] In 503, the electronic device performs a multi-step noise addition process on the first audio feature based on a diffusion model acting on the latent space to obtain a noise feature; and, under given preconditions, performs a multi-step denoising process on the noise feature based on the diffusion model to obtain a second audio feature.

[0118] In some embodiments, the diffusion model is a U-Net Transformer, which is not limited in the present disclosure.

[0119] Among them, U-Net Transformer is a deep learning model that combines the U-Net architecture and the Transformer attention mechanism. That is, the Transformer architecture is introduced into the structure of U-Net, and the idea of ​​the diffusion model is integrated into the training and generation process of the entire model.

[0120] In addition, similar to other conditional models, different conditions can be added to the LDM based on the conditional encoder to allow the model to generate personalized content. The following introduces the conditional encoder through steps 1-3. For example, Figure 4 As shown, in addition to the two conditional encoders, the text encoder and the integer encoder, a style encoder may also be optionally included. Next, the three conditional encoders are introduced respectively through steps 1-3.

[0121] 1. The electronic device obtains the phoneme features of the lyrics text based on the text encoder.

[0122] For pronunciation tasks such as LTS, converting text into phonemes helps the model learn pronunciation details. Therefore, the text encoder mentioned in this article is a text encoder that targets phonemes. The text encoder obtains the phoneme features of the lyrics text, that is, the corresponding Figure 6 The lyrics analysis process shown.

[0123] In some embodiments, see Figure 4 ,The text encoder includes two steps, one is word-to-phonetic conversion and encoding, and the other is lyrics context extraction.

[0124] In addition, if Figure 6 As shown, before the phonetic conversion is performed, the lyrics text input by the user usually needs to be preprocessed. The preprocessing includes but is not limited to: data cleaning, data normalization, lyrics sentence and word segmentation, etc., which are not limited in the present disclosure. Exemplarily, data cleaning is used to remove irrelevant information in the lyrics text, such as some explanatory text about the lyrics, because this information is usually redundant for lyrics analysis.

[0125] See also Figure 4 and Figure 6, the embodiment of the present disclosure performs a phonetic conversion on the lyrics text based on the G2P module to obtain a phoneme sequence. Among them, in addition to the rule-based dictionary mapping method, the G2P module can also perform a phonetic conversion based on a statistical method or a neural network-based method, which is not limited by the present disclosure. Afterwards, the embodiment of the present disclosure will use an embedder to vectorize the phoneme sequence, realize the mapping of high-dimensional discrete data to a low-dimensional continuous vector space, and obtain the initial phoneme features.

[0126] Among them, the G2P module can be a G2P structure based on a conditional random field or a G2P structure based on BERT, etc., which is not limited in the present disclosure.

[0127] Next, the initial phoneme features are fed into the lyrics context extractor (e.g. Figure 4 The Transformer in the lyric text is then used to obtain the context information of the lyric text through the lyrics context extractor. The context information is used to describe the potential semantic and emotional information in the lyric text. After that, the context information is fused into the initial phoneme feature to obtain the final phoneme feature Phn, which will eventually be passed to the diffusion model.

[0128] In summary, the disclosed embodiment achieves effective encoding of the pronunciation information represented in the lyrics by performing phonetic conversion on the lyrics text input by the user, which not only improves the pronounceability of the lyrics, but also provides the necessary phoneme basis for subsequent music generation. In addition, through the powerful capabilities of Transformer in the field of music, the initial phoneme sequence is analyzed to accurately extract the potential semantic and emotional information in the lyrics text. For example, the self-attention mechanism of Transformer can dynamically focus on different parts of the input sequence, so as to effectively understand the structure of the lyrics and its internal connections, which provides a strong guarantee for subsequent music generation and ensures the quality of music generation. That is, phonemes + Transformer make it possible for the scheme to accurately sing given lyrics.

[0129] 2. The electronic device vectorizes the total duration of the reference audio based on an integer encoder to obtain a duration feature.

[0130] In the disclosed embodiment, the user does not need to input the duration of each word in the lyrics, but only needs to refer to the total duration of the audio. Exemplarily, the total duration of the reference audio is represented by an integer. This integer is encoded by the integer encoder into a single-length embedding Td, thereby obtaining the duration feature.

[0131] 3. The electronic device obtains the music type characteristics of the reference audio based on the style encoder.

[0132] This step is optional. The music type is used to indicate at least one of the music style, vocal timbre or accompaniment intensity of the reference audio. For example, the music type is used to indicate the music style and vocal timbre of the reference audio.

[0133] Exemplarily, the style encoder is an optional style control module. It is a pre-trained module that compresses the reference audio into a single-length embedding S. Through this embedding, not only can the musical style of the generated music piece be controlled, but also the timbre of the accompaniment or the singing can be enhanced.

[0134] In other embodiments, taking the premise that the total duration of the reference audio and the lyrics text are related as an example, the embodiment of the present disclosure will compare the phoneme feature Phn of the lyrics text with the duration feature T d The feature concatenation is performed, and the concatenated features are input as preconditions into the reverse denoising process of the diffusion model. For example, when performing feature concatenation, the phoneme feature Phn and the duration feature T d Connect along the length dimension. At this time, the precondition can be expressed as

[0135] In other embodiments, taking the precondition that the lyrics text, the total duration of the reference audio and the music type as an example, the embodiment of the present disclosure will calculate the phoneme feature Phn, the duration feature T d The feature concatenation is performed with the music type feature S, and the concatenated feature is input as a precondition into the reverse denoising process of the diffusion model. Similarly, when performing feature concatenation, the phoneme feature Phn and the duration feature T d and the music type feature S along the length dimension. At this time, the precondition can be expressed as

[0136] In other embodiments, the disclosed embodiments input the preconditions as keys and values ​​into the reverse denoising process of the diffusion model based on the cross-attention mechanism, that is, incorporate the preconditions into the cross-attention layer of the U-Net Transformer. If the preconditions are input as keys and values ​​into the cross-attention layer of the U-Net Transformer, the U-Net Transformer will calculate the attention weights based on these inputs. In the reverse denoising process, these attention weights are used to indicate the degree of attention of the decoder (corresponding to the reverse denoising process) to different positions of the encoder (corresponding to the forward denoising process), demonstrating the model's stronger control over the generation process.

[0137] In summary, the disclosed embodiments support adding different conditions in LDM to allow the model to generate personalized content. Among them, the added conditions can be cross-modal, for example, the cross-modal conditions are obtained by feature splicing of the phoneme features of the lyrics text and the duration features of the reference audio. This conditional mechanism does not require the user to perform complex input, such as the user does not need to enter the duration and pitch of each word in the lyrics text, and the music generation efficiency is high.

[0138] In addition, the precondition is related not only to the total duration of the lyrics text and the reference audio, but also to the music type of the reference audio. This enables LDM to generate music works with multiple timbres and styles and accompaniment, improving the quality of music generation. In addition, LDM can not only control the vocal timbre and music style of the generated music works through the reference audio, but also enhance the accompaniment, etc., demonstrating the model's stronger control over the generation process.

[0139] In addition, when the precondition includes the music type of the reference audio, the cross-modal condition is obtained by concatenating the phoneme features of the lyrics text, the duration features of the reference audio, and the music type features. This conditional mechanism also does not require the user to perform complex input, such as the duration and pitch of each word in the lyrics text, and the music generation efficiency is high.

[0140] In 504, the electronic device maps the second audio feature from the latent space back to the original audio space based on the VAE decoder to obtain a musical work for singing the lyrics text.

[0141] Among them, the diffusion model can achieve various controls by introducing guidance in the reverse denoising process. Among them, classifier-free guidance is a widely used guidance method. By applying different guidance scales, the generated music samples can achieve a balance between diversity and quality. That is, the guidance scale plays a key role in balancing the impact of lyrics on the final output. Accordingly, the method provided by the embodiment of the present disclosure also includes:

[0142] Adjust the value of the guide scale. In the reverse denoising process of the diffusion model, the electronic device controls the correlation between the second audio feature and the lyrics text through the guide scale, thereby achieving control over the finally generated music work.

[0143] In summary, the disclosed embodiment achieves the generation of harmonious and high-fidelity music with vocals and accompaniment based on LDM. On the one hand, the solution simplifies the user input in the music generation process. The user only needs to input the lyrics, without the need for the user to input information such as the duration and pitch of each word in the lyrics. Since the user input is simplified, it not only lowers the threshold for users to create music by themselves, but also improves the efficiency of music generation.

[0144] In addition, the scheme also guides music generation through reference audio, which enables the scheme to control the musical style, vocal timbre or accompaniment intensity of the generated music works, thereby improving the quality of the generated music works.

[0145] In addition, LDM uses an end-to-end training method to reduce the requirements for training data. During the training phase, only lyrics text and audio clips are needed to complete the training. There is no need for manual professional annotation at the score level, saving labor costs.

[0146] Combined with the above description, it can be seen that the embodiment of the present disclosure implements an end-to-end low-latency music generation technology, and a music work including both vocals and accompaniment can be generated by simply inputting lyrics text by the user. That is, by referring to audio and lyrics text, the embodiment of the present disclosure implements flexible and controllable music generation capabilities.

[0147] This end-to-end music generation method not only significantly simplifies the music generation process and improves the efficiency of music generation, but also can create high-quality music works while maintaining the rhythm of the lyrics. For example, this solution can achieve a richer and more layered music generation effect, which can effectively improve the overall expressiveness and artistic appeal of the song.

[0148] Figure 7 is a block diagram of a music generating device according to an exemplary embodiment. Figure 7 , the device comprises:

[0149] The acquisition module 701 is configured to acquire lyrics text to be converted and reference audio;

[0150] A first encoding module 702 is configured to map the reference audio from an original audio space to a latent space to obtain a first audio feature;

[0151] The generating module 703 is configured to perform a multi-step noise addition process on the first audio feature based on the diffusion model acting on the latent space to obtain a noise feature; and, under a given precondition, perform a multi-step denoising process on the noise feature based on the diffusion model to obtain a second audio feature; wherein the precondition is related to the total duration of the reference audio and the lyrics text;

[0152] The first decoding module 704 is configured to map the second audio feature from the latent space back to the original audio space to obtain a musical work for singing the lyrics text.

[0153] In summary, the disclosed embodiments provide a new music generation solution, which simplifies the user input in the music generation process. The user only needs to input the lyrics, without the need for the user to input information such as the duration and pitch of each word in the lyrics. Since the user input is simplified, it not only lowers the threshold for users to create music by themselves, but also improves the efficiency of music generation. In addition, the solution also guides music generation through reference audio, which enables the solution to accurately sing the given lyrics, thereby ensuring the quality of music generation.

[0154] In some embodiments, the apparatus further comprises:

[0155] A second encoding module is configured to obtain the phoneme features of the lyrics text;

[0156] A third encoding module is configured to vectorize the total duration of the reference audio to obtain a duration feature;

[0157] The processing module is configured to perform feature concatenation on the phoneme feature and the duration feature, and input the concatenated feature as the precondition into the reverse denoising process of the diffusion model.

[0158] In some other embodiments, the second encoding module is configured to:

[0159] Performing phonetic conversion on the lyrics text to obtain a phoneme sequence;

[0160] Vectorizing the phoneme sequence to obtain initial phoneme features;

[0161] Based on the initial phoneme features, context information of the lyrics text is obtained; wherein the context information is used to describe potential semantic and emotional information in the lyrics text;

[0162] The context information is integrated into the initial phoneme feature to obtain a final phoneme feature.

[0163] In other embodiments, the precondition is related to the lyrics text, the total duration of the reference audio and the music type; wherein the music type is used to indicate at least one of the musical style, vocal timbre or accompaniment intensity of the reference audio.

[0164] In some other embodiments, the device further comprises:

[0165] A fourth encoding module is configured to obtain a music type feature of the reference audio;

[0166] The processing module is configured as follows:

[0167] The phoneme feature, the duration feature and the music type feature are concatenated, and the concatenated features are input as the preconditions into the reverse denoising process of the diffusion model.

[0168] In some other embodiments, the processing module is configured to:

[0169] Based on the cross-attention mechanism, the preconditions are input as keys and values ​​into the reverse denoising process of the diffusion model.

[0170] In some other embodiments, the reference audio is an audio signal having a specific sampling rate;

[0171] The first encoding module is configured as follows:

[0172] An encoder based on a variational autoencoder maps the audio signal from the original audio space to the latent space to obtain the first audio feature.

[0173] In some other embodiments, the first decoding module is configured to:

[0174] Based on the decoder of the variational autoencoder, the second audio feature is mapped from the latent space back to the original audio space to obtain a musical work for singing the lyrics text; wherein the musical work includes vocals and accompaniment.

[0175] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, which will not be described in detail here.

[0176] It should be noted that the music generation device provided in the above embodiment only uses the division of the above functional units as an example when performing music generation. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the music generation device provided in the above embodiment and the music generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0177] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0178] In some embodiments, an electronic device is further provided, the electronic device comprising:

[0179] one or more processors;

[0180] a memory for storing program code executable by the processor;

[0181] Wherein, the processor is configured to execute the program code to implement the above-mentioned music generation method.

[0182] In response to the electronic device being provided as a server, Figure 8 800 is a block diagram of a server 800 according to an exemplary embodiment. The server 800 may have relatively large differences due to different configurations or performances, and includes one or more processors (Central Processing Units, CPU) 801 and one or more memories 802, wherein the memory 802 stores at least one program code, and the at least one program code is loaded and executed by the processor 801 to implement the music generation method provided by the above-mentioned various method embodiments. Of course, the server also has components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 800 also includes other components for realizing device functions, which will not be repeated here.

[0183] In some embodiments, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the instructions are executed by a processor of an electronic device to implement the above-mentioned music generation method. In other embodiments, the computer-readable storage medium is a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0184] In some embodiments, a computer program product is also provided, including a computer program, which, when executed by a processor of an electronic device, implements the above-mentioned music generation method.

[0185] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0186] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A music generation method, characterized in that: The method comprises: Get the lyrics text and reference audio to be converted; Mapping the reference audio from the original audio space to the latent space to obtain a first audio feature; Based on the diffusion model acting on the latent space, performing a multi-step noise addition process on the first audio feature to obtain a noise feature; Under a given precondition, performing a multi-step denoising process on the noise feature based on the diffusion model to obtain a second audio feature; wherein the precondition is related to the total duration of the reference audio and the lyrics text; The second audio feature is mapped from the latent space back to the original audio space to obtain a musical work for singing the lyrics text.

2. The music generation method according to claim 1, characterized in that: The method further comprises: Acquire the phoneme features of the lyrics text; Vectorizing the total duration of the reference audio to obtain a duration feature; The phoneme feature and the duration feature are concatenated, and the concatenated feature is input as the precondition into the reverse denoising process of the diffusion model.

3. The music generation method according to claim 2, characterized in that: The step of obtaining the phoneme features of the lyrics text includes: Performing phonetic conversion on the lyrics text to obtain a phoneme sequence; Vectorizing the phoneme sequence to obtain initial phoneme features; Based on the initial phoneme features, context information of the lyrics text is obtained; wherein the context information is used to describe potential semantic and emotional information in the lyrics text; The context information is integrated into the initial phoneme feature to obtain a final phoneme feature.

4. The music generation method according to claim 2, characterized in that: The precondition is related to the lyrics text, the total duration of the reference audio and the music type; The music type is used to indicate at least one of the music style, vocal timbre or accompaniment intensity of the reference audio.

5. The music generation method according to claim 4, characterized in that: The method further comprises: Acquire music type features of the reference audio; The step of concatenating the phoneme feature and the duration feature, and inputting the concatenated feature as the precondition into the reverse denoising process of the diffusion model includes: The phoneme feature, the duration feature and the music type feature are concatenated, and the concatenated features are input as the preconditions into the reverse denoising process of the diffusion model.

6. The music generation method according to claim 2 or 5, characterized in that: The step of inputting the spliced ​​features as the precondition into the reverse denoising process of the diffusion model includes: Based on the cross-attention mechanism, the preconditions are input as keys and values ​​into the reverse denoising process of the diffusion model.

7. The music generation method according to claim 1, characterized in that: The reference audio is an audio signal with a specific sampling rate; Mapping the reference audio from the original audio space to the latent space to obtain the first audio feature includes: An encoder based on a variational autoencoder maps the audio signal from the original audio space to the latent space to obtain the first audio feature.

8. The music generating method according to claim 1 or 7, characterized in that: The step of mapping the second audio feature from the latent space back to the original audio space to obtain a musical work for singing the lyrics text comprises: Based on the decoder of the variational autoencoder, the second audio feature is mapped from the latent space back to the original audio space to obtain a musical work for singing the lyrics text; wherein the musical work includes vocals and accompaniment.

9. A music generating device, characterized in that: The device comprises: An acquisition module is configured to acquire lyrics text and reference audio to be converted; A first encoding module is configured to map the reference audio from an original audio space to a latent space to obtain a first audio feature; A generating module is configured to perform a multi-step noise addition process on the first audio feature based on a diffusion model acting on the latent space to obtain a noise feature; and, under a given precondition, perform a multi-step denoising process on the noise feature based on the diffusion model to obtain a second audio feature; wherein the precondition is related to the total duration of the reference audio and the lyrics text; The first decoding module is configured to map the second audio feature from the latent space back to the original audio space to obtain a musical work for singing the lyrics text.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the music generation method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the music generating method as claimed in any one of claims 1 to 8.

12. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor of an electronic device, implements the music generating method as claimed in any one of claims 1 to 8.

Images