An end-to-end song generation method and system based on user colloquial language expression

Abstract

The application discloses an end-to-end song generation method and system based on user colloquial language expression, which comprises the following steps: obtaining the colloquial language expression text input by a user and taking the text as a description text prompt word; using a large language model to generate a song lyric sequence with a music section structure according to the description text prompt word; using a generation model to generate a latent probability distribution representing music by carrying out denoising processing on the song lyric sequence with the music section structure, and searching in an audio data space of a VAE model based on the latent probability distribution to obtain an audio distribution corresponding to the song lyric sequence, wherein each point in the audio distribution represents a piece of audio; converting the audio distribution into a frequency spectrum graph, and then converting the frequency spectrum graph into audio data of a song; thus, a song is created based on a fully understood colloquial expression, and the colloquial language expression of the user is matched with the created song.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence music generation technology, specifically relating to an end-to-end song generation method and system based on users' colloquial expressions. Background Technology

[0002] Text-to-song generation aims to create songs from descriptive text. Previous research has typically focused on specific aspects of song creation, such as text-to-music, lyrics-to-melody, and sheet music-to-song, but has not covered the entire song creation process. In text-to-music generation, models such as MusicLM and MusicGen utilize a single-stage Transformer language model with a quantized audio codec for text-to-music generation, employing an efficient codebook interleaving pattern and large language and diffusion models to generate high-quality audio. For lyrics-to-melody generation, SongMASS, TeleMelody, and SongGLM take lyrics as input and output a melody, ensuring a correlation between lyrics and melody. For sheet music-to-song generation, Melodist employs a two-stage approach, including generating a vocal from a given sheet music and generating an accompaniment based on the vocal and natural language cues. Text-to-song generation platforms, such as Suno, SkyMusic, and SongR, have received considerable attention for their capabilities. However, these platforms typically do not provide access to their datasets and methods, which may limit reproducibility and pose challenges for future research. They may also encounter problems such as poor understanding of the input text description and a lack of structure in the generated songs.

[0003] In the field of music generation, although many datasets have been proposed to date, including those derived from automatic annotation using Music Information Retrieval (MIR) algorithms or Large Language Models (LLMs) as well as manually annotated datasets, automatically annotated datasets employ existing MIR algorithms to extract musical attributes from symbolic music or audio music. The extracted attributes are then incorporated into complete descriptive text, or treated as descriptive tags. MSD collects a large amount of music data, including audio, MIDI, and tags obtained through the Echo Nest Analyze API (MIR toolkit). POP909 provides a dataset containing audio, melody scores, and other musical attributes such as key signatures. MuseCoco and Mustango extract features from raw audio and then integrate these features into descriptive text using ChatGPT. MuLaMCap in Noise2Music uses a large language model to generate a set of music descriptive texts and then uses MuLan (a text-music embedding model) to match these texts with the music audio in the dataset. Nevertheless, a significant semantic gap remains between datasets obtained through automatic annotation and complex human descriptions, which reduces the accuracy of the dataset and limits the performance of the model.

[0004] Manually labeled datasets collect descriptions or tags from music websites, or include data annotated by professional musicians. Hooktheory is a music website where users upload audio with annotations such as melody, chords, and beat. MTG and Mousai use corresponding tags from music websites as description tags, while ERNIEMusic uses music reviews as music descriptions and builds its dataset accordingly. MusicLM provides a music description dataset, MusicCaps, annotated by professional musicians. However, current manually labeled datasets are limited by expert annotations and a limited range of descriptions, which differs significantly from descriptions provided by the general public. Furthermore, existing manually labeled datasets typically have each record annotated by only one person, which can also lead to inaccuracies due to human error or bias. Summary of the Invention

[0005] In view of the above, the purpose of this invention is to provide an end-to-end song generation method and system based on users' colloquial expressions, which creates songs based on fully understood colloquial expressions and matches users' colloquial expressions with the created songs.

[0006] To achieve the above-mentioned objectives, an embodiment provides an end-to-end song generation system based on users' colloquial expressions, comprising:

[0007] The acquisition module is used to acquire the colloquial expression text input by the user and use it as descriptive text prompts;

[0008] The lyrics generation module is used to generate a sequence of lyrics with a musical phrase structure based on descriptive text prompts using a large language model.

[0009] The audio generation module is used to generate a potential probability distribution representing music by denoising the lyrics sequence with musical segment structure using a generative model. Based on the potential probability distribution, it searches in the audio data space of the VAE model to obtain the audio distribution corresponding to the lyrics sequence. Each point in the audio distribution represents an audio segment. After converting the audio distribution into a spectrogram, the spectrogram is converted into the audio data of the song.

[0010] Preferably, the lyrics sequence with musical phrase structure includes a sequence of lyrics, in which the lyrics are tagged, wherein the tags include timbre, instrument, sound effect, style, and musical phrase structure tags, and the musical phrase structure tags include intro, verse, pre-chorus, chorus, bridge or interlude, and conclusion;

[0011] Ignoring the lyrics, the musical phrase structure tags are arranged by separating them with separator tags to form the musical phrase structure.

[0012] Preferably, the step of using a generative model to generate a potential probability distribution representing music by denoising a lyric sequence with a musical phrase structure includes:

[0013] Multiple consecutive time windows are initialized. For each time window, at least a portion of the lyrics sequence matching the length of the time window is extracted from the lyrics sequence with musical phrase structure as the current input lyrics sequence. Based on the current input lyrics sequence and the generated audio data corresponding to the previous time window, continuous denoising is performed through the generative model to generate the potential probability distribution representing the music corresponding to the current time window.

[0014] Preferably, the step of continuously denoising the generated audio data based on the current input lyrics sequence and the previous time window using a generative model to generate a potential probability distribution representing the music corresponding to the current time window includes:

[0015] For the current time window, the generated audio data and descriptive text prompts corresponding to the previous time window are processed by CLAP to form an embedding vector, which is then concatenated with the current noise to form a concatenated vector and input into the generation model. At the same time, the current input lyrics sequence is also input into the generation model. In the generation model, denoising is performed at time step t to obtain the denoising result corresponding to time step t. This denoising result is used as the current noise at time step t-1.

[0016] Where t takes the value [0,T]. When t is T, the current noise corresponding to time step T is the initial random noise, and the generated audio data corresponding to the previous time window corresponding to time step T is an empty vector. When t is 0, the denoising result corresponding to time step 0 is considered to be a noise-free result, which is used to represent the potential probability distribution of the music.

[0017] Preferably, the generative model includes a DIT sub-model for noise generation based on splicing vectors and lyrics sequences, and a DDPM sub-model for denoising calculation based on generated noise;

[0018] In the DIT model, the input concatenated vector, as a noise latent vector, is processed by block partitioning and the first layer of normalization. Then, multi-head self-attention processing is used to focus on the first important feature. The first important feature and the input vector of multi-head self-attention processing are concatenated through the first residual connection and then processed by the second layer of normalization to obtain the first vector corresponding to the input concatenated vector after processing.

[0019] The current input lyrics sequence is processed to obtain an embedding vector, which is then subjected to multi-head cross-attention processing with the first vector to focus on the second important feature. The second important feature is then concatenated with the first residual and concatenated with the second residual to obtain the second vector.

[0020] After the second vector is normalized and fed forward by the third layer, it is concatenated with the second vector. The concatenated result is then normalized by the fourth layer, linearly mapped and shaped to obtain the noise distribution.

[0021] Preferably, the DIT sub-model is replaced by the SIT sub-model. In the SIT sub-model, during the shape adjustment process, random sampling is performed by selecting a sampler, and a noise distribution is generated based on the random sampling results.

[0022] In the interpolation framework of the DIT sub-model, a mapping relationship is found between the latent probability distribution of the generated model output and the high-dimensional normal distribution space of VAE by selecting interpolation parameters. Based on the mapping relationship, the audio distribution corresponding to the lyrics sequence is searched and determined.

[0023] Preferably, the VAE model includes a VAE encoder and a VAE decoder, which are obtained based on self-supervised training. The VAE encoder is used to compress and encode the input audio data into an audio distribution in the audio data space, and the VAE decoder is used to decode the audio distribution in the audio data space to generate a spectrogram.

[0024] The audio distribution of the audio data space corresponding to the trained VAE encoder is sampled as sample data for training the generative model;

[0025] The trained VAE decoder is used to convert the audio distribution into a spectrogram when applied.

[0026] Preferably, the generative model needs to be optimized before it is applied. During parameter optimization, the audio distribution of the audio data space of the VAE model is sampled to obtain sample data, and the audio data is used as labels to perform supervised training of the generative model based on the sample data under the labels, so as to optimize the parameters of the generative model.

[0027] Preferably, the large language model is fine-tuned before being applied. The text data used in the fine-tuning includes: music annotations annotated by professionals and enthusiasts, wherein the music annotations use professional descriptions and colloquial descriptions, and the professional descriptions and colloquial descriptions form data pairs. The large language model is fine-tuned using data pairs so that it can be converted into a lyric sequence with musical phrase structure based on the colloquial description text.

[0028] To achieve the above-mentioned objectives, embodiments of the present invention also provide an end-to-end song generation method based on user colloquial expressions, employing the aforementioned end-to-end song generation system. The method includes the following steps:

[0029] The acquisition module is used to obtain colloquial expressions input by the user and use them as descriptive text prompts;

[0030] The lyrics generation module uses a large language model to generate a lyric sequence with a musical phrase structure based on descriptive text prompts.

[0031] The music generation module generates audio data based on lyric sequences. Specifically, within each time window, a generative model is used to denoise the lyric sequence with a musical segment structure to generate a potential probability distribution representing the music. Based on the potential probability distribution, a high-dimensional audio distribution corresponding to the lyric sequence is obtained by searching in the audio data space of the VAE model. Each point in the audio distribution represents a segment of audio. After converting the high-dimensional audio distribution into a spectrogram, the spectrogram is converted into the audio data of the song. The audio data generated in sequence from multiple time windows constitute the final audio data.

[0032] To achieve the above-mentioned objectives, embodiments of the present invention also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the above-mentioned end-to-end song generation method based on user colloquial expressions.

[0033] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0034] This invention employs a large language model to convert colloquial expressions into lyric sequences with musical phrase structures. This approach allows for a full understanding of spoken language, resulting in more universally applicable song compositions and a thorough capture of the song's musical structure, including phrase structure and rhyme scheme. Building upon this foundation, a generative model generates the latent probability distribution of the music, and a VAE model searches for and clusters high-dimensional audio distributions. By converting these high-dimensional audio distributions into spectrograms and then into the song's audio data, the automatically generated songs become richer and more closely matched to colloquial expressions. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the structure of an end-to-end song generation system based on users' colloquial expressions, provided in the embodiment.

[0037] Figure 2 This is a flowchart illustrating the process of generating songs using the application song generation system provided in this embodiment.

[0038] Figure 3 This is a flowchart illustrating the potential probability distribution of music generation based on a generative model within a single time window, as provided in the embodiment.

[0039] Figure 4 This is a schematic diagram of the structure that generates noise in the DIT sub-model provided in the embodiment;

[0040] Figure 5 This is a flowchart of an end-to-end song generation method based on user colloquial expressions provided in the embodiment;

[0041] Figure 6 This is a schematic diagram of the computing device provided in the embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0043] like Figure 1 As shown, the embodiment provides an end-to-end song generation system based on users' colloquial expressions, including an acquisition module 11, a lyrics generation module 12, and an audio generation module 13.

[0044] In this embodiment, the acquisition module 11 is used to acquire colloquial expressions input by the user. Colloquial expressions are not conducive to the model's understanding and audio bio-generation. Therefore, it is necessary to use colloquial expressions as descriptive text prompts and convert them into a lyric sequence that the model can understand.

[0045] In this embodiment, the lyrics generation module 12 is used to generate a lyric sequence with a musical phrase structure based on descriptive text prompts using a large language model. The lyric sequence with a musical phrase structure includes a sequence of lyrics, each lyric labeled with tags including timbre, instrument, sound effect, style, and musical phrase structure tags, where the musical phrase structure tags include an intro. <intro>verse <verse>, pre-chorus <pre-chorus>,refrain <chorus>Bridge or interlude<Bridgeor Interlude> and conclusion <outro>These tags constitute the rhythm of the entire song. Ignoring the lyrics, the musical phrase structure tags are separated and arranged by separator tags. The sequence of structural descriptions of the song is named musical phrase structure.

[0046] Among them, the prelude <intro>It's the opening intro of a song. It's usually a piece of instrumental music to grab the listener's attention. Another popular design is to start the song with vocals, followed by an instrumental intro. Verse <verse>This refers to a more narrative section after the intro, which generally sounds somewhat bland. (Prelude to the chorus) <pre-chorus>Its function is to transition the mood of the verse to the chorus. Many songs don't specifically design a pre-chorus section; the latter part of the verse serves the same purpose. Chorus <chorus>This refers to the climax of a song, usually following the chorus or verse. The chorus is typically repeated multiple times. It can also be a bridge or interlude.<Bridge or Interlude> This refers to the section that is generally used to connect the verse and chorus, both of which are usually instrumental.

[0047] The large language model chosen is Qwen's 14B model. Based on official recommendations and subsequent practical results, the quantized dialogue model is best suited for quickly validating the lyrics generation requirements of this invention. Therefore, the final base model is Qwen-14B-Chat-Int4. Of course, other large language models can also be used. The large language model needs fine-tuning before application.

[0048] The text data used for fine-tuning included music annotations tagged by professionals and enthusiasts. These annotations employed both professional and colloquial descriptions, forming data pairs to ensure diversity of perspectives. This improved the accuracy of the music annotations and their correspondence with common semantics. The large language model was then fine-tuned using these data pairs to convert colloquial text into song structure & lyric sequences. Specifically, the tags used for annotation included timbre, instrument, sound effects, style, and song structure.

[0049] In this embodiment, the audio generation module 13 is used to generate audio data based on the lyrics sequence. Specifically, the generation model performs noise reduction processing on the lyrics sequence with musical segment structure to generate a potential probability distribution representing the music. Based on the potential probability distribution, the audio distribution corresponding to the lyrics sequence is obtained by searching in the audio data space of the VAE model. Each point in the audio distribution represents a segment of audio. After converting the audio distribution into a spectrogram, the spectrogram is converted into the audio data of the song.

[0050] In this process, when using a generative model to denoise lyrics sequences with musical phrase structures to generate a potential probability distribution representing the music, denoising is performed according to time windows. Specifically, multiple consecutive time windows are first initialized, with the length of each time window corresponding to the length of the generated audio data. This length can be defined by the user, and the lengths of the multiple consecutive time windows can be the same or different, without any constraints. Within each time window, after generating the potential probability distribution based on the generative model, the potential probability distribution generated in each time window is then used to generate the corresponding audio data through a subsequent transformation process.

[0051] Specifically, for each time window, at least a portion of the lyrics sequence matching the length of the time window is extracted from the lyrics sequence with musical phrase structure as the current input lyrics sequence. Based on the current input lyrics sequence and the generated audio data corresponding to the previous time window, continuous denoising is performed through the generative model to generate the latent probability distribution (Latent Sample) representing the music corresponding to the current time window.

[0052] like Figure 2 and Figure 3 As shown, based on the current input lyrics sequence and the generated audio data corresponding to the previous time window, continuous denoising is performed through the generative model to generate the potential probability distribution representing the music corresponding to the current time window, including:

[0053] Regarding the current time window, such as Figure 2 As shown, the text description prompts are processed using CLAP to form description embeddings. This process ensures that the input text description prompts are converted into a dense vector representation, capturing the semantic nuances required for music generation. Simultaneously, the generated audio data corresponding to the previous time window is also processed using CLAP to form audio embeddings. The description embeddings and audio embeddings are concatenated, and then concatenated with the current noise to form a concatenated vector, which is then input into the generation model. Simultaneously, the current input lyrics sequence is converted into lyrics embeddings and also input into the generation model. Figure 3 As shown, denoising is performed at time step t in the generative model to obtain the denoising result corresponding to time step t, and this denoising result is used as the current noise at time step t-1.

[0054] In generative models, the denoising process is multi-time-step, with t taking the value [0, T]. After T+1 time steps of denoising, the final latent probability distribution is obtained. It's important to note that when t is T, the noise corresponding to the first time step T is the initial random noise, and the generated audio data corresponding to the previous time window is an empty vector, i.e., a vector composed of 0s, with the same dimension as the audio data generated in other time steps. When t is 0, the denoising result corresponding to the last time step 0 is considered a noise-free result, and this denoising result represents the latent probability distribution of the music obtained from the entire denoising process within the current time window.

[0055] The generation model in the embodiment includes a DIT sub-model and a DDPM sub-model. At each time step of a single time window, the model is processed by the DIT sub-model and the DDPM sub-model. The DIT sub-model is used to generate noise based on the splicing vector and the lyrics embedding vector corresponding to the lyrics sequence. The DDPM sub-model is used to denoise the denoising result of the previous time step by using the noise generated by the DIT sub-model, that is, to perform subtraction calculation to obtain the denoising result of the current time step.

[0056] like Figure 4 As shown, in the DIM model, the input concatenated vector, as a noised latent, is processed by patchify and the first layer normalization. Tokens are generated by dividing the vector into multiple small blocks. Then, multi-head self-attention is used to focus on different parts of the noised latent to capture complex dependencies and patterns, obtaining the first important feature. The first important feature and the input vector of the multi-head self-attention are concatenated through the first residual connection and then processed by the second layer normalization to obtain the first vector corresponding to the processed input concatenated vector.

[0057] Meanwhile, the current input lyrics sequence carries labels related to timbre, instruments, sound effects, style, and musical structure. Since the length of the lyrics vector corresponding to the labeled current input lyrics sequence is variable, it is input into the DIT sub-model for multi-head cross-attention learning. This mechanism enables the DIT sub-model to learn lyrics of variable length in conjunction with audio. Specifically, after the current input lyrics sequence is embedded to obtain a lyrics embedding vector, it undergoes multi-head cross-attention processing with the first vector to focus on the second important feature. The second important feature is then concatenated with the first residual and concatenated with the second residual to obtain the second vector.

[0058] The second vector is processed by the third layer of normalization and pointwise feedforward, and then concatenated with the second vector. The concatenated result is then processed by the fourth layer of normalization, linear mapping, and reshape adjustment to obtain the noise distribution.

[0059] It should also be noted that in the generative model, the DIT sub-model can be replaced by the SIT model. The SIT model is a continuous, velocity-predictable, linearly schedulable model with SDE sampling. It integrates a flexible interpolation framework on the basis of the DIT model architecture, allowing two data distributions to be connected in a more flexible way than the standard diffusion model. Therefore, compared with DiT, it has more diverse and flexible choices in interpolation (selection of interpolation parameters αt, σt) and sampler (ODE or SDE).

[0060] Specifically, the latent probability distribution generated by the original DIT sub-model is searched in the audio data space of the VAE model to obtain the audio distribution corresponding to the lyrics sequence. In the interpolation framework of the SIT sub-model, by selecting the interpolation parameters, a mapping relationship can be found more efficiently between the disordered latent probability distribution and the normal high-dimensional distribution space of VAE. Based on the mapping relationship, the audio distribution corresponding to the lyrics sequence is searched and determined, thereby improving the inference speed.

[0061] Furthermore, compared to the deterministic shape adjustment (Resample) step of the DiT sub-model, the SiT sub-model introduces a random sampling process in the Resample step through the selection of samplers to generate a potential probability distribution. This allows the model's diffusion coefficient to be independent of inference and training, reducing the plagiarism rate.

[0062] The essence of song generation lies in simulating the probability distribution p of the original sound. However, this p can be extremely complex and cannot be calculated precisely. Therefore, we try to start with a very simple distribution q, such as the standard normal distribution N(0,1). If we can find a functional F: q->p, then taking a point in q corresponds to a song in p. Directly transforming the unknown distribution p into the known standard normal distribution N(0,1) is too difficult for the model, so multiple modules are used for implementation.

[0063] The desired audio space is similar to a normal distribution. Therefore, a spatial transformation is performed to convert the irregular space into a regular normal distribution. This requires transforming the complexly distributed music dataset into a normal high-dimensional distribution through a trained VAE model. The VAE contains all the songs from the training data, with each song representing a point. These points are arranged around a normal distribution, which determines the upper limit of the diversity of generated songs.

[0064] Therefore, a variational autoencoder model, namely the VAE model, is also pre-trained. This VAE model includes a VAE encoder and a VAE decoder, which are obtained based on self-supervised training. The VAE encoder is used to compress and encode the input audio data into an audio distribution in the audio data space, and the VAE decoder is used to decode the audio distribution in the audio data space to generate a spectrogram. The audio distribution in the audio data space corresponding to the trained VAE encoder is sampled as sample data for training the model. The trained VAE decoder is used to convert the audio distribution into a spectrogram when applied.

[0065] In this embodiment, samples are constructed based on the audio distribution of the audio data space of the trained VAE model to optimize the parameters of the generative model. Specifically, only the parameters of the DIT sub-model are optimized. The optimization process is as follows: sample data is obtained by sampling the audio distribution of the audio data space of the VAE model, and the audio data is used as labels to perform supervised training of the generative model based on the sample data under the labels in order to optimize the parameters of the generative model.

[0066] Since the training sample data of the generative model is sampled from the audio data space obtained by the VAE model through compression of the audio dataset using the VAE encoder, the training sample data of the generative model is essentially a subset of the audio data space corresponding to the VAE model. Therefore, the latent probability distribution representing music obtained using the trained generative model is also a subset of the audio data space, with each point in the latent probability distribution representing a song. The desired audio space is similar to a normal distribution; therefore, an audio space transformation is needed to convert the irregular latent probability distribution into a regular spatial distribution. The audio data space of the trained VAE model is considered a dictionary. Within each time window, the latent probability distribution is used as an index to search the dictionary of the VAE model to obtain the audio distribution corresponding to the lyrics sequence. This audio distribution satisfies a normal distribution, and each point in the audio distribution represents a segment of audio.

[0067] like Figure 2 As shown, after obtaining the audio distribution within each time window, the VAE decoder transforms the normally distributed audio distribution into a specific scatter distribution. Each point in the scatter distribution also represents a segment of audio. Viewed within the entire audio data space, the shift from a normal distribution to a scatter distribution groups similar songs together, which can then be used to extract song information. The scatter distribution is then visualized and converted into a spectrogram in the form of a heatmap. This spectrogram can be a Mel spectrogram, where each bright spot corresponds to a small segment of audio within the song.

[0068] A spectrogram is merely a visual representation of audio, not actual sound. Therefore, it needs to be converted into the audio data of a song. It's well known that converting sound to a heatmap is easy, but the reverse is not. This is because heatmaps only show volume levels (high or low), not the relationships between notes (phase). To address this issue, this invention trains a conversion model, HiFi-GAN, designed to convert a given heatmap into sound-based audio. Then, within each time window, the trained HiFi-GAN model is used to convert the heatmap-based spectrogram into the song's audio data, which serves as the input for the next time window.

[0069] The aforementioned end-to-end song generation system enables the generation of song audio from colloquial descriptions, aiming to create songs based on given colloquial descriptive text. Unlike previous work that focused on specific aspects of text-to-song generation, this invention's system encompasses the entire creative process from abstract description to complete song production, possessing unique characteristics:

[0070] (1) Full understanding of spoken language: Since people usually use spoken language rather than professional descriptions, supporting both spoken and professional descriptions is crucial. However, spoken descriptions are more difficult to understand and process. The system of this invention fine-tunes a large language model through a self-constructed labeled dataset, enabling the large language model to transform spoken description text into a lyric sequence with a musical phrase structure, thus achieving full understanding of spoken descriptions and high-quality generation of lyric sequences;

[0071] (2) Fully capture the musical structure of the song: The musical structure of the song mainly includes the musical phrase structure and the rhyme structure. By generating the song with musical structure through the generation model, VAE model and HiFiGAN transformation model in the audio generation module, the coherence and overall quality are ensured.

[0072] like Figure 5 As shown, the embodiment also provides an end-to-end song generation method based on users' colloquial expressions. This method uses the aforementioned end-to-end song generation system and includes the following steps:

[0073] S1, use the acquisition module to obtain the colloquial expression text input by the user and use it as descriptive text prompt words;

[0074] S2, using the lyrics generation module, a large language model is employed to generate a lyric sequence with a musical phrase structure based on descriptive text prompts;

[0075] S3 uses the audio generation module to generate audio data based on the lyrics sequence.

[0076] Specifically, within each time window, a generative model is used to denoise the lyrics sequence with musical phrase structure to generate a potential probability distribution representing the music. Based on the potential probability distribution, a search is performed in the audio data space of the VAE model to obtain the high-dimensional audio distribution corresponding to the lyrics sequence. Each point in the audio distribution represents a segment of audio. After converting the high-dimensional audio distribution into a spectrogram, the spectrogram is converted into the audio data of the song. The audio data generated in sequence from multiple time windows constitute the final audio data.

[0077] The embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it implements the aforementioned end-to-end song generation method based on user-generated colloquial expressions, specifically including the following steps:

[0078] S1, use the acquisition module to obtain the colloquial expression text input by the user and use it as descriptive text prompt words;

[0079] S2, using the lyrics generation module, a large language model is employed to generate a lyric sequence with a musical phrase structure based on descriptive text prompts;

[0080] S3 uses the audio generation module to generate audio data based on the lyrics sequence.

[0081] like Figure 6 As shown, the computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the end-to-end song generation method based on user colloquial expressions described in S1-S3 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0082] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.< / chorus> < / verse> < / intro> < / outro> < / chorus> < / verse> < / intro>

Claims

1. An end-to-end song generation system based on user colloquial language expression, characterized in that, include: The acquisition module is used to acquire the colloquial expression text input by the user and use it as descriptive text prompts; The lyrics generation module is used to generate a sequence of lyrics with a musical phrase structure based on descriptive text prompts using a large language model. The audio generation module is used to generate a potential probability distribution representing music by denoising a lyric sequence with a musical phrase structure using a generative model. This includes: initializing multiple consecutive time windows; for each time window, extracting at least a portion of the lyric sequence matching the length of the time window from the lyric sequence with a musical phrase structure as the current input lyric sequence; and continuously denoising the current input lyric sequence and the generated audio data corresponding to the previous time window using the generative model to generate the potential probability distribution representing music corresponding to the current time window. Specifically, this includes: for the current time window, combining the generated audio data and descriptive text from the previous time window... The prompt word is processed by CLAP to form an embedding vector, which is then concatenated with the current noise to form a concatenated vector and input into the generation model. At the same time, the current input lyrics sequence is also input into the generation model. In the generation model, denoising is performed at time step t to obtain the denoising result corresponding to time step t. This denoising result is used as the current noise at time step t-1. Here, t takes the value [0,T]. When t is T, the current noise corresponding to time step T is the initial random noise, and the generated audio data corresponding to the previous time window at time step T is an empty vector. When t is 0, the denoising result corresponding to time step 0 is considered to be a noise-free result and is used to represent the potential probability distribution of the music. The audio generation module is also used to search in the audio data space of the VAE model based on the potential probability distribution to obtain the audio distribution corresponding to the lyrics sequence. Each point in the audio distribution represents a segment of audio. After converting the audio distribution into a spectrogram, the spectrogram is converted into the audio data of the song.

2. The end-to-end song generation system based on user colloquial expression according to claim 1, wherein, The lyrics sequence with musical phrase structure includes a sequence of lyrics, in which the lyrics are tagged, wherein the tags include timbre, instrument, sound effect, style, and musical phrase structure tags, which include intro, verse, pre-chorus, chorus, bridge or interlude, and conclusion. Ignoring the lyrics, the musical phrase structure tags are arranged by separating them with separator tags to form the musical phrase structure.

3. The end-to-end song generation system based on user colloquial expression of claim 1, wherein, The generative model includes a DIT sub-model for noise generation based on spliced ​​vectors and lyric sequences, and a DDPM sub-model for denoising calculation based on generated noise. In the DIM model, the input concatenated vector, as a noise latent vector, is processed by block partitioning and the first layer of normalization. Then, multi-head self-attention processing is used to focus on the first important feature. The first important feature and the input vector of multi-head self-attention processing are concatenated through the first residual connection and then processed by the second layer of normalization to obtain the first vector corresponding to the input concatenated vector after processing. The current input lyrics sequence is processed to obtain an embedding vector, which is then subjected to multi-head cross-attention processing with the first vector to focus on the second important feature. The second important feature is then concatenated with the first residual and concatenated with the second residual to obtain the second vector. After the second vector is normalized and fed forward by the third layer, it is concatenated with the second vector. The concatenated result is then normalized by the fourth layer, linearly mapped and shaped to obtain the noise distribution.

4. The end-to-end song generation system based on user colloquial language expression according to claim 3, characterized in that, The DIT sub-model is replaced by the SIT sub-model. In the SIT sub-model, random sampling is performed by selecting a sampler during the shape adjustment process, and a noise distribution is generated based on the random sampling results. In the interpolation framework of the DIT sub-model, a mapping relationship is found between the latent probability distribution of the generated model output and the high-dimensional normal distribution space of VAE by selecting interpolation parameters. Based on the mapping relationship, the audio distribution corresponding to the lyrics sequence is searched and determined.

5. The end-to-end song generation system based on user colloquial language expression according to claim 1, characterized in that, The VAE model includes a VAE encoder and a VAE decoder, which are obtained based on self-supervised training. The VAE encoder is used to compress and encode the input audio data into an audio distribution in the audio data space, and the VAE decoder is used to decode the audio distribution in the audio data space to generate a spectrogram. The audio distribution of the audio data space corresponding to the trained VAE encoder is sampled as sample data for training the generative model; The trained VAE decoder is used to convert the audio distribution into a spectrogram when applied.

6. The end-to-end song generation system based on user colloquial language expression according to claim 1, characterized in that, Before the generative model is applied, it needs to be optimized. During the optimization, the audio distribution of the audio data space of the VAE model is sampled to obtain sample data. The audio data is used as labels to perform supervised training of the generative model based on the sample data under the labels, so as to optimize the parameters of the generative model.

7. The end-to-end song generation system based on user colloquial language expression according to claim 1, characterized in that, The large language model was fine-tuned before being applied. The text data used in the fine-tuning included music annotations by professionals and enthusiasts, in which professional descriptions and colloquial descriptions were used, and the professional and colloquial descriptions formed data pairs. The large language model was fine-tuned using these data pairs so that it could be converted into a lyric sequence with a musical phrase structure based on the colloquial description text.

8. An end-to-end song generation method based on users' colloquial expressions, characterized in that, The end-to-end song generation system according to any one of claims 1-7, the method comprising the following steps: The acquisition module is used to obtain colloquial expressions input by the user and use them as descriptive text prompts; The lyrics generation module uses a large language model to generate a lyric sequence with a musical phrase structure based on descriptive text prompts. The music generation module generates audio data based on lyric sequences. Specifically, within each time window, a generative model is used to denoise the lyric sequence with a musical segment structure to generate a potential probability distribution representing the music. Based on the potential probability distribution, a high-dimensional audio distribution corresponding to the lyric sequence is obtained by searching in the audio data space of the VAE model. Each point in the audio distribution represents a segment of audio. After converting the high-dimensional audio distribution into a spectrogram, the spectrogram is converted into the audio data of the song. The audio data generated in sequence from multiple time windows constitute the final audio data.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the end-to-end song generation method based on user colloquial expressions as described in claim 8.

Images