Music generation model training method and music generation method

Abstract

The present disclosure relates to a music generation model training method and a music generation method. The music generation model includes a timbre encoder, a lyric encoder, a style encoder, and a music generator. The training method includes training the lyric encoder and the music encoder based on sample music and lyric text; inputting the mel-spectrogram features of the sample music into the trained music encoder to obtain a reference music representation; inputting the sample music into the timbre encoder and the style encoder to obtain sample timbre features and sample style features; inputting the lyric text into the trained lyric encoder to obtain sample lyric features; inputting the sample timbre features, sample style features, and sample lyric features into the music generator to obtain predicted sample features; training the style encoder based on the sample style features, style labels, and timbre labels; and training the music generator based on the predicted sample features and the reference music representation. The present disclosure can generate music that expresses specific semantics, specific singer's biological characteristics, and emotional characteristics.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a music generation model training method and a music generation method. Background Art

[0002] Music generation models are undergoing a paradigm shift from rule-driven to data-driven, but they suffer from poor ability to generate targeted music styles and low semantic fidelity of music lyrics. Specifically, when generating music based on sample music, the music generation models trained with relevant technologies may be disturbed by the timbre of the sample music, resulting in insufficient ability to represent the style of the sample music, making it difficult for the generated music to accurately match the style of the sample music. Moreover, when generating music based on lyrics, the music generated by the music generation models trained with relevant technologies may also have pronunciation errors such as syllable misplacement and vowel distortion. These problems make it difficult for the music generation models trained with relevant technologies to support fine-grained style control and high-fidelity vocal synthesis. Summary of the Invention

[0003] The present disclosure provides a music generation model training method and a music generation method to solve the problems in the related art. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for training a music generation model is provided, wherein the music generation model includes a timbre encoder, a lyrics encoder, a style encoder, and a music generator, and the method includes:

[0005] Obtain sample music, corresponding lyrics text, corresponding style tag and corresponding timbre tag;

[0006] Based on the sample music and the lyrics text, the lyrics encoder and the preset music encoder are jointly trained based on semantic alignment;

[0007] Inputting the mel spectrum features of the sample music into a trained music encoder for encoding to obtain a reference music representation;

[0008] Inputting the sample music into the timbre encoder and the style encoder to obtain sample timbre features and sample style features respectively;

[0009] Inputting the lyrics text into the trained lyrics encoder to obtain sample lyrics features;

[0010] Inputting the sample timbre feature, the sample style feature and the sample lyrics feature into the music generator to obtain predicted sample features;

[0011] Based on the sample style feature, the style label and the timbre label, training the style encoder based on timbre and style decoupling;

[0012] The music generator is trained based on the difference between the predicted sample features and the reference music representation.

[0013] In an exemplary embodiment, the training of the style encoder based on the decoupling of timbre and style based on the sample style feature, the style label, and the timbre label includes:

[0014] Performing timbre classification on the sample style features to obtain a timbre classification result;

[0015] Performing style classification on the style features of the sample to obtain a style classification result;

[0016] Determining a timbre loss based on a difference between the timbre classification result and the timbre label; determining a style loss based on a difference between the style classification result and the style label;

[0017] The parameters of the style encoder are adjusted with the training goal of reducing the style loss and increasing the timbre loss.

[0018] In an exemplary embodiment, the performing of semantic alignment-based joint training on the lyrics encoder and the preset music encoder based on the sample music and the lyrics text includes:

[0019] Based on the duration of the sample music, performing phoneme extraction on the sample lyrics to obtain a sample phoneme sequence;

[0020] Performing phoneme encoding on the sample phoneme sequence based on the lyrics encoder to obtain sample phoneme features;

[0021] Determining a mel spectrum feature corresponding to the sample music, inputting the mel spectrum feature into the music encoder for music encoding to obtain a sample music feature;

[0022] Based on the sample phoneme features and the sample music features, performing comparative learning training on the lyrics encoder and the music encoder;

[0023] The number of the sample phoneme codes in the sample phoneme feature is the same as the number of the sample music codes in the sample music feature.

[0024] In an exemplary embodiment, the performing comparative learning training on the lyrics encoder and the music encoder based on the sample phoneme features and the sample music features includes:

[0025] Based on the sample phoneme features and the sample music features, a lyrics-music alignment matrix is ​​constructed; the sample phoneme code corresponding to the first position and the sample music code corresponding to the first position constitute a positive sample pair, and the sample phoneme code corresponding to the second position and the sample music code corresponding to the second position constitute a negative sample pair, the first position being a position with the same row number and column number in the lyrics-music alignment matrix, and the second position being other positions in the lyrics-music alignment matrix excluding the first position;

[0026] Performing contrastive learning based on the positive sample pair and the negative sample pair, and constructing a contrastive learning loss;

[0027] Based on the contrastive learning loss, parameters of the lyrics encoder and the music encoder are adjusted.

[0028] In an exemplary embodiment, the extracting phonemes from the sample lyrics based on the duration of the sample music to obtain a sample phoneme sequence includes:

[0029] Parsing the phonemes in the sample lyrics to obtain an initial phoneme sequence;

[0030] The length of the initial phoneme sequence is adjusted based on the duration corresponding to the sample music to obtain the sample phoneme sequence, and the length of the sample phoneme sequence is consistent with the length of the sample music.

[0031] In an exemplary embodiment, the duration corresponding to the sample music includes the duration of each phoneme in the sample lyrics in the sample music, and adjusting the length of the initial phoneme sequence based on the duration corresponding to the sample music to obtain the sample phoneme sequence includes:

[0032] Based on the duration of each phoneme in the sample music, the initial phoneme sequence is upsampled to obtain the sample phoneme sequence.

[0033] In an exemplary embodiment, the training of the style encoder based on the decoupling of timbre and style based on the sample style features, the style label, and the timbre label; and the training of the music generator based on the difference between the predicted sample features and the reference music representation include:

[0034] The training of the style encoder based on the decoupling of timbre and style based on the sample style features, the style label and the timbre label is jointly implemented with the training of the music generator based on the difference between the predicted sample features and the reference music representation.

[0035] In an exemplary embodiment, the loss of the training based on timbre and style decoupling is represented by a cross entropy loss, and the difference between the predicted sample features and the reference music representation is represented by a mean square error loss.

[0036] According to a second aspect of the embodiments of the present disclosure, a music generation method is provided, the method comprising:

[0037] Obtain target voice, target lyrics and sample music;

[0038] By inputting the target speech, the target lyrics and the sample music into a music generation model, a target music is obtained, wherein the lyrics of the target music are the target lyrics, and the target music has a timbre corresponding to the target speech and a musical style corresponding to the sample music;

[0039] The music generation model is trained using the music generation model training method described in any one of the first aspects.

[0040] In an exemplary embodiment, the target music is obtained by inputting the target speech, the target lyrics, and the sample music into a music generation model, including:

[0041] Inputting the target speech into the timbre encoder of the music generation model to obtain target timbre features;

[0042] Inputting the target lyrics into the lyrics encoder of the music generation model to obtain target lyrics features;

[0043] Inputting the sample music into the style encoder of the music generation model to obtain target style features;

[0044] The target timbre feature, the target lyrics feature and the target style feature are input into the music generator of the music generation model to obtain a target music representation; and the target music is obtained by decoding the target music representation.

[0045] According to a third aspect of an embodiment of the present disclosure, a music generation model training device is provided, wherein the music generation model includes a timbre encoder, a lyrics encoder, a style encoder, and a music generator, and the device includes:

[0046] A sample data acquisition module is configured to acquire sample music, corresponding lyrics text, corresponding style tag and corresponding timbre tag;

[0047] A first training module is configured to perform joint training based on semantic alignment on the lyrics encoder and the preset music encoder based on the sample music and the lyrics text;

[0048] A reference music representation prediction module is configured to input the mel spectrum features of the sample music into a trained music encoder for encoding to obtain a reference music representation;

[0049] The music prediction module is configured to input the sample music into the timbre encoder and the style encoder to obtain sample timbre features and sample style features respectively; input the lyrics text into the trained lyrics encoder to obtain sample lyrics features; and input the sample timbre features, the sample style features, and the sample lyrics features into the music generator to obtain predicted sample features.

[0050] The second training module is configured to perform training on the style encoder based on the decoupling of timbre and style based on the sample style features, the style label and the timbre label; and the music generator is trained based on the difference between the predicted sample features and the reference music representation.

[0051] In an exemplary embodiment, the second training module is configured to execute:

[0052] Performing timbre classification on the sample style features to obtain a timbre classification result;

[0053] Performing style classification on the style features of the sample to obtain a style classification result;

[0054] Determining a timbre loss based on a difference between the timbre classification result and the timbre label; determining a style loss based on a difference between the style classification result and the style label;

[0055] The parameters of the style encoder are adjusted with the training goal of reducing the style loss and increasing the timbre loss.

[0056] In an exemplary embodiment, the first training module is configured to execute:

[0057] Based on the duration of the sample music, performing phoneme extraction on the sample lyrics to obtain a sample phoneme sequence;

[0058] Performing phoneme encoding on the sample phoneme sequence based on the lyrics encoder to obtain sample phoneme features;

[0059] Determining a mel spectrum feature corresponding to the sample music, inputting the mel spectrum feature into the music encoder for music encoding to obtain a sample music feature;

[0060] Based on the sample phoneme features and the sample music features, performing comparative learning training on the lyrics encoder and the music encoder;

[0061] The number of the sample phoneme codes in the sample phoneme feature is the same as the number of the sample music codes in the sample music feature.

[0062] In an exemplary embodiment, the first training module is configured to execute:

[0063] Based on the sample phoneme features and the sample music features, a lyrics-music alignment matrix is ​​constructed; the sample phoneme code corresponding to the first position and the sample music code corresponding to the first position constitute a positive sample pair, and the sample phoneme code corresponding to the second position and the sample music code corresponding to the second position constitute a negative sample pair, the first position being a position with the same row number and column number in the lyrics-music alignment matrix, and the second position being other positions in the lyrics-music alignment matrix excluding the first position;

[0064] Performing contrastive learning based on the positive sample pair and the negative sample pair, and constructing a contrastive learning loss;

[0065] Based on the contrastive learning loss, parameters of the lyrics encoder and the music encoder are adjusted.

[0066] In an exemplary embodiment, the first training module is configured to execute:

[0067] Parsing the phonemes in the sample lyrics to obtain an initial phoneme sequence;

[0068] The length of the initial phoneme sequence is adjusted based on the duration corresponding to the sample music to obtain the sample phoneme sequence, and the length of the sample phoneme sequence is consistent with the length of the sample music.

[0069] In an exemplary embodiment, the duration corresponding to the sample music includes the duration of each phoneme in the sample lyrics in the sample music, and the first training module is configured to execute:

[0070] Based on the duration of each phoneme in the sample music, the initial phoneme sequence is upsampled to obtain the sample phoneme sequence.

[0071] In an exemplary embodiment, the second training module is configured to execute:

[0072] The training of the style encoder based on the decoupling of timbre and style based on the sample style features, the style label and the timbre label is jointly implemented with the training of the music generator based on the difference between the predicted sample features and the reference music representation.

[0073] In an exemplary embodiment, the loss of the training based on timbre and style decoupling is represented by a cross entropy loss, and the difference between the predicted sample features and the reference music representation is represented by a mean square error loss.

[0074] According to a fourth aspect of an embodiment of the present disclosure, there is provided a music generating apparatus, the apparatus comprising:

[0075] A target data acquisition module is configured to acquire target speech, target lyrics and sample music;

[0076] a music generation module configured to obtain target music by inputting the target speech, the target lyrics, and the sample music into a music generation model, wherein the target music has the target lyrics and has a timbre corresponding to the target speech and a musical style corresponding to the sample music;

[0077] The music generation model is trained using the music generation model training method described in any one of the first aspects.

[0078] In an exemplary embodiment, the music generation module is configured to execute:

[0079] Inputting the target speech into the timbre encoder of the music generation model to obtain target timbre features;

[0080] Inputting the target lyrics into the lyrics encoder of the music generation model to obtain target lyrics features;

[0081] Inputting the sample music into the style encoder of the music generation model to obtain target style features;

[0082] The target timbre feature, the target lyrics feature and the target style feature are input into the music generator of the music generation model to obtain a target music representation; and the target music is obtained by decoding the target music representation.

[0083] According to a fifth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0084] processor;

[0085] a memory for storing instructions executable by the processor;

[0086] The processor is configured to execute the instructions to implement the music generation model training method or music generation method as described in any of the above embodiments.

[0087] According to the sixth aspect of an embodiment of the present disclosure, a computer storage medium is provided. When the instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device executes the music generation model training method or the music generation method described in any of the above embodiments.

[0088] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements the music generation model training method or music generation method described in any of the above-mentioned embodiments.

[0089] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0090] The music generation model training method provided by the embodiments of the present disclosure first provides a model architecture for a music generation model based on feature fusion. The architecture can fuse timbre features indicating the timbre of music (from the timbre encoder of the music generation model), semantic features indicating the lyrics of music (from the lyrics encoder of the music generation model), and style features indicating the style of music (from the style encoder of the music generation model). The timbre features and style features can respectively characterize the biological characteristics and emotional characteristics of a specific singer, and the style features can also characterize the melody characteristics. Therefore, the architecture enables the music generation model to have the basic conditions for generating music that can express specific semantics, specific music styles, and the biological characteristics and emotional characteristics of a specific singer.

[0091] During the training process of the music generation model, the lyrics encoder and the preset music encoder of the music generation model are first jointly trained based on semantic alignment. The lyrics encoder is used to encode lyrics into corresponding semantic features (lyrics features), and the music encoder is used to encode music into corresponding music features. The joint training based on semantic alignment enables the lyrics features to be frame aligned with the music features. And because this joint training is implemented before the training of the music generator of the music generation model, the music generator is trained under the guidance of the music features output by the trained music encoder. Therefore, the music features of the music predicted by the music generator are also trained to achieve frame alignment with the lyrics features output by the trained lyrics encoder. As a result, the music predicted by the music generation model and the lyrics referenced by the music generation model have a frame alignment relationship, minimizing pronunciation errors such as syllable misalignment and vowel distortion caused by frame misalignment.

[0092] During the training process of the music generation model, the style encoder of the music generation model is also trained based on the decoupling of timbre and style, which enables the style encoder to extract style features without being interfered by timbre.

[0093] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The accompanying drawings herein are incorporated into the specification and constitute a part of the present disclosure, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation to the present disclosure.

[0095] Figure 1 is an architectural diagram of a music generation model according to an exemplary embodiment;

[0096] Figure 2 is a flowchart of a music generation model training method according to an exemplary embodiment;

[0097] Figure 3 is a flowchart of training based on decoupling of timbre and style according to an exemplary embodiment;

[0098] Figure 4 is a flowchart of joint training based on semantic alignment according to an exemplary embodiment;

[0099] Figure 5 is a schematic diagram of a comparative learning process according to an exemplary embodiment;

[0100] Figure 6 is a schematic diagram showing a process of music generator and style encoder learning according to an exemplary embodiment;

[0101] Figure 7 is a flow chart of a music generation method according to an exemplary embodiment;

[0102] Figure 8 is a schematic diagram of a music generation process according to an exemplary embodiment;

[0103] Figure 9 is a block diagram of a music generation model training device according to an exemplary embodiment;

[0104] Figure 10 is a block diagram of a music generating apparatus according to an exemplary embodiment;

[0105] Figure 11 A structural block diagram of a computer device according to an exemplary embodiment is shown. Figure 1 ;

[0106] Figure 12 A structural block diagram of a computer device according to an exemplary embodiment is shown. Figure 2 . DETAILED DESCRIPTION

[0107] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0108] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0109] Figure 1 1 is an architecture diagram of a music generation model according to an exemplary embodiment. The music generation model includes a timbre encoder 110, a lyrics encoder 120, a style encoder 130, and a music generator 140. The timbre encoder 110, the lyrics encoder 120, and the style encoder 130 are all connected to the music generator 140.

[0110] The timbre encoder 110 is used to extract and encode the timbre of the input audio, obtaining corresponding timbre features. This application does not limit the audio; it can be from speech or music. These timbre features accurately capture the unique properties of timbre. Timbre can reflect the biological characteristics of the performer, such as brightness and softness, providing an accurate foundation for timbre information for subsequent music generation. The lyrics encoder 120 is dedicated to encoding music lyrics, converting the linguistic information embodied in the lyrics into a suitable vector representation, namely, lyric features, thereby accurately capturing the phoneme information conveyed by the lyrics. Phonemes are the smallest units of speech, classified according to the natural properties of speech, and are divided into two categories: vowels and consonants. From an acoustic perspective, phonemes are the smallest units of speech, classified from the perspective of sound quality. From a physiological perspective, one pronunciation action forms one phoneme. For example, [da] contains two pronunciation actions, [d] and [a], and therefore two phonemes. The sound produced by the same pronunciation action is the same phoneme, while the sounds produced by different pronunciation actions are different phonemes.

[0111] The style encoder 130 is responsible for encoding the stylistic features of the music to generate style characteristics. These style characteristics can not only indicate the melodic style of the music, such as pop, rock, and classical, but also the emotional style of the singer, such as cheerful, sad, and passionate. These rich style characteristics provide a diverse range of style dimensions for music generation, allowing the generated music to accurately meet different emotional expressions and aesthetic needs.

[0112] The music generator 140 receives features output from the timbre encoder 110 , the lyrics encoder 120 , and the style encoder 130 , and integrates these features to generate music with a specific timbre, matching the lyrics, and conforming to a specific style.

[0113] The embodiment of the present application does not limit the music generator 140. For example, it can be a model that runs Flowmatching, Diffusion, and Language Model. Flowmatching, Diffusion, and Language Model refer to different model algorithms respectively. The Flowmatching algorithm is a new generative modeling method that generates new data by learning how to convert a simple prior distribution sample into a data distribution sample. In music generation, this conversion mechanism can be used to create novel music clips that meet specific feature requirements. The Diffusion algorithm is based on the diffusion process, which gradually adds noise to the data and then learns to recover the original data from the noise. In music generation, various elements of music such as melody and rhythm can be gradually adjusted and optimized to generate high-quality music. The Language Model language model was originally mainly used for natural language processing, but it also has applications in the field of music generation. It can learn the structure, patterns and rules of music, just like understanding the grammar and semantics of language, and generate coherent and logical music according to specific feature requirements.

[0114] In summary, the model architecture of the music generation model provided by the embodiment of the present disclosure can integrate timbre features indicating the timbre of music (from the timbre encoder of the music generation model), semantic features indicating the lyrics of music (from the lyrics encoder of the music generation model), and style features indicating the style of music (from the style encoder of the music generation model). The timbre features and style features can characterize the biological characteristics, emotional characteristics, and melodic style of a specific singer. Therefore, this architecture enables the music generation model to have the basic conditions for generating music that can express specific musical semantics, specific music styles, and the biological characteristics and emotional characteristics of a specific singer.

[0115] Figure 2 This is a flowchart of a music generation model training method according to an exemplary embodiment. The music generation model training method is for Figure 1The music generation model implementation shown. The music generation model training method can be applied to electronic devices, which can be implemented by a server or a terminal alone, or can be implemented by a terminal and a server in collaboration. Among them, the terminal can be but not limited to physical devices such as smart phones, tablet computers, laptops, desktop computers, smart speakers, smart wearable devices, digital assistants, augmented reality devices, virtual reality devices, etc., and can also include software such as applications running in physical devices. The server can be but not limited to an independent server, or a server cluster or distributed system composed of multiple physical servers, or one or more cloud servers that provide cloud services, cloud databases, cloud computing, cloud, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, and basic cloud computing services such as big data and artificial intelligence platforms. Reference Figure 2 As shown, the method includes the following steps.

[0116] In step S210, sample music, corresponding lyrics text, corresponding style label and corresponding timbre label are obtained.

[0117] The examples in this application do not limit the sample music; it can encompass a wide range of genres, such as pop, classical, rock, and jazz. Its sources can also be diverse, ranging from music from public music repositories to original music or covers uploaded by users. Music samples of varying genres and sources can provide rich data for training, helping to improve the model's generalization and generation performance.

[0118] The corresponding lyric text, corresponding style label and corresponding timbre label indicate information of different dimensions of the sample music. The lyric text reflects the semantic information and phoneme information of the music, and indicates the expression logic of the sample music at the text level; the style label indicates the style type of the sample music, which can include not only the melody style but also the personal style of the singer, indicating the overall style tone of the sample music; the timbre label indicates the characteristics of the sound in the sample music, including pitch, sound quality, volume and other aspects.

[0119] In step S220, based on the sample music and the lyrics text, the lyrics encoder and the preset music encoder are jointly trained based on semantic alignment.

[0120] The embodiments of this application do not limit the music encoder and may employ various types of encoders, such as an encoder based on a recurrent neural network (RNN), an encoder based on a long short-term memory network (LSTM), or an encoder based on a gated recurrent unit (GRU). The embodiments of this application also do not limit the structure of the lyrics encoder and may employ an encoder based on a convolutional neural network (CNN) architecture, which can extract local features from the lyrics text through convolutional layers; or an encoder based on an attention mechanism, which can focus on key parts of the lyrics to better capture the semantic information and contextual relationships of the lyrics.

[0121] Related technologies can use cross-modal alignment to perform joint training of lyrics and music when training music generation models, but this type of training cannot achieve frame-level lyrics synchronization, which leads to semantic distortion and causes the generated music to have problems such as syllable errors, vowel distortion, consonant loss (such as "tian" → "ian") and tone shift. The semantic alignment-based joint training of step S220 of the present disclosure is a pre-training, which is implemented before the following steps S230-S260. After this pre-training, the music representations and lyrics features output by the trained music encoder and the lyrics encoder can establish a frame-level correspondence, so that the music representations output by the trained music encoder are used as supervision information in the following steps S230-S260. The features of the music generated by the trained music generator can also establish a frame-level correspondence with the lyrics features output by the lyrics encoder, so that the music generator can fully learn the frame-level semantic alignment knowledge in the music, ensuring that the lyrics semantics are accurately transferred to the music during the generation process.

[0122] In step S230, the mel spectrum features of the sample music are input into the trained music encoder for encoding to obtain a reference music representation.

[0123] The lyrics encoder and music encoder obtained after the pre-training of step S220 have the ability of frame-level semantic alignment. In this case, the mel-spectrogram features of the sample music are input into the trained music encoder for encoding to obtain a reference music representation. The reference music representation can accurately reflect the characteristic information of the sample music, and due to the frame-level correspondence established by the pre-training, the reference music representation can be accurately matched with the lyrics features output by the trained lyrics encoder. In the subsequent training process, the reference music representation will be used as key supervision information to train the music generator. In this way, the music generator can learn the knowledge of frame-level semantic alignment between music and lyrics, and then when generating music, it can ensure that the generated music features are closely corresponding to the lyrics features, avoiding semantic distortion problems such as syllable errors, vowel distortions, consonant drops and tone shifts, so that the generated music is highly consistent with the lyrics at the semantic level, achieving high-quality music generation.

[0124] In step S240, the sample music is input into the timbre encoder and the style encoder to obtain sample timbre features and sample style features respectively; the lyrics text is input into the trained lyrics encoder to obtain sample lyrics features; the sample timbre features, the sample style features and the sample lyrics features are input into the music generator to obtain predicted sample features.

[0125] The present disclosure does not limit the source of the timbre encoder, which can be an open source component or a specific model obtained through custom training. Exemplarily, the open source components may include CREPE, CREPE-S, MFA, etc. These open source timbre encoder components have certain versatility and effectiveness in timbre feature extraction. The present disclosure does not limit the structure of the style encoder. Exemplarily, it can be a structure built based on a convolutional neural network (CNN), or a structure built based on a recurrent neural network (RNN) or its variants such as a long short-term memory network (LSTM) and a gated recurrent unit (GRU) to process sequence information in the music style. Step S240 discloses the step of using a music generator to predict music that matches the sample timbre features, sample style features, and sample lyrics features. By inputting these three features into the music generator, the corresponding features of the music can be predicted, that is, the sample features can be predicted.

[0126] In step S250 , the style encoder is trained based on the decoupling of timbre and style based on the sample style features, the style label, and the timbre label.

[0127] The purpose of training based on the decoupling of timbre and style is to enable the style encoder to learn and capture the style features in music more accurately, avoid the interference of timbre features on the learning of style features, and the learned and captured style features can include not only conventional genre features, but also personalized features that reflect the biological characteristics of the singer, so that the music generated based on the style features is more unique and recognizable, can better fit the characteristics of different singers, and meet the diverse needs of music creation.

[0128] Music generation models in related technologies can introduce style information based on conditional generation of style embedding vectors (Transformer-GANs (SCTG model)), but this type of music generation model cannot solve the problem of timbre leakage, resulting in timbre-style coupling. Its style encoder mixes in the speaker's timbre information when extracting features, causing the singing generated by the music generation model based on this style encoder to be "contaminated" by the timbre of the reference singer. For example: the user specifies that the music generated by the music generation model has a "sad style + male timbre", but because the source of the sad style voice is a female singer, the generated music sounds like a female singer. The present disclosure, by training the style encoder based on timbre and style decoupling, decouples timbre and style features, and obtains a timbre-independent style encoder that can support arbitrary style encoding.

[0129] In step S260, the music generator is trained based on the difference between the predicted sample features and the reference music representation.

[0130] The present disclosure does not limit the quantification method of the difference between the predicted sample features and the reference music representation. Exemplarily, the mean square error loss representation can be used. The mean square error (MSE) loss can be used to characterize the loss generated when reconstructing music, thereby guiding the music generator to learn music representation, and providing an indicator to measure the reconstruction accuracy through the MSE loss to optimize the training effect of the music generator. The present disclosure does not limit the parameter adjustment method and parameter adjustment stop condition of the music generator. For example, the gradient descent method can be used. For the parameter adjustment stop condition, the maximum number of iterations can be set, and the parameter adjustment can be stopped when the preset number of iterations is reached; or the parameter adjustment can also be stopped when the mean square error loss is less than a preset threshold.

[0131] The present disclosure does not limit the order in which steps S250 and S260 are executed; they may be executed sequentially or simultaneously. In an exemplary embodiment, the style encoder is trained based on the decoupling of timbre and style based on the sample style features, the style label, and the timbre label, and the music generator is trained based on the difference between the predicted sample features and the reference music representation.

[0132] The present disclosure does not limit the stopping conditions of the joint implementation. For example, the maximum number of iterations of the joint implementation can be set as a stopping condition. Through the joint implementation, the performance of the style encoder and the music generator can be optimized at the same time, and the training efficiency can be improved. During the joint implementation, the training process of the style encoder can focus on extracting style features that are not related to timbre, so that the generated style features are more pure and accurate; and the music generator can, on this basis, continuously adjust its own parameters according to the differences between the predicted sample features and the reference music representation, so as to better learn the characteristics and laws of music. In this way, the training of the two models promotes each other, and the entire music generation model can achieve better results in the generation of timbre and style. At the same time, the joint implementation can also reduce the time and resource consumption of training, share some training data and computing resources, and avoid repeated training processes. In addition, the generalization ability of the model can be enhanced through joint training, so that the music generation model can perform well when facing music of different styles and timbres.

[0133] In the present disclosure, the timbre encoder, the trained lyrics encoder, the trained style encoder and the trained music generator together constitute a trained music generation model, which has the ability to generate music that can express specific musical semantics, specific music styles, and specific biological and emotional characteristics of the singer.

[0134] In an exemplary embodiment, please refer to Figure 3 , which shows a flowchart of training based on timbre and style decoupling in an exemplary embodiment. The training of the style encoder based on timbre and style decoupling based on the sample style features, the style label and the timbre label includes:

[0135] Step S310: Perform timbre classification on the sample style features to obtain a timbre classification result; perform style classification on the sample style features to obtain a style classification result.

[0136] The present disclosure does not limit the execution method of timbre classification, and an open source or pre-acquired timbre classifier may be used. The present disclosure does not limit the execution method of style classification, and an open source or pre-acquired style classifier may be used. In order to improve the style generalization ability of the style encoder, a single style label or a mixed style label may be set for the sample music. Exemplarily, a single style label may be a clear style identifier such as "rock", "classical", and "pop", so that the style encoder can focus on learning the characteristics of that specific style. The mixed style label may be a combination of "rock and pop fusion" and "classical and electronic combination". In this way, the style encoder can be exposed to more complex and diverse combinations of style features, thereby enhancing its generalization ability between different styles, so that when faced with music samples of various styles, it can accurately extract and learn the style information therein, so as to better serve the subsequent music generation process.

[0137] Step S320: Determine the timbre loss based on the difference between the timbre classification result and the timbre label; determine the style loss based on the difference between the style classification result and the style label.

[0138] The present disclosure does not limit the representation method of the music loss and the style loss. Exemplarily, both can be represented using cross entropy loss.

[0139] Step S330 : Adjust the parameters of the style encoder with the training goal of reducing the style loss and increasing the timbre loss.

[0140] The present disclosure does not limit the parameter adjustment method and parameter adjustment stop conditions, and reference may be made to the previous text. In order to solve the problem of timbre coupling in style coding, the present disclosure uses style classification to strengthen the modeling of style information, and uses the gradient reversal of timbre classification to decouple the timbre information, so that the style features encoded by the trained style encoder contain as little timbre information as possible. The training goal of reducing the style loss reflects the enhanced extraction of style information by the style encoder, and the training goal of increasing the timbre loss reflects the weakening and decoupling of timbre information by the style encoder. Increasing the timbre loss means reversing the gradient generated by the timbre loss (Gradient Reversal Layer, GRL), so that the style encoder ignores the timbre information as much as possible when solving the timbre coupling problem in style coding. By reversing the gradient generated by the timbre loss, during the back propagation process, the gradient direction originally used to reduce the timbre loss is reversed and becomes the direction of increasing the timbre loss. In this way, during training, the style encoder will adjust its parameters in the direction of weakening the capture of timbre information, thereby achieving the decoupling of style features from timbre information, allowing the style encoder to more focus on extracting style information from music without being disturbed by timbre information. Ultimately, the style features encoded by the trained style encoder will contain as little timbre information as possible.

[0141] In an exemplary embodiment, please refer to Figure 4 , which shows a flowchart of joint training based on semantic alignment in an exemplary embodiment. The joint training based on semantic alignment of the lyrics encoder and the preset music encoder based on the sample music and the lyrics text includes:

[0142] S410. Based on the duration corresponding to the sample music, perform phoneme extraction on the sample lyrics to obtain a sample phoneme sequence.

[0143] In the present disclosure, the duration of the sample music can be obtained by identifying the sample music. Optionally, the duration of the sample music includes the duration of each phoneme in the sample lyrics in the sample music. If a pronunciation is sung slower, the duration of the corresponding phoneme in the sample music will be longer, and if a pronunciation is sung faster, the duration of the corresponding phoneme in the sample music will be shorter. The difference in duration reflects the individual differences of the sample music.

[0144] In an exemplary embodiment, extracting phonemes from the sample lyrics based on the duration of the sample music to obtain a sample phoneme sequence may include: parsing the phonemes in the sample lyrics to obtain an initial phoneme sequence; and adjusting the length of the initial phoneme sequence based on the duration of the sample music to obtain the sample phoneme sequence, wherein the length of the sample phoneme sequence is consistent with the length of the sample music. Adjusting the length of the initial phoneme sequence based on the duration of the sample music to obtain the sample phoneme sequence may include: upsampling the initial phoneme sequence based on the duration of each phoneme in the sample music to obtain the sample phoneme sequence.

[0145] For example, the phonemes in the sample lyrics can be parsed to obtain an initial phoneme sequence. This initial phoneme sequence is composed of the phonemes extracted from the sample lyrics. For example, for the sample lyric "Ocean," the corresponding initial phoneme sequence is (h, a, i, y, a, n, g). Accordingly, the duration of each phoneme can be [1, 1, 2, 2, 1, 3, 3].

[0146] Optionally, the duration of the initial phoneme sequence and the corresponding sample music can be input into a length adjuster. The length adjuster is preset and does not require parameter adjustment through training. The length adjuster upsamples the initial phoneme sequence based on the duration of each phoneme in the sample music to obtain a sample phoneme sequence. For example, for a certain sample lyric 1, according to the duration of its corresponding phoneme d1=[1,1,2,2,1,3], the initial phoneme sequence (t,i,a,n,q,i) of the sample lyric 1 is upsampled to obtain a sample phoneme sequence of (t,i,a,a,n,n,q,i,i,i). For sample lyric 2, according to the duration of its corresponding phoneme d2=[2,3,1,1,1], the initial phoneme sequence (s,h,u,m,u) of the sample lyric 2 is upsampled to obtain a sample phoneme sequence of (s,s,h,h,h,u,m,u).

[0147] The present disclosure upsamples the initial phoneme sequence based on the duration corresponding to the sample music, so that the length of the sample phoneme sequence can be consistent with the length of the above-mentioned sample music, which facilitates the effective comparison of the differences between the two types of features, phonemes and music, effectively solves the problem of length mismatch between the two types of data, and achieves semantic alignment at the frame level.

[0148] S420. Perform phoneme encoding on the sample phoneme sequence based on the lyrics encoder to obtain sample phoneme features.

[0149] The sample phoneme features are used to reflect the representation related to the phoneme information in the sample lyrics, thereby introducing the information of the semantic modality of the lyrics in the learning process.

[0150] S430. Determine the mel-spectrogram feature corresponding to the sample music, input the mel-spectrogram feature into the music encoder for music encoding, and obtain a sample music feature; wherein the number of sample phoneme codes in the sample phoneme feature is the same as the number of sample music codes in the sample music feature.

[0151] Mel-scale frequency cepstral coefficients (MFCCs) are a type of acoustic feature data. The sample mel spectrum corresponding to a music sample can refer to the mel-scale cepstral coefficients obtained based on the music sample. Sample music features reflect the inherent information of the music sample and can be understood as the information expression of the music sample.

[0152] If the sample phoneme features and the sample music features are one-dimensional sequences, they have the same length. If the sample phoneme features and the sample music features are in the form of a matrix, the sample phoneme features and the sample music features in the matrix form can be reduced in dimensionality by, for example, unfolding or pooling to obtain the sample phoneme features and the sample music features in the form of a one-dimensional vector. Therefore, the sample phoneme features and the sample music features obtained in the present disclosure can be considered to have a one-dimensional form. The sample phoneme features in one-dimensional form can include multiple elements, each element being understood as a sample phoneme code. Similarly, the sample music features in one-dimensional form can include multiple elements, each element being understood as a sample music code. The number of sample phoneme codes and sample music codes is the same.

[0153] S440. Based on the sample phoneme features and the sample music features, perform comparative learning training on the lyrics encoder and the music encoder.

[0154] Since the number of sample phoneme codes in the sample phoneme features is the same as the number of sample music codes in the sample music features, the lyrics encoder and the music encoder can be subjected to comparative learning training based on the sample phoneme features and the sample music features. The present disclosure does not limit the parameter adjustment method and parameter adjustment stop method of the comparative learning training, and reference can be made to the above text. The comparative learning training is used to model the frame-level dissimilarity between the sample phoneme features and the above-mentioned sample music features, so that the phoneme representation and the music representation are connected in a joint multimodal space, guiding the style encoder and the music encoder to learn fine-grained semantic information at the frame level.

[0155] The present disclosure does not limit the contrastive learning training method. For example, a lyrics-music alignment matrix can be constructed based on the sample phoneme features and the sample music features; the sample phoneme encoding corresponding to the first position and the sample music encoding corresponding to the first position constitute a positive sample pair, and the sample phoneme encoding corresponding to the second position and the sample music encoding corresponding to the second position constitute a negative sample pair, the first position is the position with the same row number and column number in the lyrics-music alignment matrix, and the second position is the other position in the lyrics-music alignment matrix except the first position; contrastive learning is performed based on the positive sample pairs and the negative sample pairs to construct a contrastive learning loss; based on the contrastive learning loss, the parameters of the lyrics encoder and the music encoder are adjusted.

[0156] Specifically, a two-dimensional lyrics-music alignment matrix can be constructed based on the element order of the sequences of sample phoneme features and sample music features. Each matrix parameter in the lyrics-music alignment matrix is ​​composed of the sample phoneme code at the corresponding position in the sample phoneme feature and the sample music code at the corresponding position in the sample music feature.

[0157] In the lyrics-music alignment matrix, positions with the same row and column numbers are the first positions. These first positions form the diagonal lines of the lyrics-music alignment matrix. Positions not on the diagonal lines are the second positions. The matrix parameters at the first position, i.e., the sample phoneme code and sample music code at the first position, constitute a positive sample pair; the matrix parameters at the second position, i.e., the sample phoneme code and sample music code at the second position, constitute a negative sample pair.

[0158] This disclosure does not limit the function used for contrastive loss. Contrastive loss is a loss function commonly used in machine learning and deep learning. It brings the feature representations of positive pairs closer together and negative pairs farther apart. The basic idea of ​​contrastive loss is to minimize the feature distance between positive pairs and maximize the feature distance between negative pairs. The goal of contrastive loss is to ensure that the similarity between the feature representations of positive pairs output by the model is much greater than the similarity between the feature representations of negative pairs, thereby learning effective and discriminative feature representations. This disclosure uses contrastive learning based on positive and negative pairs. During this contrastive learning process, effective and discriminative feature representations are learned by adjusting model parameters. Exemplarily, this feature distance can be determined by, but not limited to, cosine similarity. During the contrastive learning process, the representations of phonemes and music are connected in a joint multimodal space. Contrastive loss is used to capture frame-level dissimilarities and similarities between sample lyrics and sample music. Through frame-level contrastive learning, fine-grained information is learned. Experiments have shown that this disclosure achieves strict frame alignment of phonemes and music by combining a length adjuster with contrastive loss training, with an alignment error of <10ms.

[0159] In some other implementations, a cross-entropy loss based on ASR forced alignment can be used for training instead of contrastive loss. This involves using automatic speech recognition (ASR) technology to force alignment between lyrics and music, determining the mapping between each phoneme in the lyrics and the corresponding frame in the audio. The cross-entropy loss measures the difference between the phoneme distribution and the true phoneme distribution. Minimizing this cross-entropy loss can also achieve frame alignment.

[0160] Please refer to Figure 5 , which shows a schematic diagram of the comparative learning process in an exemplary embodiment. The comparative learning process is as follows:

[0161] (1) Perform phoneme extraction on the sample lyrics, input the extracted initial phoneme sequence and the duration of each phoneme into the length regulator, and obtain the sample phoneme sequence after upsampling.

[0162] (2) Input the sample phoneme sequence into the Lyrics Encoder to obtain the encoded sample phoneme features (Phoneme Embedding (P)).

[0163] (3) Extract the Mel-spectrogram of the sample music (Music) and input it into the music encoder (Music Encoder) to obtain the encoded sample music features (Music Embedding (S)).

[0164] (4) Based on the sample phoneme features (Phoneme Embedding (P)) and the sample music features (SpeechEmbedding (S)), comparative learning is performed to adjust the parameters of the lyrics encoder (Lyrics Encoder) and the music encoder (MusicEncoder) until the preset parameter adjustment stop condition is reached. Figure 5 The k in represents any element, and k is an integer greater than or equal to 0 and less than the total number of elements.

[0165] After contrastive learning is completed, the music generator and style encoder can be trained. Figure 6 , which shows a schematic diagram of the process of music generator and style encoder learning in an exemplary embodiment. The process is as follows:

[0166] (10) Inputting the mel spectrum features of the sample music into the trained music encoder for encoding to obtain a reference music representation.

[0167] (20) Inputting the sample music into the timbre encoder (Speaker Encoder) and the style encoder (Style Encoder) to obtain sample timbre features and sample style features respectively;

[0168] (30) Inputting the lyrics text into the trained lyrics encoder to obtain sample lyrics features;

[0169] (40) Inputting the sample timbre feature, the sample style feature and the sample lyrics feature into the music generator (Acoustic Model) to obtain predicted sample features.

[0170] (50) Performing timbre classification on the sample style features to obtain a timbre classification result; performing style classification on the sample style features to obtain a style classification result; determining timbre loss based on the difference between the timbre classification result and the timbre label corresponding to the sample music; determining style loss based on the difference between the style classification result and the style label corresponding to the sample music; adjusting parameters of the style encoder with the training goal of reducing the style loss and increasing the timbre loss. Furthermore, adjusting parameters of the music generator based on the reference music representation and the predicted sample features.

[0171] The music generation model training method disclosed in this disclosure solves at least the following two problems:

[0172] Solved the problem of single style and timbre leakage: Music generation often requires artificial definition of music styles, such as jazz, rock, pop, rap, etc. However, with the diversification of music creation, many new styles of music continue to emerge. A lot of music cannot be classified into a specific music type. However, related technologies often require explicit provision of specific music types as input when generating music, which greatly limits the modeling ability of the model and makes it difficult to generate other personalized styles outside the existing defined styles. If a style encoder is used to extract music styles, there will be a timbre leakage problem because these music contain human voices. The present disclosure solves the timbre leakage problem by designing a unique training method for a style encoder. Music generation based on a style encoder avoids explicitly providing a specific music style, so it can generate music of any personalized style, significantly improving the use scenarios and performance of the music generation model.

[0173] Solve the problem of pronunciation errors caused by semantic distortion: Related technologies often use traditional pre-trained audio encoders, such as DAC and WavTokenizer, to generate intermediate representations for music. While these audio encoders support music encoding and restoration, they lack semantic supervision during the pre-training process, resulting in semantic alignment issues such as pronunciation errors in the generated music. This disclosure achieves semantically aligned joint learning of lyrics and music by pre-training lyrics encoders and music encoders, eliminating semantic alignment issues in the generated music.

[0174] Please refer to Figure 7 , which shows a flow chart of a music generation method in an exemplary embodiment, the method comprising:

[0175] S710. Get the target voice, target lyrics and sample music;

[0176] The target speech can be any recorded voice clip from a performer, used to convey the performer's timbre. The target lyrics provide the music generation model with lyric information for use in synthesizing music. The sample music can be any music, used to provide the music generation model with style information that the synthesized music should match.

[0177] S720. Target music is obtained by inputting the target speech, the target lyrics and the sample music into a music generation model, wherein the lyrics of the target music are the target lyrics, and the target music has the timbre corresponding to the target speech and the musical style corresponding to the sample music; wherein the music generation model is trained using the aforementioned music generation model training method.

[0178] Specifically, the target music is obtained by inputting the target speech, the target lyrics and the sample music into a music generation model, including: inputting the target speech into the timbre encoder of the music generation model to obtain target timbre features; inputting the target lyrics into the lyrics encoder of the music generation model to obtain target lyrics features; inputting the sample music into the style encoder of the music generation model to obtain target style features; inputting the target timbre features, the target lyrics features and the target style features into the music generator of the music generation model to obtain target music representation; and obtaining the target music by decoding the target music representation.

[0179] The present disclosure can extract information of any timbre, any style and any lyrics through target voice, target lyrics and sample music, thereby generating personalized music works that meet diverse needs. This method expands the possibility of music creation, allowing creators to express their creative intentions more freely without being restricted by traditional music production conditions. At the same time, for ordinary users, they can also easily customize exclusive music according to their preferences, which enhances the sense of participation and fun in music creation. In practical applications, this technology can be widely used in many fields such as film and television soundtracks, game music production, and advertising music creation, providing strong support for the development of different industries.

[0180] Please refer to Figure 8 , which shows a schematic diagram of the process of music generation in an exemplary embodiment. The process is as follows:

[0181] (a) Obtain the input information required by the music generation model, including: target lyrics to be synthesized (Lyrics), target speech to provide timbre information (Speaker Speech), and target style of music (Style Music)

[0182] (b) Lyrics feature extraction: The target lyrics (Lyrics) are extracted through the pre-trained lyrics encoder (LyricsEncoder);

[0183] (c) Voice feature extraction: The target voice (Speaker Speech) is extracted through the voice encoder (SpeakerEncoder);

[0184] (d) Style feature extraction: The target style music (Style Music) is encoded with a style encoder (StyleEncoder) to extract the target style features;

[0185] (e) Music Generation: Inputting the target timbre features, the target lyrics features, and the target style features into the music generator (Acoustic Model) to obtain a target music representation (Music Embedding);

[0186] (f) The target music representation is converted back into music using a music decoder and vocoder. This disclosure does not limit the types of decoders and vocoders; any decoder and vocoder can be used. The decoder and vocoder have different functions. The decoder further processes the target music representation into an intermediate form that conforms to the logic of musical expression, such as an encoded representation of basic musical elements such as melody and rhythm. It can convert the abstract music representation into musical parameters that can be further interpreted by the vocoder. The vocoder converts the intermediate form output by the decoder into an actual audible audio signal. Based on the input musical parameters, it simulates the sound's timbre, pitch, volume, and other characteristics, ultimately generating audible music.

[0187] Figure 9 This is a block diagram of a music generation model training device according to an exemplary embodiment. The music generation model includes a timbre encoder, a lyrics encoder, a style encoder and a music generator. Figure 9 , the device comprises:

[0188] The sample data acquisition module 910 is configured to acquire sample music, corresponding lyrics text, corresponding style tag and corresponding timbre tag;

[0189] A first training module 920 is configured to perform joint training based on semantic alignment of the lyrics encoder and the preset music encoder based on the sample music and the lyrics text;

[0190] The reference music representation prediction module 930 is configured to input the mel-spectrogram features of the sample music into a trained music encoder for encoding to obtain a reference music representation;

[0191] The music prediction module 940 is configured to input the sample music into the timbre encoder and the style encoder to obtain sample timbre features and sample style features, respectively; input the lyrics text into the trained lyrics encoder to obtain sample lyrics features; and input the sample timbre features, the sample style features, and the sample lyrics features into the music generator to obtain predicted sample features.

[0192] The second training module 950 is configured to perform training based on the decoupling of timbre and style on the style encoder based on the sample style features, the style label and the timbre label; and the music generator is trained based on the difference between the predicted sample features and the reference music representation.

[0193] In an exemplary embodiment, the second training module 950 is configured to execute:

[0194] Performing timbre classification on the sample style features to obtain a timbre classification result;

[0195] Performing style classification on the style features of the sample to obtain a style classification result;

[0196] Determining a timbre loss based on a difference between the timbre classification result and the timbre label; determining a style loss based on a difference between the style classification result and the style label;

[0197] The parameters of the style encoder are adjusted with the training goal of reducing the style loss and increasing the timbre loss.

[0198] In an exemplary embodiment, the first training module 920 is configured to execute:

[0199] Based on the duration of the sample music, performing phoneme extraction on the sample lyrics to obtain a sample phoneme sequence;

[0200] Performing phoneme encoding on the sample phoneme sequence based on the lyrics encoder to obtain sample phoneme features;

[0201] Determining a mel spectrum feature corresponding to the sample music, inputting the mel spectrum feature into the music encoder for music encoding to obtain a sample music feature;

[0202] Based on the sample phoneme features and the sample music features, performing comparative learning training on the lyrics encoder and the music encoder;

[0203] The number of the sample phoneme codes in the sample phoneme feature is the same as the number of the sample music codes in the sample music feature.

[0204] In an exemplary embodiment, the first training module 920 is configured to execute:

[0205] Based on the sample phoneme features and the sample music features, a lyrics-music alignment matrix is ​​constructed; the sample phoneme code corresponding to the first position and the sample music code corresponding to the first position constitute a positive sample pair, and the sample phoneme code corresponding to the second position and the sample music code corresponding to the second position constitute a negative sample pair, the first position being a position with the same row number and column number in the lyrics-music alignment matrix, and the second position being other positions in the lyrics-music alignment matrix excluding the first position;

[0206] Performing contrastive learning based on the positive sample pair and the negative sample pair, and constructing a contrastive learning loss;

[0207] Based on the contrastive learning loss, parameters of the lyrics encoder and the music encoder are adjusted.

[0208] In an exemplary embodiment, the first training module 920 is configured to execute:

[0209] Parsing the phonemes in the sample lyrics to obtain an initial phoneme sequence;

[0210] The length of the initial phoneme sequence is adjusted based on the duration corresponding to the sample music to obtain the sample phoneme sequence, and the length of the sample phoneme sequence is consistent with the length of the sample music.

[0211] In an exemplary embodiment, the duration corresponding to the sample music includes the duration of each phoneme in the sample lyrics in the sample music. The first training module 920 is configured to execute:

[0212] Based on the duration of each phoneme in the sample music, the initial phoneme sequence is upsampled to obtain the sample phoneme sequence.

[0213] In an exemplary embodiment, the second training module 950 is configured to execute:

[0214] The training of the style encoder based on the decoupling of timbre and style based on the sample style features, the style label and the timbre label is jointly implemented with the training of the music generator based on the difference between the predicted sample features and the reference music representation.

[0215] In an exemplary embodiment, the loss of the training based on timbre and style decoupling is represented by a cross entropy loss, and the difference between the predicted sample features and the reference music representation is represented by a mean square error loss.

[0216] Regarding the device in the above embodiment, the specific manner of each step has been described in detail in the embodiment of the above method and will not be elaborated here.

[0217] Figure 10 FIG. 1 is a block diagram of a music generating device according to an exemplary embodiment. Figure 10 , the device comprises:

[0218] The target data acquisition module 1010 is configured to acquire target speech, target lyrics and sample music;

[0219] The music generation module 1020 is configured to generate target music by inputting the target speech, the target lyrics, and the sample music into a music generation model, wherein the target music has the target lyrics and has a timbre corresponding to the target speech and a musical style corresponding to the sample music;

[0220] The music generation model is trained using the aforementioned music generation model training method.

[0221] In an exemplary embodiment, the music generation module 1020 is configured to execute:

[0222] Inputting the target speech into the timbre encoder of the music generation model to obtain target timbre features;

[0223] Inputting the target lyrics into the lyrics encoder of the music generation model to obtain target lyrics features;

[0224] Inputting the sample music into the style encoder of the music generation model to obtain target style features;

[0225] The target timbre feature, the target lyrics feature and the target style feature are input into the music generator of the music generation model to obtain a target music representation; and the target music is obtained by decoding the target music representation.

[0226] Regarding the device in the above embodiment, the specific manner of each step has been described in detail in the embodiment of the aforementioned method and will not be elaborated here.

[0227] Please refer to Figure 11 , which shows a structural block diagram of a computer device provided by an exemplary embodiment of the present disclosure Figure 1 The computer device may be a terminal. The computer device is used to implement the music generation model training method or music generation method provided in the above embodiment. Specifically:

[0228] Typically, the computer device 1100 includes a processor 1101 and a memory 1102 .

[0229] Processor 1101 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 1101 may be implemented in hardware using at least one of a DSP (Digital Signal Processing), an FPGA (Field Programmable Gate Array), and a PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In one exemplary embodiment, processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content displayed on the display screen. In one exemplary embodiment, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0230] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In an exemplary embodiment, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is configured to be executed by one or more processors to implement the above-mentioned music generation model training method or music generation method.

[0231] In an exemplary embodiment, computer device 1100 may optionally include a peripheral device interface 1103 and at least one peripheral device. Processor 1101, memory 1102, and peripheral device interface 1103 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1103 via a bus, signal lines, or circuit boards. Specifically, the peripheral device includes at least one of a radio frequency circuit 1104, a touchscreen display 1105, a camera assembly 1106, an audio circuit 1107, a positioning assembly 1108, and a power supply 1109.

[0232] Those skilled in the art will understand that Figure 11The structure shown in the figure does not constitute a limitation on the computer device 1100, and the computer device 1100 may include more or fewer components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0233] Please refer to Figure 12 It shows a structural block diagram of a computer device provided by another exemplary embodiment of the present disclosure. Figure 2 The computer device may be a server for executing the above-mentioned music generation model training method or music generation method. Specifically:

[0234] Computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including random access memory (RAM) 1202 and read-only memory (ROM) 1203, and a system bus 1205 connecting system memory 1204 and CPU 1201. Computer device 1200 also includes a basic input / output system (I / O) 1206 that facilitates information transfer between various components within the computer, and a mass storage device 1207 for storing an operating system 1213, application programs 1214, and other program modules 1211.

[0235] The basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209, such as a mouse and keyboard, for user input. Both the display 1208 and the input device 1209 are connected to the central processing unit 1201 via an input / output controller 1120 connected to the system bus 1205. The basic input / output system 1206 may also include an input / output controller 1120 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1120 also provides output to a display screen, printer, or other types of output devices.

[0236] Mass storage device 1207 is connected to central processing unit 1201 via a mass storage controller (not shown) connected to system bus 1205. Mass storage device 1207 and its associated computer-readable media provide non-volatile storage for computer device 1200. In other words, mass storage device 1207 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0237] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media is not limited to the aforementioned types. The aforementioned system memory 1204 and mass storage device 1207 may be collectively referred to as memory.

[0238] According to various embodiments of the present disclosure, the computer device 1200 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1200 may be connected to the network 1212 via the network interface unit 1211 connected to the system bus 1205. Alternatively, the network interface unit 1211 may be used to connect to other types of networks or remote computer systems (not shown).

[0239] The above-mentioned memory also includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned music generation model training method or music generation method.

[0240] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the music generation model training method or the music generation method is implemented.

[0241] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disk. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0242] In an exemplary embodiment, a computer-readable storage medium including program code is also provided, such as a memory including the program code. The program code can be executed by a processor to perform the music generation model training method or the music generation method described above. Alternatively, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0243] In an exemplary embodiment, a computer program product is also provided, including a computer program, which implements the above-mentioned music generation model training method or music generation method when executed by a processor.

[0244] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0245] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A music generation model training method, characterized in that: The music generation model includes a timbre encoder, a lyrics encoder, a style encoder and a music generator, and the method includes: Obtain sample music, corresponding lyrics text, corresponding style tag and corresponding timbre tag; Extracting phonemes from the sample lyrics based on the duration of the sample music to obtain a sample phoneme sequence; Performing phoneme encoding on the sample phoneme sequence based on the lyrics encoder to obtain sample phoneme features; Determining a mel spectrum feature corresponding to the sample music, inputting the mel spectrum feature into a preset music encoder for music encoding to obtain a sample music feature; performing comparative learning training on the lyrics encoder and the music encoder based on the sample phoneme features and the sample music features, wherein the number of sample phoneme codes in the sample phoneme features is the same as the number of sample music codes in the sample music features; Inputting the mel spectrum features of the sample music into a trained music encoder for encoding to obtain a reference music representation; Inputting the sample music into the timbre encoder and the style encoder to obtain sample timbre features and sample style features respectively; Inputting the lyrics text into the trained lyrics encoder to obtain sample lyrics features; Inputting the sample timbre feature, the sample style feature and the sample lyrics feature into the music generator to obtain predicted sample features; Based on the sample style feature, the style label and the timbre label, training the style encoder based on timbre and style decoupling; The music generator is trained based on the difference between the predicted sample features and the reference music representation.

2. The method according to claim 1, characterized in that The training of the style encoder based on timbre and style decoupling based on the sample style feature, the style label and the timbre label includes: Performing timbre classification on the sample style features to obtain a timbre classification result; Performing style classification on the style features of the sample to obtain a style classification result; Determining a timbre loss based on a difference between the timbre classification result and the timbre label; determining a style loss based on a difference between the style classification result and the style label; The parameters of the style encoder are adjusted with the training goal of reducing the style loss and increasing the timbre loss.

3. The method according to claim 2, characterized in that The performing comparative learning training on the lyrics encoder and the music encoder based on the sample phoneme features and the sample music features includes: Based on the sample phoneme features and the sample music features, a lyrics-music alignment matrix is ​​constructed; the sample phoneme code corresponding to the first position and the sample music code corresponding to the first position constitute a positive sample pair, and the sample phoneme code corresponding to the second position and the sample music code corresponding to the second position constitute a negative sample pair, the first position being a position with the same row number and column number in the lyrics-music alignment matrix, and the second position being other positions in the lyrics-music alignment matrix excluding the first position; Performing contrastive learning based on the positive sample pair and the negative sample pair, and constructing a contrastive learning loss; Based on the contrastive learning loss, parameters of the lyrics encoder and the music encoder are adjusted.

4. The method according to claim 1, wherein The extracting phonemes from the sample lyrics based on the duration corresponding to the sample music to obtain a sample phoneme sequence includes: Parsing the phonemes in the sample lyrics to obtain an initial phoneme sequence; The length of the initial phoneme sequence is adjusted based on the duration corresponding to the sample music to obtain the sample phoneme sequence, and the length of the sample phoneme sequence is consistent with the length of the sample music.

5. The method according to claim 4, characterized in that The duration corresponding to the sample music includes the duration of each phoneme in the sample lyrics in the sample music, and adjusting the length of the initial phoneme sequence based on the duration corresponding to the sample music to obtain the sample phoneme sequence includes: Based on the duration of each phoneme in the sample music, the initial phoneme sequence is upsampled to obtain the sample phoneme sequence.

6. The method according to any one of claims 1 to 5, characterized in that The style encoder is trained based on the decoupling of timbre and style based on the sample style feature, the style label and the timbre label; Training the music generator based on the difference between the predicted sample features and the reference music representation includes: The training of the style encoder based on the decoupling of timbre and style based on the sample style features, the style label and the timbre label is jointly implemented with the training of the music generator based on the difference between the predicted sample features and the reference music representation.

7. The method according to claim 6, characterized in that The loss of the training based on timbre and style decoupling is characterized by cross entropy loss, and the difference between the predicted sample features and the reference music representation is characterized by mean square error loss.

8. A music generation method, characterized in that: The method comprises: Obtain target voice, target lyrics and sample music; By inputting the target speech, the target lyrics and the sample music into a music generation model, a target music is obtained, wherein the lyrics of the target music are the target lyrics, and the target music has a timbre corresponding to the target speech and a musical style corresponding to the sample music; Wherein, the music generation model is trained using the music generation model training method described in any one of claims 1 to 7.

9. The method according to claim 8, characterized in that The target music is obtained by inputting the target speech, the target lyrics and the sample music into a music generation model, including: Inputting the target speech into the timbre encoder of the music generation model to obtain target timbre features; Inputting the target lyrics into the lyrics encoder of the music generation model to obtain target lyrics features; Inputting the sample music into the style encoder of the music generation model to obtain target style features; The target timbre feature, the target lyrics feature and the target style feature are input into the music generator of the music generation model to obtain a target music representation; and the target music is obtained by decoding the target music representation.

10. A music generation model training device, characterized in that: The music generation model includes a timbre encoder, a lyrics encoder, a style encoder and a music generator, and the device includes: A sample data acquisition module is configured to acquire sample music, corresponding lyrics text, corresponding style tag and corresponding timbre tag; The first training module is configured to perform phoneme extraction on the sample lyrics based on the duration corresponding to the sample music to obtain a sample phoneme sequence; perform phoneme encoding on the sample phoneme sequence based on the lyrics encoder to obtain a sample phoneme feature; determine the mel-spectrogram feature corresponding to the sample music, input the mel-spectrogram feature into a preset music encoder for music encoding to obtain a sample music feature; and perform comparative learning training on the lyrics encoder and the music encoder based on the sample phoneme feature and the sample music feature, wherein the number of sample phoneme codes in the sample phoneme feature is the same as the number of sample music codes in the sample music feature; A reference music representation prediction module is configured to input the mel spectrum features of the sample music into a trained music encoder for encoding to obtain a reference music representation; The music prediction module is configured to input the sample music into the timbre encoder and the style encoder to obtain sample timbre features and sample style features respectively; input the lyrics text into the trained lyrics encoder to obtain sample lyrics features; and input the sample timbre features, the sample style features, and the sample lyrics features into the music generator to obtain predicted sample features. The second training module is configured to perform training on the style encoder based on the decoupling of timbre and style based on the sample style features, the style label and the timbre label; and the music generator is trained based on the difference between the predicted sample features and the reference music representation.

11. A music generating device, characterized in that: The device comprises: A target data acquisition module is configured to acquire target speech, target lyrics and sample music; a music generation module configured to obtain target music by inputting the target speech, the target lyrics, and the sample music into a music generation model, wherein the target music has the target lyrics and has a timbre corresponding to the target speech and a musical style corresponding to the sample music; Wherein, the music generation model is trained using the music generation model training method described in any one of claims 1 to 7.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the music generation model training method according to any one of claims 1 to 7, or the music generation method according to claim 8 or 9.

13. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the music generation model training method as described in any one of claims 1 to 7, or the music generation method as described in claim 8 or 9.

14. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a readable storage medium. At least one processor of a computer device reads and executes the computer program from the readable storage medium, so that the device executes the music generation model training method as described in any one of claims 1 to 7, or the music generation method as described in claim 8 or 9.

Images