Melody generation method and device, electronic equipment and storage medium

Abstract

The present application provides a melody generation method, device, electronic equipment and storage medium, wherein the method comprises: performing lyric feature coding on a lyric text to obtain lyric features; and performing note decoding on each to-be-decoded note based on the lyric features and a corresponding lyric sentence and / or lyric word of the to-be-decoded note in the lyric text to obtain a note sequence corresponding to the lyric text. The method, device, electronic equipment and storage medium provided by the present application perform note decoding on each to-be-decoded note based on the corresponding lyric sentence and / or lyric word of the to-be-decoded note in the lyric text. Since the corresponding relationship between the notes and the lyric words is referred to in the decoding process, each note in the decoded note sequence can be strictly aligned with the words in the lyric, thereby ensuring that the melody generated based on the lyric can be more consistent with the lyric itself and improving the reliability of the melody generation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a melody generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] The task of generating lyrics into melody is a highly challenging research topic in the field of artificial intelligence composition.

[0003] Currently, the task of generating lyrics to melodies is mainly achieved through two types of methods. One type is a purely data-driven end-to-end modeling approach, which improves the quality of generated melodies by optimizing neural network structures and musical representations. The other type utilizes musical knowledge and uses common musical elements such as rhythm to bridge lyrics and melodies, breaking down the end-to-end modeling of lyrics and melodies into a two-stage modeling learning process: lyrics to musical elements and musical elements to melodies.

[0004] However, none of the methods mentioned above can learn the strict alignment relationship between the words in the lyrics and the notes in the melody, which directly leads to a low degree of matching between the generated melody and the lyrics. Summary of the Invention

[0005] This invention provides a melody generation method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies where the matching degree between generated melodies and lyrics is not high.

[0006] This invention provides a melody generation method, comprising:

[0007] Lyric features are obtained by encoding the lyrics text.

[0008] Based on the lyric features and the corresponding lyric phrases and / or lyric words of each note to be decoded in the lyric text, the notes to be decoded are decoded to obtain the note sequence corresponding to the lyric text.

[0009] According to a melody generation method provided by the present invention, the step of decoding each note to be decoded based on the lyric features and the corresponding lyric phrases and / or lyric words in the lyric text to obtain a note sequence corresponding to the lyric text includes:

[0010] Based on the corresponding lyric phrases and / or lyric characters in the lyrics text for each note to be decoded, determine the alignment correlation between each note to be decoded and each lyric phrase and / or lyric character in the lyrics text;

[0011] Based on the alignment correlation and the lyric features, the lyric text is obtained by decoding each note to be decoded.

[0012] According to a melody generation method provided by the present invention, determining the alignment correlation between each note to be decoded and each lyric phrase in the lyrics text based on the corresponding lyric phrase in the lyrics text includes:

[0013] If any lyric phrase is the lyric phrase corresponding to the note to be decoded, then the correlation between the sentence feature of the lyric phrase in the lyric features and the note state of the note to be decoded is calculated as the alignment correlation between the note to be decoded and any lyric phrase;

[0014] Otherwise, the alignment correlation between the note to be decoded and any of the lyrics is determined as a first preset value.

[0015] According to a melody generation method provided by the present invention, determining the alignment correlation between each note to be decoded and each lyric character in the lyrics text based on the corresponding lyric character in the lyrics text includes:

[0016] If any lyric character is the lyric character corresponding to the note to be decoded, then the alignment correlation between the note to be decoded and any lyric character is determined based on the number of characters in the lyric character corresponding to the decoded note;

[0017] Otherwise, the alignment correlation between the note to be decoded and any of the lyrics words is determined as the second preset value.

[0018] According to a melody generation method provided by the present invention, the step of decoding each note to be decoded based on the alignment correlation and the lyric features to obtain the note sequence corresponding to the lyric text includes:

[0019] Based on at least one of the lyric emotion information, lyric density information, and lyric tone information of the lyric text, as well as the alignment correlation and the lyric features, the lyric to be decoded is decoded to obtain the lyric text corresponding to the lyric text.

[0020] According to a melody generation method provided by the present invention, the step of decoding each note to be decoded includes:

[0021] Each note to be decoded is decoded within the musical hierarchy structure;

[0022] The musical hierarchy includes progressively refined sections, phrases, measures, beats, and time steps.

[0023] According to a melody generation method provided by the present invention, the step of decoding each note to be decoded based on the lyric features and the corresponding lyric phrases and / or lyric words in the lyric text to obtain a note sequence corresponding to the lyric text includes:

[0024] Based on the generative model, the lyric features are applied, and the corresponding lyric phrases and / or lyric words of each note to be decoded in the lyric text are used to decode each note to obtain the note sequence corresponding to the lyric text.

[0025] The generation model is trained based on sample lyrics text and the corresponding sample note sequence, where each note in the sample note sequence carries an alignment label with the lyrics phrases and / or lyrics words in the sample lyrics text.

[0026] The present invention also provides a melody generation device, comprising:

[0027] The encoding unit is used to encode the lyrics text to obtain lyrics features;

[0028] The decoding unit is used to perform note decoding on each note to be decoded based on the lyric features and the corresponding lyric phrases and / or lyric words in the lyric text, so as to obtain the note sequence corresponding to the lyric text.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the melody generation method as described above.

[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the melody generation method as described above.

[0031] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the melody generation method as described above.

[0032] The melody generation method, apparatus, electronic device, and storage medium provided by this invention perform note decoding on each note to be decoded based on the corresponding lyric phrases and / or lyric characters in the lyrics text. Since the correspondence between notes and lyric phrases is referenced during the decoding process, each note in the decoded note sequence can be strictly aligned with the characters in the lyrics, thereby ensuring that the melody generated based on the lyrics can better fit the lyrics themselves and improving the reliability of melody generation. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is one of the flowcharts illustrating the melody generation method provided by the present invention;

[0035] Figure 2 This is a flowchart illustrating the decoding method for note sequences provided by the present invention;

[0036] Figure 3 This is a schematic diagram of the music hierarchy structure provided by the present invention;

[0037] Figure 4 This is the second flowchart of the melody generation method provided by the present invention;

[0038] Figure 5 This is a schematic diagram of the melody generation device provided by the present invention;

[0039] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] The task of generating melody from lyrics requires not only learning the coupling relationship between lyrics and melody, but also learning the strict alignment relationship between words and notes in the lyrics. To ensure that the notes in the generated melody are strictly aligned with the words in the lyrics, this invention provides a melody generation method. Figure 1 This is one of the flowcharts illustrating the melody generation method provided by the present invention, such as... Figure 1 As shown, the method includes:

[0042] Step 110: Encode the lyrics text using lyrics features to obtain lyrics features.

[0043] Specifically, the lyrics text is the lyrics that need to be used to generate the melody. The lyrics text can include multiple lines of lyrics, referred to here as lyrics lines. Each lyrics line can include one or more words, referred to here as lyrics words.

[0044] For lyric text, lyric feature encoding can be used to obtain lyric features that reflect the information contained in the lyrics. Here, lyric features refer to the lyric text itself. Corresponding to the text structure of the lyric text, lyric features can also include sentence features of multiple lyric phrases, each sentence feature reflecting the information contained in the corresponding lyric phrase at the sentence level; in addition, each sentence feature can also include one or more word features of lyric characters, each word feature reflecting the information contained in the corresponding lyric character at the word level.

[0045] The lyrics feature encoding here can be implemented using models such as Transformer and BERT (Bidirectional Encoder Representation from Transformers). Alternatively, each high-dimensional lyric sentence in the lyrics text can be mapped to a low-dimensional latent space, and then sampled in the latent space to obtain a compact representation vector of each lyric sentence as the sentence feature of each lyric sentence. This embodiment of the invention does not impose specific limitations on this.

[0046] Step 120: Based on the lyric features and the lyric phrases and / or lyric words corresponding to each note to be decoded in the lyric text, perform note decoding on each note to be decoded to obtain the note sequence corresponding to the lyric text.

[0047] Specifically, in order to ensure that the notes in the generated melody are strictly aligned with the words in the lyrics, during the process of decoding notes based on lyric features to obtain the note sequence representing the melody, the corresponding lyric phrases and / or lyric words in the lyric text for each note to be decoded can be referenced.

[0048] Here, the note to be decoded refers to the note that needs to be decoded one by one during the note decoding process. When decoding the current note to be decoded, the corresponding lyric phrase and / or lyric word in the lyrics text is clear. Therefore, the sentence-level features and / or word-level features of the lyric phrase corresponding to the current note to be decoded can be determined from the lyric features. Then, when decoding the note, the sentence-level features and / or word-level features of the lyric phrase corresponding to the current note to be decoded can be referenced, while the interference of sentence-level features and / or word-level features of lyric phrases and / or lyric words that do not correspond to the current note to be decoded can be filtered out, thereby decoding the note that can better fit the lyric words in the lyric phrase.

[0049] According to this rule, by decoding each note individually, a note sequence including multiple notes can be obtained, which is the melody corresponding to the lyrics.

[0050] The method provided in this invention decodes each note based on the corresponding lyric phrases and / or lyric characters in the lyrics text. Since the correspondence between the notes and the lyric phrases is referenced during the decoding process, each note in the decoded note sequence can be strictly aligned with the characters in the lyrics, thereby ensuring that the melody generated based on the lyrics can better fit the lyrics themselves and improving the reliability of melody generation.

[0051] Based on the above embodiments, Figure 2 This is a flowchart illustrating the decoding method for note sequences provided by the present invention, as shown below. Figure 2 As shown, step 120 includes:

[0052] Step 121: Based on the corresponding lyric phrases and / or lyric characters in the lyrics text for each note to be decoded, determine the alignment correlation between each note to be decoded and each lyric phrase and / or lyric character in the lyrics text.

[0053] Specifically, when decoding a note, the alignment correlation between the note to be decoded and each lyric phrase and / or each lyric character in the lyrics text can be intervened based on the correspondence between the note to be decoded and the lyric phrases and / or lyric characters in the lyrics text.

[0054] The alignment relevance here refers to the relevance determined by whether the note to be decoded is aligned with the lyrics and / or words. It can be understood that the alignment relevance between the note to be decoded and its corresponding lyrics and / or words is necessarily higher than the alignment relevance between the note to be decoded and non-corresponding lyrics and / or words. For example, given the pre-calculated relevance between the note to be decoded and each lyrics and / or word in the lyrics text, the calculated relevance can be enhanced, maintained, or weakened based on whether the note to be decoded corresponds to that lyrics and / or word. Alternatively, the relevance between the note to be decoded and its corresponding lyrics and / or words can be calculated as the alignment relevance, while the alignment relevance between the note to be decoded and non-corresponding lyrics and / or words can be set to zero.

[0055] Step 122: Based on the alignment correlation and the lyrics features, perform note decoding on each note to be decoded to obtain the note sequence corresponding to the lyrics text.

[0056] Specifically, after obtaining the alignment correlation between the note to be decoded and each lyric phrase and / or each lyric character in the lyrics text, the note to be decoded can be decoded by combining the alignment correlation and lyric features, and the decoded note can be placed into a note sequence. In the specific decoding process, the attention weight of the note to be decoded relative to each lyric phrase and / or each lyric character can be calculated based on the alignment correlation between the note to be decoded and each lyric phrase and / or each lyric character in the lyrics text. Based on the attention weight, the note state of the note to be decoded is fused with the sentence-level features of each lyric phrase and / or the character-level features of each lyric character in the lyrics features, and the note decoding is performed based on the fused features.

[0057] The method provided in this invention intervenes in the alignment correlation between the notes to be decoded and the lyric phrases and / or lyric characters in the lyric text by exploiting the correspondence between the notes to be decoded and the lyric phrases and / or lyric characters in the lyric text. This guides the decoding of the notes, ensuring that each note in the decoded note sequence is strictly aligned with the characters in the lyrics, thus making the melody generated based on the lyrics more closely match the lyrics themselves.

[0058] Based on any of the above embodiments, in step 121, determining the alignment correlation between each note to be decoded and each lyric phrase in the lyrics text based on the corresponding lyric phrase in the lyrics text includes:

[0059] If any lyric phrase is the lyric phrase corresponding to the note to be decoded, then the correlation between the sentence feature of the lyric phrase in the lyric features and the note state of the note to be decoded is calculated as the alignment correlation between the note to be decoded and any lyric phrase;

[0060] Otherwise, the alignment correlation between the note to be decoded and any of the lyrics is determined as a first preset value.

[0061] Specifically, the calculation of the alignment correlation between the note to be decoded and the lyrics can be performed based on whether the lyrics are the lyrics corresponding to the note to be decoded:

[0062] Specifically, when the lyric phrase corresponds to the note to be decoded, the sentence features of the lyric phrase can be located from the lyric features. The correlation between the sentence features of the lyric phrase and the note state of the note to be decoded can then be calculated, and the calculated correlation is used as the alignment correlation between the two. Here, the note state of the note to be decoded can be determined based on the decoding result of the previous note. That is, when the lyric phrase corresponds to the note to be decoded, the correlation calculation between the lyric phrase and the note to be decoded can be left uninterrupted, and the correlation between the two can be directly used as the alignment correlation.

[0063] If the lyrics do not correspond to the lyrics of the note to be decoded, the alignment correlation between the two can be directly set to the first preset value. It is understood that this first preset value is a pre-defined low correlation value; for example, the first preset value could be negative infinity. By setting the alignment correlation to the first preset value, direct intervention can be achieved regarding the alignment correlation between the note to be decoded and the lyrics where there is no corresponding relationship.

[0064] For example, the alignment correlation between the i-th note to be decoded and the j-th lyric phrase can be expressed as:

[0065]

[0066]

[0067]

[0068] In the formula, x j For the j-th lyric, y i For the i-th note to be decoded, ID(x) j ) and ID(y i ) represent x respectively j and y i Alignment identifier, ID(y) i ) = ID(x j That is, x j For y i The corresponding lyric, ID(y) i )≠ID(x j That is, x j Not for y i The corresponding lyric, M(i,j), is based on x. j and y i Does it correspond to a specific mask element?

[0069] Used to calculate x j With y i The correlation between them, among which For y i The state of the notes, For x j Sentence features in lyric characteristics, W Q W K and d z The parameters are pre-learned; under the intervention of M(i,j), f(i,j) reflects the influence of x. j and y i Whether it corresponds to the intervention. A(i,j) is the relevance after normalizing f(i,j), which is x. j With y iThe alignment correlation between them can also be understood as x j With y i Attention scores between them.

[0070] The method provided in this invention calculates alignment relevance based on whether a lyric is the lyric corresponding to the note to be decoded. This allows more attention to be paid to the sentence features of the lyric corresponding to the note to be decoded when decoding the note based on the alignment relevance, thereby improving the alignment between the note and the lyrics.

[0071] Based on any of the above embodiments, in step 121, determining the alignment correlation between each note to be decoded and each lyric character in the lyrics text based on the corresponding lyric character in the lyrics text includes:

[0072] If any lyric character is the lyric character corresponding to the note to be decoded, then the alignment correlation between the note to be decoded and any lyric character is determined based on the number of characters in the lyric character corresponding to the decoded note;

[0073] Otherwise, the alignment correlation between the note to be decoded and any of the lyrics words is determined as the second preset value.

[0074] Specifically, the calculation of the alignment correlation between the note to be decoded and the lyrics can be performed based on whether the lyrics are the lyrics corresponding to the note to be decoded:

[0075] Specifically, when the lyric character corresponds to the lyric character of the note to be decoded, the number of characters in the lyric character corresponding to the note to be decoded can be counted, and the alignment correlation between the lyric character and the note to be decoded can be determined accordingly. It is understood that the more characters in the lyric character corresponding to a note to be decoded, the more diluted the importance of a single lyric character in the decoding of that note will be, i.e., the lower the alignment correlation between a single lyric character and the note to be decoded; conversely, the fewer characters in the lyric character corresponding to a note to be decoded, the less likely the importance of a single lyric character in the decoding of that note will be diluted, i.e., the higher the alignment correlation between a single lyric character and the note to be decoded. Therefore, the reciprocal of the number of characters in the lyric character corresponding to the note to be decoded can be used as the alignment correlation, or the alignment correlation can be determined based on the reciprocal of the number of characters in the lyric character corresponding to the note to be decoded. This embodiment of the invention does not specifically limit this approach.

[0076] When the lyric character is not the same as the lyric character corresponding to the note to be decoded, the alignment correlation between the two can be directly set to the second preset value. It is understood that this second preset value is a pre-set low correlation value, and it is necessarily lower than the alignment correlation determined based on the number of lyric characters corresponding to the note to be decoded; for example, the second preset value can be set to 0. By setting the alignment correlation to the second preset value, direct intervention can be achieved regarding the alignment correlation between lyric characters and notes to be decoded that do not have a corresponding relationship.

[0077] For example, the alignment correlation between the i-th note to be decoded and the k-th lyric word can be expressed as:

[0078]

[0079] In the formula, w(i,k) represents the alignment correlation between the i-th note to be decoded k(i) and the k-th lyric character x(k), and Z is the total number of lyric characters corresponding to the i-th note to be decoded y(i).

[0080] Assuming that the above alignment relevance calculation needs to be implemented through a pre-trained model, the following loss function can be set during model training to ensure that the alignment relevance calculation during model operation is based on whether the lyric character is the lyric character corresponding to the note to be decoded:

[0081]

[0082] In the formula, L att is the loss function, calculated using L2 regularization. Here, M is the number of notes to be decoded, N is the number of characters in the lyrics text, A(i,k) is the alignment correlation between the i-th note to be decoded and the k-th lyric character obtained during model training, and w(i,k) is the preset label for the alignment correlation between the i-th note to be decoded and the k-th lyric character.

[0083] The method provided in this invention calculates alignment relevance based on whether the lyrics character is the same as the lyrics character corresponding to the note to be decoded. This allows more attention to be paid to the character features of the lyrics character corresponding to the note to be decoded when decoding the note based on the alignment relevance, thereby improving the alignment between the note and the lyrics.

[0084] In practical applications, because the relationship between lyrics and melody is weak, a melody can be paired with different lyrics, and a single lyric can be paired with different melodies. This increases the learning difficulty of the neural network for the task of generating melody from lyrics. To address this problem, based on any of the above embodiments, step 122 includes:

[0085] Based on at least one of the lyric emotion information, lyric density information, and lyric tone information of the lyric text, as well as the alignment correlation and the lyric features, the lyric to be decoded is decoded to obtain the lyric text corresponding to the lyric text.

[0086] Specifically, to enhance the coupling strength between lyrics and melody, embodiments of the present invention mine information related to the relationship between lyrics and melody from the lyric text, which is used to control the generation of note sequences during note decoding. The information related to the relationship between lyrics and melody referred to herein includes at least one of lyric emotional information, lyric density information, and lyric pitch information.

[0087] The lyric emotional information is used to represent the emotions contained in the lyric text. Specifically, it can be the category of emotions contained in the lyric text, such as positive, negative, and neutral emotions, or it can be the probability or intensity of each emotion category in the lyric text. This embodiment of the invention does not impose specific limitations on this. The lyric emotional information can be obtained by performing text sentiment analysis on the lyric text, for example, using methods based on sentiment dictionaries, traditional machine learning methods, and deep learning methods. Therefore, based on the lyric emotional information, note decoding control can be performed to achieve global control over the generated note sequence, that is, the melody, at the emotional level, ensuring the consistency between the emotions conveyed by the generated melody and the emotions expressed in the lyric text.

[0088] Lyric density information is used to characterize the number of words in the lyrics. Generally, the more words in the lyrics, the higher the density; conversely, the fewer words, the lower the density. Lyric density information reflects the rhythmic correlation between the lyrics and the melody. Based on this density information, note decoding control can be implemented to control the generated note sequence, i.e., the rhythm of the melody, thereby ensuring that the generated melody matches the lyrics rhythmically.

[0089] Lyric tone information is used to represent the tone of each word in the lyrics text. For example, the tone of each word can be represented using a word embedding model, resulting in a tone representation vector for each word as the lyrics tone information. Considering that in musical works, musical notes are organized sequentially according to certain thought processes, logical relationships, and the tendency of linguistic tones to form a melodic line, the variation in pitch value of each tone, its relative pitch position, and its relative pitch difference directly affect the melody style. By using lyric tone information for note decoding control, the generated note sequence—that is, the melody—can be controlled on the melodic pitch plane, thereby ensuring that the pitch of the generated melody matches the pronunciation of the lyrics.

[0090] Therefore, when decoding musical notes, we can not only refer to the lyric features and alignment relevance, but also refer to at least one of the lyric emotional information, lyric density information and lyric pitch information in the lyric text. This allows us to constrain melody generation from at least one dimension of emotion, rhythm and pitch, thereby reducing the learning difficulty of melody generation and improving the compatibility between the generated melody and the lyric text.

[0091] Furthermore, commonly used neural network structures such as RNNs (Recurrent Neural Networks), VAEs (Variational Auto-Encoders), and GANs (Generative Adversarial Networks) have limited ability to model long sequences. They often break down the melody into short segments and then splice them together. This approach fails to guarantee the continuity and stylistic consistency of the melody, resulting in unsatisfactory performance in long-term melody generation tasks. Based on any of the above embodiments, step 120, which involves decoding each note to be decoded, includes:

[0092] Each note to be decoded is decoded within the musical hierarchy structure;

[0093] The musical hierarchy includes progressively refined sections, phrases, measures, beats, and time steps.

[0094] Specifically, the note decoding method involved here is based on a multi-level music hierarchy structure. Figure 3 This is a schematic diagram of the music hierarchy structure provided by the present invention. Figure 3 The musical hierarchy shown has a layered characteristic, that is, a complete song is composed of multiple sections, a section is composed of multiple phrases, a phrase is composed of multiple measures, and a measure is composed of multiple beats. For example, for a 4 / 4 time song, a measure is composed of 4 beats, and the beats are further composed of smaller units of note values, that is, time steps.

[0095] The embodiments of the present invention can apply a hierarchical structure of music to perform hierarchical modeling and learning of the entire song. Correspondingly, when decoding notes, hierarchical decoding of musical phrases, measures, beats, and time steps can be performed in sequence, thereby improving the structural hierarchy of the generated melody.

[0096] Based on any of the above embodiments Figure 4 This is the second flowchart illustrating the melody generation method provided by the present invention, as shown below. Figure 4As shown, for lyrics text, in addition to extracting the lyrics emotional information, lyrics tone information and lyrics density information, it is also necessary to extract lyrics features through an encoder. The lyrics features here can include the sentence features of each lyric sentence in the lyrics text.

[0097] Considering that the encoder and decoder in the traditional VAE model are both RNN structures, and that RNN structures are poor at modeling long sequences, this invention addresses this problem by replacing the RNN structures in the VAE encoder and decoder with Transformer, which has advantages in modeling long sequences. The VAE-Transformer encoder is thus constructed to encode each lyric sentence in the lyrics text in parallel, and obtains a sentence-level compact representation vector for each lyric sentence as a sentence feature to construct lyrics features.

[0098] The emotional, tonal, and density information of the lyrics, along with other lyrical features, can all be used as control conditions and fed into the word segmentation decoder to control melody generation. It can be understood that the hierarchical decoder here is based on... Figure 3 The musical hierarchy structure is shown, and modeling and learning are carried out sequentially for musical phrases, measures, beats, and time steps.

[0099] Based on any of the above embodiments, in order to ensure that the notes in the generated melody are strictly aligned with the words in the lyrics, in addition to applying the corresponding lyric phrases and / or lyric words in the lyrics text to decode each note to achieve implicit alignment learning, explicit alignment learning can also be achieved by directly incorporating the alignment relationship into the musical representation during model training. That is, step 120 includes:

[0100] Based on the generative model, the lyric features are applied, and the corresponding lyric phrases and / or lyric words of each note to be decoded in the lyric text are used to decode each note to obtain the note sequence corresponding to the lyric text.

[0101] The generation model is trained based on sample lyrics text and the corresponding sample note sequence, where each note in the sample note sequence carries an alignment label with the lyrics phrases and / or lyrics words in the sample lyrics text.

[0102] Specifically, the note decoding in step 120 can be implemented based on a generative model. Here, the generative model can be understood as a decoder that implements note decoding, or as an encoder + decoder that implements lyric feature extraction and note decoding. This embodiment of the invention does not make specific limitations on this.

[0103] The generative model needs to be pre-trained. In the sample note sequence corresponding to the sample lyrics text used for training the generative model, each note carries an alignment label for the lyric phrase and / or lyric word in the sample lyrics text. Correspondingly, during training, the sample lyrics text is input into the initial generative model. Each note in the predicted note sequence corresponding to the sample lyrics text output by the initial generative model typically carries an identifier for the corresponding lyric phrase and / or lyric word in its note representation. For example, the note representation can be a discrete token sequence arranged according to musical phrase, measure, position, pitch, duration, lyric character alignment identifier, and lyric rhythm alignment identifier. Here, the lyric character alignment identifier reflects the identifier of the lyric phrase corresponding to the note, and the lyric rhythm alignment identifier reflects the identifier of the lyric word corresponding to the note.

[0104] Subsequently, the identifiers of the lyrics and / or words corresponding to each note in the predicted note sequence can be compared with the alignment labels of the lyrics and / or words carried by the corresponding notes in the sample note sequence. This generates a loss function to iterate the parameters of the initial generative model, thus obtaining the final generative model. During training, the generative model can explicitly learn the alignment relationship between notes and lyrics and / or words, thereby ensuring that the notes in the generated melody are strictly aligned with the lyrics during subsequent melody generation.

[0105] Based on any of the above embodiments, for training the generative model, the relationship between lyrics and melody in terms of pitch, rhythm, and structure can be used to define three types of rewards: pitch, rhythm, and structure. These rewards are added to the score obtained from the generative model test, and the score with added rewards is used as the final score. This constrains the generation process, enables controlled generation of the melody, and improves the fit between the generated melody and the lyrics.

[0106] Based on any of the above embodiments, this invention provides a melody generation method, which includes the following steps:

[0107] First, obtain the lyrics text used to generate the melody;

[0108] Subsequently, the lyrics text was subjected to lyric sentiment analysis, lyric density statistics, and lyric tone encoding to obtain lyric sentiment information, lyric density information, and lyric tone information. In addition, an encoder based on VAE and Transformer was applied to encode each lyric sentence in the lyric text in parallel, thereby obtaining lyric features composed of sentence features of each lyric sentence.

[0109] Subsequently, the emotional information, density information, and tone information of the lyrics, as well as the lyrical features of the lyrics, can all be used as control conditions and fed into a hierarchical decoder built on the music hierarchy structure to decode the notes and obtain the note sequence corresponding to the lyrics.

[0110] During the process of decoding musical notes, for each note to be decoded, the alignment correlation between the note to be decoded and each lyric phrase and / or lyric character in the lyrics text can be determined based on the corresponding lyric phrase and / or lyric character in the lyrics text. The lyrics features are then weighted based on the alignment correlation to obtain the attention features used to decode the note to be decoded. On this basis, the lyrics emotional information, lyrics density information, and lyrics tone information of the lyrics text, as well as the attention features of the note to be decoded, can be combined to perform note decoding to obtain the note representation in the note sequence. Here, the note representation can be a discrete token sequence arranged according to musical phrase, measure, position, pitch, duration, lyric character alignment identifier, and lyric rhythm alignment identifier.

[0111] In the above scheme, by combining VAE and Transformer and using layered decoding, the problem of poor performance in the generation of long melodies in related technologies can be solved, thereby ensuring the coherence and stylistic consistency of the generated melodies.

[0112] By combining the emotional information, density information, and tonal information of the lyrics, melody generation fully explores the relationship between lyrics and melody, and can constrain the emotion, rhythm, and pitch of the melody during melody generation.

[0113] In addition, by referring to the corresponding lyric phrases and / or lyric words in the lyrics text when decoding the notes, the alignment relationship between the notes and the lyrics can be learned implicitly. By adding lyric character alignment marks and lyric phrase rhythm alignment marks to the note representation, the alignment relationship between the notes and the lyrics can be learned explicitly, thereby ensuring that the notes and lyric words in the generated melody are strictly aligned and improving the fit between the melody and the lyrics.

[0114] Based on any of the above embodiments Figure 5 This is a schematic diagram of the melody generation device provided by the present invention, as shown below. Figure 5 As shown, the device includes:

[0115] Encoding unit 510 is used to encode the lyrics text to obtain lyrics features;

[0116] The decoding unit 520 is used to perform note decoding on each note to be decoded based on the lyric features and the lyric phrases and / or lyric words corresponding to each note to be decoded in the lyric text, so as to obtain the note sequence corresponding to the lyric text.

[0117] The apparatus provided in this invention decodes each note based on the corresponding lyric phrases and / or lyric characters in the lyrics text. Since the correspondence between the notes and the lyric phrases is referenced during the decoding process, each note in the decoded note sequence can be strictly aligned with the characters in the lyrics, thereby ensuring that the melody generated based on the lyrics can better fit the lyrics themselves and improving the reliability of melody generation.

[0118] Based on any of the above embodiments, the decoding unit 520 is specifically used for:

[0119] Based on the corresponding lyric phrases and / or lyric characters in the lyrics text for each note to be decoded, determine the alignment correlation between each note to be decoded and each lyric phrase and / or lyric character in the lyrics text;

[0120] Based on the alignment correlation and the lyric features, the lyric text is obtained by decoding each note to be decoded.

[0121] Based on any of the above embodiments, the decoding unit 520 is specifically used for:

[0122] If any lyric phrase is the lyric phrase corresponding to the note to be decoded, then the correlation between the sentence feature of the lyric phrase in the lyric features and the note state of the note to be decoded is calculated as the alignment correlation between the note to be decoded and any lyric phrase;

[0123] Otherwise, the alignment correlation between the note to be decoded and any of the lyrics is determined as a first preset value.

[0124] Based on any of the above embodiments, the decoding unit 520 is specifically used for:

[0125] If any lyric character is the lyric character corresponding to the note to be decoded, then the alignment correlation between the note to be decoded and any lyric character is determined based on the number of characters in the lyric character corresponding to the decoded note;

[0126] Otherwise, the alignment correlation between the note to be decoded and any of the lyrics words is determined as the second preset value.

[0127] Based on any of the above embodiments, the decoding unit 520 is specifically used for:

[0128] Based on at least one of the lyric emotion information, lyric density information, and lyric tone information of the lyric text, as well as the alignment correlation and the lyric features, the lyric to be decoded is decoded to obtain the lyric text corresponding to the lyric text.

[0129] Based on any of the above embodiments, the decoding unit 520 is specifically used for:

[0130] Each note to be decoded is decoded within the musical hierarchy structure;

[0131] The musical hierarchy includes progressively refined sections, phrases, measures, beats, and time steps.

[0132] Based on any of the above embodiments, the decoding unit 520 is specifically used for:

[0133] Based on the generative model, the lyric features are applied, and the corresponding lyric phrases and / or lyric words of each note to be decoded in the lyric text are used to decode each note to obtain the note sequence corresponding to the lyric text.

[0134] The generation model is trained based on sample lyrics text and the corresponding sample note sequence, where each note in the sample note sequence carries an alignment label with the lyrics phrases and / or lyrics words in the sample lyrics text.

[0135] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a melody generation method, which includes: encoding lyrics features into the lyrics text to obtain lyrics features; and, based on the lyrics features and the corresponding lyrics phrases and / or lyrics words of each note to be decoded in the lyrics text, decoding each note to be decoded to obtain a note sequence corresponding to the lyrics text.

[0136] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the melody generation method provided by the above methods. The method includes: encoding lyrics features into the lyrics text to obtain lyrics features; and decoding each note to be decoded based on the lyrics features and the corresponding lyrics phrases and / or lyrics words in the lyrics text to obtain a note sequence corresponding to the lyrics text.

[0138] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the melody generation method provided by the above methods. The method includes: encoding lyrics features into lyrics text to obtain lyrics features; and decoding each note to be decoded based on the lyrics features and the corresponding lyrics phrases and / or lyrics words in the lyrics text to obtain a note sequence corresponding to the lyrics text.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A melody generation method, characterized in that, include: Lyric features are obtained by encoding the lyrics text. Based on the lyric features, and the lyric phrases and / or lyric words corresponding to each note to be decoded in the lyric text, the note to be decoded is decoded to obtain the note sequence corresponding to the lyric text; The step of decoding each note based on the lyric features and the corresponding lyric phrases and / or lyric words in the lyric text to obtain the note sequence corresponding to the lyric text includes: Based on the corresponding lyric phrases and / or lyric characters in the lyrics text for each note to be decoded, determine the alignment correlation between each note to be decoded and each lyric phrase and / or lyric character in the lyrics text; Based on the alignment correlation and the lyric features, the lyric text is obtained by decoding each note to be decoded.

2. The melody generation method according to claim 1, characterized in that, The step of determining the alignment correlation between each note to be decoded and each lyric phrase in the lyrics text based on the corresponding lyric phrase in the lyrics text includes: If any lyric phrase is the lyric phrase corresponding to the note to be decoded, then the correlation between the sentence feature of the lyric phrase in the lyric features and the note state of the note to be decoded is calculated as the alignment correlation between the note to be decoded and any lyric phrase; Otherwise, the alignment correlation between the note to be decoded and any of the lyrics is determined as a first preset value.

3. The melody generation method according to claim 1, characterized in that, The step of determining the alignment correlation between each note to be decoded and each lyric character in the lyrics text based on the corresponding lyric character in the lyrics text includes: If any lyric character is the lyric character corresponding to the note to be decoded, then the alignment correlation between the note to be decoded and any lyric character is determined based on the number of characters in the lyric character corresponding to the decoded note; Otherwise, the alignment correlation between the note to be decoded and any of the lyrics words is determined as the second preset value.

4. The melody generation method according to claim 1, characterized in that, The step of decoding each note to be decoded based on the alignment relevance and the lyrics features to obtain the note sequence corresponding to the lyrics text includes: Based on at least one of the lyric emotion information, lyric density information, and lyric tone information of the lyric text, as well as the alignment correlation and the lyric features, the lyric to be decoded is decoded to obtain the lyric text corresponding to the lyric text.

5. The melody generation method according to any one of claims 1 to 4, characterized in that, The process of decoding each note to be decoded includes: Each note to be decoded is decoded within the musical hierarchy structure; The musical hierarchy includes progressively refined sections, phrases, measures, beats, and time steps.

6. The melody generation method according to any one of claims 1 to 4, characterized in that, The step of decoding each note based on the lyric features and the corresponding lyric phrases and / or lyric words in the lyric text to obtain the note sequence corresponding to the lyric text includes: Based on the generative model, the lyric features are applied, and the corresponding lyric phrases and / or lyric words of each note to be decoded in the lyric text are used to decode each note to obtain the note sequence corresponding to the lyric text. The generation model is trained based on sample lyrics text and the corresponding sample note sequence, where each note in the sample note sequence carries an alignment label with the lyrics phrases and / or lyrics words in the sample lyrics text.

7. A melody generation device, characterized in that, include: The encoding unit is used to encode the lyrics text to obtain lyrics features; The decoding unit is used to perform note decoding on each note to be decoded based on the lyric features and the lyric phrases and / or lyric words corresponding to each note to be decoded in the lyric text, so as to obtain the note sequence corresponding to the lyric text; The decoding unit is specifically used for: Based on the corresponding lyric phrases and / or lyric characters in the lyrics text for each note to be decoded, determine the alignment correlation between each note to be decoded and each lyric phrase and / or lyric character in the lyrics text; Based on the alignment correlation and the lyric features, the lyric text is obtained by decoding each note to be decoded.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the melody generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the melody generation method as described in any one of claims 1 to 6.

Images