A structured lyrics-melody generation method and system based on generative AI

Abstract

The present invention proposes a structured lyrics-melody generation method and system based on generative AI, including two key modules: Theme-to-Lyric (TTL) and Structure Identification and Optimization (SIO). The TTL module uses a large-scale language model to generate high-quality Chinese and English lyrics, and continuously optimizes the lyrics content through interaction with users to meet the personalized needs of users. The SIO module ensures that the generated melody conforms to the basic rules of music theory and enhances the sense of hierarchy and emotional expression of the music work by analyzing and extracting the structural features in the music. The advantage of the present invention is that by introducing the collaborative work of the TTL module and the SIO module, efficient generation and optimization of lyrics and melodies are achieved. The generated music works not only have a reasonable structure and coherence, but also can achieve higher quality standards in practical applications, promoting the intelligent and automated process in the field of music creation.

Description

Technical Field

[0001] The present invention relates to an AI composition generation technology, and in particular to a structured lyrics-melody generation method Chat Structure Transformer (CST) based on generative AI. Background Art

[0002] With the continuous advancement of artificial intelligence technology, AI composition technology has attracted widespread attention in the field of music creation. Traditional music creation relies on the personal experience and inspiration of the composer, which is time-consuming and has creative bottlenecks. In recent years, music generation technology based on machine learning and deep learning has developed rapidly, providing new possibilities for automated and intelligent music creation. Existing AI composition technology mainly focuses on the generation of lyrics and melody. However, there are usually the following problems: First, the lyrics usually need to be specified in advance, and the technology of automatic lyrics generation combined with melody generation is not mature enough. This deficiency makes the generated music works have great limitations in practical applications. Secondly, most AI composition systems lack a deep understanding of the overall structure of music. They often only focus on local melody or chord generation, while ignoring the rationality and coherence of the overall structure of the music work (such as the verse and chorus). This limitation makes it difficult for the generated music works to meet the expected standards in terms of melody fluency, harmony, layering, and emotional expression, affecting the practical application value of the music works. Summary of the invention

[0003] To solve the above problems, the present invention proposes an innovative structured lyrics-melody generation method based on generative AI, Chat Structure Transformer (CST), which includes the following steps:

[0004] Step 1, generating lyrics that match the theme specified by the user;

[0005] Step 2, using the lyrics obtained in step 1, obtaining tonality and rhythm information by analyzing the lyrics information, then obtaining rhythm pattern information through an existing lyrics-rhythm model, and combining the chord data provided by the user, storing the extracted tonality, chords, rhythm pattern and rhythm in the form of sequence data as subsequent input data;

[0006] Step 3, collecting a training data set, the training data set includes a template sequence consisting of four musical elements: tonality, chords, rhythmic patterns, and rhythms, and a melody sequence consisting of corresponding melody tracks;

[0007] Step 4: Input the training data set into the Structure Transformer model based on the Transformer improvement. The model is an encoder-decoder structure. The encoder part is responsible for encoding the input data, i.e., the template sequence data, and converting it into an internal representation. The decoder part generates the corresponding melody sequence data based on these internal representations.

[0008] Step 5: Design a loss function to optimize the Structure Transformer model.

[0009] Step 6: Input the data in step 2 into the trained Structure Transformer model to generate lyrics-melody.

[0010] Furthermore, in step 1, ChatGPT is used to generate lyrics that satisfy the user and conform to the user-specified theme.

[0011] Furthermore, in step 2, the acquisition of tonality is achieved by analyzing the data in the lyrics through the text sentiment model. The tonality is divided into major and minor, corresponding to the positive and negative emotions of the lyrics respectively;

[0012] The rhythm is obtained based on the punctuation marks in the lyrics, and the lyrics are mapped as follows: the commas and periods in the lyrics are transformed into half-stops and stops to indicate the end of the sentence;

[0013] By introducing an existing Transformer-based lyrics-rhythm model to obtain the rhythmic pattern, the rhythmic pattern in a given lyrics template is predicted in an autoregressive manner.

[0014] Furthermore, the specific implementation of step 3 is as follows;

[0015] Step 3.1, prepare MIDI music dataset;

[0016] Step 3.2, process MIDI files: First, extract melody track events from multi-track MIDI files and encode these melody tracks; Next, filter the segments according to the following principles: segments whose time signatures change more than once in the entire melody, segments whose length exceeds the time limit, and segments whose time units are not granular with sixteenth notes; After filtering, the segments that meet the requirements will form a melody sequence dataset;

[0017] Step 3.3, extract the four elements of tonality, chord progression, rhythmic pattern and rhythm: first, infer the tonality based on the note pitch distribution of the melody in the melody sequence dataset; second, infer the chords based on the note pitch distribution through the Viterbi algorithm; infer the rhythmic pattern based on the position information of the notes; finally, infer the rhythm based on the pitch and duration rules;

[0018] Step 3.4, based on the obtained music elements, the note elements are digitally represented as multidimensional tensors to form sequence data in the template, and these sequence data are aligned with the previous music melody sequence data set to form a corresponding template-melody data set, that is, the training data set.

[0019] Furthermore, the Structure Transformer model introduces the structure identification and optimization module SIO based on the Transformer. This module is located between the position encoding and layer normalization. The specific implementation method is as follows;

[0020] (1) Let the input of SIO be the sequence X = [x 1 ,x 2 ,x 3 ,…,x N ]∈R N*Dx , is the sequence after position encoding, where N represents the sequence length and Dx represents the dimension of each music element; first, the input sequence is respectively compared with the weight matrix W q ∈R N*Dk , W k ∈R N*Dk , W v ∈R N*Dv Perform a dot product operation to obtain the query vector matrix Q = [q 1 ,q 2 ,q 3 ,…,q N ], key vector matrix K = [k 1 ,k 2 ,k 3 ,…,k N ] and value vector matrix V = [v 1 ,v 2 ,v 3 ,…,v N ]; then calculate the self-attention output of X, using the following formula:

[0021]

[0022] At this time, Z=[z 1 ,z 2 ,z 3 ,…,z N ] represents the output calculated by the attention mechanism. The output here is the output of a self-attention head. Since a multi-head self-attention mechanism is used, that is, multiple groups of W are used q , W k and W vGet multiple groups of Q, K, and V, then calculate multiple Z matrices respectively, and finally concatenate the multiple Z matrices to get the output of the multi-head self-attention mechanism;

[0023] (2) Next, we calculate the similarity matrix of the input sequence to help identify the structure of the melody. The similarity matrix is ​​calculated by the Euclidean distance between the elements of the input sequence. By calculating the similarity matrix for each element x in the sequence X i The similarity with other elements finally results in an N*N similarity matrix; the calculation formula is as follows:

[0024]

[0025] Here, S ij Represents the sequence x i and x j The similarity between them, euclidean_distance(x i , x j ) represents x i and x j The Euclidean distance between them; where ε is a constant to avoid the situation where the denominator is zero; then, the similarity matrix is ​​normalized;

[0026] (3) The output of the SIO module is obtained through the dot product operation of the similarity matrix and the self-attention mechanism and the softmax calculation.

[0027] Furthermore, the total loss function used in the training of the Structure Transformer model includes label smoothed cross entropy loss function, alignment loss function, and reinforcement loss function;

[0028] Loss=LSCELoss+β*AlignLoss+γ*ReinLoss

[0029] Among them, the parameters β and γ before the alignment loss AlignLoss and the reinforcement learning loss function ReinLoss in the total loss function Loss are adjustable, and LSCELoss is the label smoothed cross entropy loss function;

[0030] The calculation formula of the label smoothed cross entropy loss function is as follows:

[0031]

[0032] Where N is the number of label categories, P(y pred ) is the probability of the model predicting the true label of the tth token of the nth input sequence, x n：Tn is the nth input sequence, P(y n,t ∣x n:Tn) is given an input sequence x n：Tn Under the condition, the probability of the tth token in the target sequence, T n is the number of tokens in the nth sequence and ∈ is the smoothing parameter.

[0033] Furthermore, the expression of sequence-to-sequence alignment loss is as follows:

[0034]

[0035] Among them, y n,t is the tth token in the nth target sequence, x n,t is the token in the corresponding input sequence, attn is the multi-head self-attention distribution value in Structure Transformer, and align_weights is the inverse of the frequency of the target melody, which is used for normalization.

[0036] Furthermore, the formula for the reinforcement loss is as follows:

[0037]

[0038] Among them, P(y n ) is the probability of the true label, is the given reward value.

[0039] Furthermore, the method further includes step 7, wherein the quality of the generated melody is evaluated by using the similarity of pitch distribution, the similarity of tone length distribution, the starting interval between melodies, the pitch diversity, and the tone length diversity, wherein the calculation formulas for the pitch diversity and the tone length diversity are as follows:

[0040] First, extract the pitch information from the generated melody and convert the melody into a series of notes, each of which contains pitch and duration information; then count the number of different pitch values, denoted as N pitch , the total number of pitches is recorded as T pitch , the calculation formula of pitch diversity PD is as follows:

[0041]

[0042] Similarly, note duration information is extracted from the generated melody, the number of different note duration values ​​is counted, and then divided by the total number of note durations in the entire melody to obtain the note duration diversity.

[0043] The present invention also provides a structured lyrics-melody generation system based on generative AI, comprising the following modules:

[0044] A lyrics generation module, used to generate lyrics that match the theme specified by the user;

[0045] An input data acquisition module is used to obtain tonality and rhythm information by analyzing the lyrics information using the lyrics obtained by the lyrics generation module, and then obtain rhythm pattern information through an existing lyrics-rhythm model, and at the same time, combined with the chord data provided by the user, the extracted tonality, chords, rhythm pattern and rhythm are stored in the form of sequence data as subsequent input data;

[0046] A training data set collection module is used to collect a training data set, wherein the training data set includes a template sequence consisting of four musical elements, namely, tonality, chords, rhythmic patterns and rhythms, and a melody sequence consisting of corresponding melody tracks;

[0047] The Structure Transformer model building module inputs the training data set into the Structure Transformer model based on the Transformer improvement. The model is an encoder-decoder structure. The encoder part is responsible for encoding the input data, i.e., the template sequence data, and converting it into an internal representation. The decoder part generates the corresponding melody sequence data based on these internal representations.

[0048] Model training module, used to design loss function to optimize Structure Transformer model;

[0049] The melody generation module is used to input the data in the input data acquisition module into the trained Structure Transformer model to generate lyrics-melody.

[0050] The present invention has the following positive effects and advantages:

[0051] 1) Efficient and accurate lyrics-melody generation: This invention adopts an innovative Chat Structure Transformer (CST) framework, which has excellent lyrics-melody generation performance. Through the collaborative work of TTL and SIO modules, lyrics that meet the user's specified themes and needs can be quickly and accurately generated, and melodies with coherence and structure can be generated. This advantage is particularly important in the field of music creation, because in these fields, the efficient and accurate generation of high-quality music works can significantly improve the creation efficiency and quality of the works, and meet the diverse needs of creators.

[0052] 2) The model is lightweight and widely applicable: The limitations of computing and storage resources are fully considered in the design process of this invention, so an efficient and lightweight model is adopted, which is applicable to various computing environments, including smart mobile terminal devices. This feature enables the CST method to be widely used in practical applications, which not only improves the practicality of the technology, but also reduces hardware resources and computing costs, making it more feasible and economical in the field of music creation.

[0053] 3) Structural Identification and Optimization to Improve Generation Quality: The significant advantage of the framework of the present invention lies in its innovative Structure Identification and Optimization (SIO) module. The SIO module uses structural recognition technology to analyze and extract structural features in music (such as chorus and verse), ensuring that the generated melody not only conforms to the basic rules of music theory, but also enhances the emotional expression and artistic value of the music work. The introduction of the SIO module enables the generated melody to reach higher quality standards in practical applications and improves the overall coherence and layering of the generated music.

[0054] 4) User interaction optimizes the generated results: The TTL module allows users to interact with the system. After the user provides the theme and specific requirements, the system generates the corresponding lyrics. If the user is not satisfied with the generated lyrics, the lyrics can be adjusted and optimized repeatedly until a satisfactory result is obtained. This advantage ensures that the generated lyrics are more in line with the user's needs, lays a solid foundation for subsequent melody generation, and improves the user experience and the customization of the generated music.

[0055] In general, the present invention significantly improves the efficiency and quality of lyrics-melody generation through the implementation of the CST framework, making the music creation process more intelligent and automated, and has broad practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is an overall flow chart of an embodiment of the present invention.

[0057] Figure 2 The present invention is a flow chart of the dialogue in the theme-to-lyrics generation (TTL) module according to an embodiment of the present invention.

[0058] Figure 3 The figure is a flow chart of the structure identification and optimization (SIO) module in an embodiment of the present invention.

[0059] Figure 4 This is the Structure Transformer network diagram in step 3.2 of the embodiment of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0061] This paper proposes a structured lyrics-melody generation method based on generative AI, named ChatStructure Transformer (CST). The framework consists of two key modules: Theme-to-Lyric (TTL) and Structure Identification and Optimization (SIO). Figure 2 As shown, the TTL module generates high-quality Chinese and English lyrics using a large-scale language model, and continuously optimizes the lyrics content through interaction with users to meet the personalized needs of users. The SIO module ensures that the generated melody conforms to the basic rules of music theory by analyzing and extracting the structural features in the music, and enhances the sense of hierarchy and emotional expression of the music work. The advantage of the present invention is that by introducing the collaborative work of the SIO module and the TTL module, efficient generation and optimization of lyrics and melodies are achieved. The generated music work not only has a reasonable structure and coherence, but also can reach higher quality standards in practical applications, promoting the intelligent and automated process in the field of music creation.

[0062] Among them, the TTL module has the ability to automatically identify and select Chinese or English language models to ensure that generation tasks can be performed efficiently in different language environments. First, due to ChatGPT's powerful dialogue generation capabilities, users can interact with ChatGPT. After the user provides the topic and specific requirements, ChatGPT will generate lyrics that meet these requirements. The generated lyrics will be displayed to the user, and the user can evaluate whether they are satisfied. If the user is satisfied with the generated lyrics, proceed to the subsequent music generation steps; otherwise, the user can ask ChatGPT to continue generating new lyrics until they are satisfied.

[0063] The SIO module is a module specially designed to automatically identify music structure, aiming to improve the quality of lyrics-melody generation. The SIO module analyzes and extracts structural features (chorus, verse) in music through structure recognition technology. This structural information not only helps to maintain the coherence and hierarchy of music when generating melodies, but also enhances the emotional expression and artistic value of musical works. The introduction of the SIO module makes the generated melody not only conform to the basic rules of music theory, but also achieve higher quality standards in practical applications.

[0064] The collaboration between the two modules is achieved through a series of steps to optimize the entire lyrics-melody generation process:

[0065] 1. The TTL module first processes the user's input and generates the corresponding Chinese and English lyrics using ChatGPT, without the user having to specify the lyrics in advance;

[0066] 2. During the lyrics-melody generation process, the SIO module analyzes the feature relationship of the intermediate template and parses the verse and chorus parts in the final melody generation.

[0067] Next, the two modules work together, first through the TTL module, to further improve the convenience of lyrics-melody generation. We use an intermediate template to generate lyrics-melody, and the intermediate template is a musical element (tonality, chords, rhythmic patterns, and rhythm). In the template-melody generation process, we then use the StructureTransformer introduced by the SIO module to generate structured melodic music. The goal of this entire framework is to provide an efficient and accurate lyrics-melody generation method that overcomes the limitations of existing methods. The collaborative work of the two modules (TTL and SIO) ensures the high quality and consistency of the generated lyrics and melodies.

[0068] like Figure 1 As shown, an embodiment of the present invention provides a structured lyrics-melody generation method based on generative AI, comprising the following steps:

[0069] Step 1: Build a network platform and have a conversation with ChatGPT's text model. This method can identify and generate lyrics that match the user's specified theme. If the user is not satisfied with the generated lyrics, they can be generated again to meet their needs.

[0070] Step 2, input data acquisition: First, based on the lyrics that the user is satisfied with generated in step 1, the lyrics information is analyzed to extract the tonality and rhythm information. Next, the rhythm information is extracted using the existing lyrics-rhythm model, and the chord data provided by the user is combined to integrate all the extracted information into the key elements of the template (tonality, chords, rhythmic pattern and rhythm), and stored in the form of sequence data.

[0071] Step 3, training dataset collection: In order to generate a template-melody alignment dataset, you first need to select a suitable music MIDI dataset as the basic dataset. Next, extract the melody track from the MIDI dataset, and use a self-supervised learning method to extract the musical elements: tonality, chords, rhythm patterns, and rhythms to form a music template. Through this process, the collection of the template-melody dataset is completed. The collected dataset is an aligned template-melody (sequence-sequence) dataset.

[0072] Step 4, network structure construction: In template-melody generation, the Structure Transformer model is used, and the Structure Identification and Optimization (SIO) module is introduced to enhance the generalization of the model in the field of melody generation and its ability to generate structured music. The SIO module implements three key strategies on the original Transformer structure. First, the Attention self-attention mechanism is used to obtain the attention score when the target sequence is generated. Secondly, by measuring the similarity between the template source sequences, the more repetitive parts are found, so that when the melody is generated, the highly repetitive parts will be closer, satisfying the repetitive characteristics of the chorus part to convey the musical emotion. Finally, the Attention and similarity matrix operations are performed to ensure the structured generation of the melody;

[0073] Step 5, loss function calculation: After building the network structure in step 4, the loss function is then designed to optimize the training of the Structure Transformer model. The present invention proposes an innovative loss function for optimization processing to generate higher quality and more accurate music melodies;

[0074] Step 6: Lyrics-melody generation. In step 5, the model loss was designed, and the Structure Transformer model was trained using the loss function. Then, the input data obtained in step 2 was obtained and input into the trained Structure Transformer model to generate lyrics-melody.

[0075] Step 7, melody generation evaluation: The MIDI music melody generated by lyrics-melody is obtained through the previous steps. In order to evaluate the melody generation, objective evaluation and subjective evaluation are used to conduct a comprehensive evaluation of the melody generation. Regarding objective evaluation, classic indicators in the field of lyrics-melody generation are used, including: pitch distribution similarity (Pitch Distribution Similarity, PDS), duration distribution similarity (Duration Distribution Similarity, DDS), and inter-onset intervals (IOIs) between melodies. The present invention also proposes indicators designed for this generation task: pitch diversity (Pitch Diversity, PD) and duration diversity (Duration Diversity, DD). Subjective evaluation here includes: fluency (Fluency), naturalness (Naturalness), repeatability (Repeatability) and overall quality (Overall quality, Overall). Through these evaluation indicators, the quality of the generated melody can be comprehensively measured to ensure its feasibility and superiority in practical applications.

[0076] In the above CST method, step 1 of building a network platform and communicating with the ChatGPT text model is the first step in the lyrics-melody generation method, which is used to generate lyrics. For this purpose, the following two sub-steps are designed:

[0077] Step 1.1: First, using ChatGPT's powerful dialogue generation capabilities, ChatGPT is integrated into the lyrics-melody generation system, allowing users to interact with ChatGPT. Users need to provide a topic or specific requirements, and ChatGPT will generate corresponding lyrics based on these inputs. This step allows users to put forward the emotions, themes, or other specific requirements they want the lyrics to convey to ensure that the generated lyrics meet their expectations.

[0078] Step 1.2: After receiving the lyrics generated by ChatGPT, the user can evaluate whether it meets expectations. If the user is satisfied with the generated lyrics, they can proceed to the next step of music generation. If the user is not satisfied, they can ask ChatGPT to regenerate the lyrics. This cyclic process ensures that users can repeatedly adjust and optimize the lyrics until they get satisfactory results. In this way, the generated lyrics are more in line with the needs of users, laying a solid foundation for the subsequent music generation steps.

[0079] In the method of the lyrics-melody generation system CST, in the step 2 of input data acquisition, the data generated by the input model is collected, and the specific steps are as follows:

[0080] Step 2.1: The acquisition of tonality is achieved by analyzing the data in the lyrics through the text sentiment model. The tonality is divided into major and minor, corresponding to the positive and negative emotions of the lyrics respectively. The rhythm is obtained based on the punctuation marks in the lyrics. We map the lyrics as follows: the commas and periods in the lyrics are converted into "half-stop" and "stop" to indicate the end of the sentence. The rhythm is obtained by introducing an existing lyrics-rhythm model based on Transformer to predict the rhythm pattern in a given lyrics template in an autoregressive manner. Finally, in order to maintain user controllability in the lyrics-melody generation process, the user is allowed to specify the chords in advance, and the chords are obtained through user input.

[0081] Step 2.2: Integrate the four musical elements (tonality, chords, rhythmic patterns, and rhymes) obtained in step 2.1 to form a complete template data. This template can not only reflect the emotions of the lyrics to be generated, but also ensure the controllability and personalization of the melody through the chords specified by the user.

[0082] In the method of the lyrics-melody generation system CST, in the step 3 of training data set collection, a data set selection and data processing are performed on the model training data set, and the specific steps are as follows:

[0083] Step 3.1: Prepare a sufficiently large and appropriate MIDI music dataset so that the model can fully learn the composition rules and characteristics of music. In this process, the LMD_full MIDI dataset was selected. The LMD_full MIDI dataset is a widely used music dataset that contains 176,581 MIDI files of complete tracks. Each MIDI file contains a variety of musical elements, such as melody, chords, rhythm, and timbre, which provides rich learning materials for dataset collection.

[0084] Step 3.2: Process MIDI files. First, we need to extract melody track events from multi-track MIDI files and encode these musical elements. Next, we filter the segments according to the following principles: segments whose time signatures change more than once in the entire melody, segments whose length exceeds the time limit, and segments whose time units are not granular with sixteenth notes. After filtering, the segments that meet the requirements will form our melody sequence dataset.

[0085] Step 3.3: Extract four elements from the music template: tonality, chord progression, rhythmic pattern, and rhythm. First, the tonality can be inferred based on the pitch distribution of the notes in the melody dataset; second, the chords can be inferred based on the pitch distribution of the notes through the Viterbi algorithm; the rhythmic pattern can be inferred based on the position information of the notes; finally, the rhythm can be inferred based on the rules such as pitch and duration.

[0086] Step 3.4: Based on the previously obtained music elements, the note elements are digitally represented as multidimensional tensors to form sequence data in the template. These sequence data are aligned with the previous music melody sequence dataset to form the corresponding template-melody dataset. Next, these template data will be encoded and mapped to the Structure Transformer model for learning.

[0087] In the above-mentioned lyrics-melody generation system CST method, in the step 4 of building the network structure, we designed the Structure Transformer model, such as Figure 4 As shown in Figure 1. This model is based on the Transformer architecture, replacing the original multi-head self-attention module with a SIO module designed specifically for melody generation tasks. This module is located between position encoding and layer normalization. The design steps are as follows:

[0088] Step 4.1: The Structure Transformer model is based on the Transformer architecture, and the core of the Structure Transformer model is the encoder-decoder structure. The Encoder part is responsible for encoding the input template data and converting it into internal representations. These internal representations contain the emotional information and music information of the lyrics. The Decoder part generates corresponding melody sequence data based on these internal representations. The input of the Structure Transformer is the template sequence data obtained in step 3.4, and the target sequence is the melody sequence data corresponding to the template sequence data. When designing the attention mechanism of the Encoder part, the present invention introduces a SIO module specifically for this task.

[0089] Step 4.2: If Figure 3 As shown in the figure, the design of the SIO module can effectively capture the structural information in the lyrics and optimize the attention allocation, thereby improving the structure of the model when generating melodies. The specific steps are as follows:

[0090] Step 4.2.1: Let the input sequence of the SIO module be X = [x 1 ,x 2 ,x 3 ,…,x N ]∈R N*Dx , where N represents the sequence length and Dx represents the dimension of each template element. First, the input sequence is compared with the weight matrix parameter W of the Structure Transformer. q ∈R N*Dk , W k ∈R N*Dk, W v ∈R N*Dv Perform a dot product operation to obtain the query vector matrix Q = [q 1 ,q 2 ,q 3 ,…,q N ], key vector matrix K = [k 1 ,k 2 ,k 3 ,…,k N ] and value vector matrix V = [v 1 ,v 2 ,v 3 ,…,v N ]. Then calculate the self-attention output of X, which can be calculated by the following formula:

[0091]

[0092] At this time, Z=[z 1 ,z 2 ,z 3 ,…,z N ] represents the output calculated by the attention mechanism. The output here is the output of a self-attention head. Since a multi-head self-attention mechanism is used, that is, multiple groups of W are used. q , W k and W v Multiple groups of Q, K, and V are obtained, and then multiple Z matrices are calculated respectively. Finally, the multiple Z matrices are concatenated to obtain the output of the multi-head self-attention mechanism.

[0093] Step 4.2.2: Next, calculate the similarity matrix of the input sequence to help identify the structure of the melody. The similarity matrix is ​​calculated by the Euclidean distance between the elements of the input sequence. For each element x in the input sequence X, i The similarity between the elements and other elements can finally get an N*N similarity matrix. The higher the similarity, the more similar the elements between the subsections are. The calculation formula is as follows:

[0094]

[0095] Here, S ij Represents the sequence x i and x j The similarity between them, euclidean_distance(x i , x j ) represents x i and x jThe Euclidean distance between them. ε is a small constant, usually 0.000001 or smaller, to avoid the situation where the denominator is zero. Then, for further processing, the similarity matrix is ​​normalized.

[0096] Step 4.2.3: Finally, the output of the SIO module is obtained through the dot product operation and softmax calculation of the similarity matrix and the self-attention mechanism.

[0097] Output = softmax(S·Z)

[0098] By calculating the similarity matrix of the template elements in the input sequence, we can better understand the relationship between the elements. This similarity matrix provides important information about the similarity between the melodic elements, which are also obtained from the lyrics, which helps us identify the chorus part of the melody. This allows the model to focus more on the parts with higher similarity, thereby better capturing the overall structure of the melody. This method can ensure that the generated melody structure is more complete and can more accurately express the emotions and artistic conception brought by the lyrics.

[0099] In the method of the lyrics-melody generation system CST, according to the loss function calculation in step 5, the model is trained using an optimized loss function in the model training process. The specific steps are as follows:

[0100] Step 5.1: During the training of Structure Transformer, the input is the template sequence data obtained in step 3.4, and the target sequence is the melody sequence data corresponding to the template sequence data. Therefore, it is necessary to achieve stricter alignment, that is, each template element corresponds to four melody music elements, and since the final implementation is melody generation, the generation of this step requires relatively higher accuracy. Therefore, in the design of the loss function, on the basis of the label smoothed cross entropy loss function with alignment information, the reinforcement loss is also considered, and the steps of the loss function are designed as follows:

[0101] Step 5.1.1: First is the label smoothed cross entropy loss function:

[0102]

[0103] Among them, N is the number of label categories predicted by the model, P(y pred ) is the probability of the model predicting the true label of the tth token of the nth input sequence, x n：Tn is the nth input sequence, P(y n,t ∣x n:Tn ) is the model’s n：Tn The predicted probability of the tth token in the target sequence under the condition.n represents the number of tags in the nth sequence. ∈ is the smoothing parameter.

[0104] Step 5.1.2: Consider the sequence-to-sequence alignment loss:

[0105]

[0106] Among them, y n,t is the tth token in the nth target sequence, x n,t is the token in the corresponding input sequence, attn is the multi-head self-attention distribution value in Structure Transformer, and align_weights is the inverse of the frequency of the target melody, which is used for normalization.

[0107] Step 5.1.3: Reinforcement loss is introduced to further improve the accuracy of generation. Specifically, when the generated music element is correct, the model is given a reward value; when the generated music element does not match, a penalty value slightly larger than the reward value is given. The formula of reinforcement loss is as follows:

[0108]

[0109] Among them, P(y n ) is the probability of the true label, is the given reward value.

[0110] Therefore, the final total loss function is:

[0111] Loss=LSCELoss+β*AlignLoss+γ*ReinLoss

[0112] The parameters β and γ before the alignment loss and reinforcement learning loss function in the total loss function Loss are adjustable later. The parameter values ​​used here are 0.05 and 2.

[0113] This reinforcement loss complements the previous smooth cross entropy loss function. By introducing the reinforcement loss, the original loss function is enhanced, and the possibility of the model making wrong predictions is reduced through learning, thereby encouraging the model to make predictions more cautiously and accurately to reduce the risk of misclassification. This makes the model pay more attention to accuracy during the generation process and avoids the problem of overfitting.

[0114] In the method of the lyrics-melody generation system CST, the step 6 lyrics-melody generation, the specific steps are as follows:

[0115] According to the input data obtained in the previous step 2, it is input into the Structure Transformer network in step 4. The network has been fully trained in step 5 and has the ability to generate melody. The Structure Transformer model can generate highly suitable melodies based on the lyrics and related information provided by the user, and realize the natural conversion of lyrics to melody.

[0116] In the method of the lyrics-melody generation system CST, in the step 7 melody generation evaluation, the melody generated by the model is evaluated subjectively and objectively. The specific steps are as follows:

[0117] Step 7.1: Objective evaluation. In this step, objective evaluation indicators are used to evaluate the quality of the generated melody. The main evaluation indicators are the similarity index compared with the real melody and the diversity index of melody generation. The specific measurement steps are as follows:

[0118] Step 7.1.1: Similarity evaluation. For the similarity evaluation of PDS, DDS and IOIs mentioned above, here we calculate the distribution of the pitch, length and onset interval between the melodies (represented by frequency histograms) and measure the similarity between the distribution of the generated melody and the ground truth melody (calculated by the average overlap area).

[0119] Step 7.1.2: Diversity evaluation: Pitch Diversity (PD) and Duration Diversity (DD). These two indicators are evaluation criteria specifically proposed by the present invention for this task: First, extract the pitch information from the generated melody and convert the melody into a series of notes, each of which contains pitch and duration information. Then, count the number of different pitch values, denoted as N pitch , the total number of pitches is recorded as T pitch The calculation formula of pitch diversity (PD) is as follows:

[0120]

[0121] Similarly, we extract note duration information from the generated melody, count the number of different note duration values, and divide it by the total number of note durations in the entire melody to get the note duration diversity. The closer the pitch diversity and note duration diversity are to the real melody, the higher the generated quality.

[0122] Step 7.2: Subjective evaluation. From the generated melodies, songs in Chinese and English were randomly selected from the system, and the corresponding lyrics were provided to the evaluators. When scoring, the melodies generated by the system were evaluated, where the identity of the system was hidden, the order of each melody generated by the system was random, and the entire evaluation was conducted in a blind review mode. Each reviewer was asked to answer a series of questions using a five-point system from 1 (poor) to 5 (excellent) to evaluate the performance of the generated melody in the following key aspects: 1) Fluency: Is the generated melody smooth? 2) Naturalness: Does the melody sound natural? Are the pauses in the rhythm and lyrics coordinated? 3) Repeatability: Is the melody composed of repeated impressive fragments? 4) Overall quality (Overall): What is the overall quality of the melody?

[0123] Step 7.3: Evaluation results. According to the evaluation criteria mentioned in Step 7.1 and Step 7.2, the melodies generated by our CST system were evaluated. Twenty Chinese and English songs were randomly selected. The evaluation results are as follows:

[0124] Table 1 Objective evaluation results

[0125] Model PDS(%) DDS(%) IOIs(%) PD(%) DD(%) Epoch (Chinese)CST 55.51 41.46 20.23 2.58 2.14 112 (English)CST 53.00 36.43 12.30 7.72 7.85 112

[0126] Table 2 Subjective evaluation results

[0127] Model Fluency Naturalness Repeatability Overall (Chinese)CST 3.62 3.20 3.28 3.42 (English)CST 3.54 3.22 3.32 3.38

[0128] Through the above steps, the present invention provides an efficient and accurate lyrics-melody generation method, which overcomes the limitations of existing methods and achieves a high degree of integration and optimization of lyrics and melody.

[0129] The present invention also provides a structured lyrics-melody generation system based on generative AI, comprising the following modules:

[0130] A lyrics generation module, used to generate lyrics that match the theme specified by the user;

[0131] An input data acquisition module is used to obtain tonality and rhythm information by analyzing the lyrics information using the lyrics obtained by the lyrics generation module, and then obtain rhythm pattern information through an existing lyrics-rhythm model, and at the same time, combined with the chord data provided by the user, the extracted tonality, chords, rhythm pattern and rhythm are stored in the form of sequence data as subsequent input data;

[0132] A training data set collection module is used to collect a training data set, wherein the training data set includes a template sequence consisting of four musical elements, namely, tonality, chords, rhythmic patterns and rhythms, and a melody sequence consisting of corresponding melody tracks;

[0133] The Structure Transformer model building module is used to input the training data set into the Structure Transformer model based on the Transformer improvement. The model is an encoder-decoder structure. The encoder part is responsible for encoding the input data, i.e., the template sequence data, and converting it into an internal representation. The decoder part generates the corresponding melody sequence data based on these internal representations.

[0134] Model training module, used to design loss function to optimize Structure Transformer model;

[0135] The melody generation module is used to input the data in the input data acquisition module into the trained Structure Transformer model to generate lyrics-melody.

[0136] The specific implementation method of each module is the same as each step and will not be described in detail in the present invention.

[0137] The embodiments described in this document represent only one of the many possible implementations of the present invention. Although the description of these embodiments is specific and detailed, it should not be regarded as limiting the scope of the patent of the present invention. For ordinary technicians in this field, it can be understood that there are many possibilities for variations and improvements without departing from the basic concept and innovative spirit of the present invention. These variations and improvements may involve adjustments to the implementation details, optimization of existing functions, or expansion of functions to adapt to different application scenarios and technical requirements. When considering the scope of protection of the present invention, its core principles and its practical application fields should be widely considered. The present invention aims to provide an innovative solution, and its application prospects should not be limited to the embodiments described so far. The scope of protection of the present invention should be defined around its basic innovative concept and its potential applications in practice, which includes but is not limited to various variants of implementation, technical improvements, and related innovations that may arise in future technological developments. Therefore, the scope of protection of the present invention should not be limited to these specific embodiments, but should be determined based on the interpretation of the attached claims. These claims are intended to cover all potential variations and applications of the present invention, provide comprehensive protection for the core principles of the present invention, and ensure the wide application and development of the present invention in the technical field.

Claims

1. A structured lyrics-melody generation method based on generative AI, characterized in that: The steps include: Step 1, generating lyrics that match the theme specified by the user; Step 2, using the lyrics obtained in step 1, obtaining tonality and rhythm information by analyzing the lyrics information, then obtaining rhythm pattern information through an existing lyrics-rhythm model, and combining the chord data provided by the user, storing the extracted tonality, chords, rhythm pattern and rhythm in the form of sequence data as subsequent input data; Step 3, collecting a training data set, the training data set includes a template sequence consisting of four musical elements: tonality, chords, rhythmic patterns, and rhythms, and a melody sequence consisting of corresponding melody tracks; Step 4: Input the training data set into the Structure Transformer model based on the Transformer improvement. The model is an encoder-decoder structure. The encoder part is responsible for encoding the input data, i.e., the template sequence data, and converting it into an internal representation. The decoder part generates the corresponding melody sequence data based on these internal representations. The Structure Transformer model introduces the structure recognition and optimization module SIO based on the Transformer. This module is located between position encoding and layer normalization. The specific implementation method is as follows; (1) Let the input of SIO be the sequence X=[x1, x2, x3, …, x N ] R N * Dx , is the sequence after position encoding, where N represents the sequence length and Dx represents the dimension of each music element; first, the input sequence is respectively compared with the weight matrix W q R N * Dk , W k R N * Dk , W v R N * Dv Perform a dot product operation to obtain the query vector matrix Q = [q1, q2, q3, …, q N ], key vector matrix K = [k1, k2, k3, …, k N ] and the value vector matrix V = [v1, v2, v3, …, v N ]; then calculate the self-attention output of X, using the following formula: ； Now Z = [z1, z2, z3, …, z N ] represents the output calculated by the attention mechanism. The output here is the output of a self-attention head. Since a multi-head self-attention mechanism is used, that is, multiple groups of W are used q , W k and W v Get multiple groups of Q, K, and V, then calculate multiple Z matrices respectively, and finally concatenate the multiple Z matrices to get the output of the multi-head self-attention mechanism; (2) Next, we calculate the similarity matrix of the input sequence to help identify the structure of the melody. The similarity matrix is ​​calculated by the Euclidean distance between the elements of the input sequence. i The similarity with other elements finally results in an N*N similarity matrix; the calculation formula is as follows: ； Here, S ij Representation sequence and x j The similarity between them, euclidean_distance(x i , x j ) represents x i and x j The Euclidean distance between is a constant to avoid the situation where the denominator is zero; then, the similarity matrix is ​​normalized; (3) The output of the SIO module is obtained through the dot product operation of the similarity matrix and the self-attention mechanism and the softmax calculation; Step 5: Design a loss function to optimize the Structure Transformer model. Step 6: Input the data in step 2 into the trained Structure Transformer model to generate lyrics-melody.

2. A structured lyrics-melody generation method based on generative AI as claimed in claim 1, characterized in that: In step 1, ChatGPT is used to generate lyrics that satisfy the user and conform to the user-specified theme.

3. A structured lyrics-melody generation method based on generative AI as claimed in claim 1, characterized in that: In step 2, the tonality is obtained by analyzing the data in the lyrics through the text sentiment model. The tonality is divided into major and minor, corresponding to the positive and negative emotions of the lyrics respectively; The rhythm is obtained based on the punctuation marks in the lyrics, and the lyrics are mapped as follows: the commas and periods in the lyrics are transformed into half-stops and stops to indicate the end of the sentence; By introducing an existing Transformer-based lyrics-rhythm model to obtain the rhythmic pattern, the rhythmic pattern in a given lyrics template is predicted in an autoregressive manner.

4. The structured lyrics-melody generation method based on generative AI as claimed in claim 1, characterized in that: The specific implementation of step 3 is as follows; Step 3.1, prepare MIDI music dataset; Step 3.2, processing MIDI files: first, extracting melody track events from multi-track MIDI files, and encoding these melody tracks; Next, the segments are filtered according to the following principles: segments whose time signature changes more than once in the whole melody, segments whose length exceeds the time limit, and segments whose time unit is not the granularity of sixteenth notes; after filtering, the segments that meet the requirements will form the melody sequence dataset; Step 3.3, extract the four elements of tonality, chord progression, rhythmic pattern and rhythm: first, infer the tonality based on the pitch distribution of the notes in the melody sequence dataset; second, infer the chords based on the pitch distribution of the notes using the Viterbi algorithm; infer the rhythmic pattern based on the position information of the notes; finally, infer the rhythm based on the pitch and duration rules; Step 3.4, based on the obtained music elements, the note elements are digitally represented as multidimensional tensors to form sequence data in the template, and these sequence data are aligned with the previous music melody sequence data set to form a corresponding template-melody data set, that is, the training data set.

5. The structured lyrics-melody generation method based on generative AI as claimed in claim 1, characterized in that: The total loss function used in the training of the Structure Transformer model includes the label smoothed cross entropy loss function, the alignment loss function, and the reinforcement loss function; ； Among them, the alignment loss in the total loss function Loss And the parameters before the reinforcement learning loss function ReinLoss and It is adjustable. is the label smoothed cross entropy loss function; The calculation formula of the label smoothed cross entropy loss function is as follows: ； Among them, N is the number of label categories, P(y pred ) is the probability of the model predicting the true label of the tth token of the nth input sequence, x n：Tn is the nth input sequence, P(y n,t ∣x n:Tn ) is given an input sequence x n：Tn Under the condition, the probability of the tth token in the target sequence, T n is the number of tokens in the nth sequence and ϵ is the smoothing parameter.

6. A structured lyrics-melody generation method based on generative AI as claimed in claim 5, characterized in that: The expression of sequence-to-sequence alignment loss is as follows: ； Among them, y n,t is the tth token in the nth target sequence, x n,t is the token in the corresponding input sequence, attn is the multi-head self-attention distribution value in Structure Transformer, and align_weights is the inverse of the frequency of the target melody, which is used for normalization.

7. A structured lyrics-melody generation method based on generative AI as claimed in claim 5, characterized in that: The formula for the reinforcement loss is as follows: ； Among them, P(y n ) is the probability of the true label, is the given reward value.

8. The structured lyrics-melody generation method based on generative AI as claimed in claim 1, characterized in that: The method further includes step 7, wherein the quality of the generated melody is evaluated by using the similarity of pitch distribution, the similarity of tone length distribution, the starting interval between melodies, the pitch diversity, and the tone length diversity, wherein the calculation formulas for the pitch diversity and the tone length diversity are as follows: First, extract the pitch information from the generated melody and convert the melody into a series of notes, each of which contains pitch and duration information; then count the number of different pitch values, denoted as N pitch , the total number of pitches is recorded as T pitch , the calculation formula of pitch diversity PD is as follows: ； Similarly, note duration information is extracted from the generated melody, the number of different note duration values ​​is counted, and then divided by the total number of note durations in the entire melody to obtain the note duration diversity.

9. A structured lyrics-melody generation system based on generative AI, characterized in that: Includes the following modules: A lyrics generation module, used to generate lyrics that match the theme specified by the user; An input data acquisition module is used to obtain tonality and rhythm information by analyzing the lyrics information using the lyrics obtained by the lyrics generation module, and then obtain rhythm pattern information through an existing lyrics-rhythm model, and at the same time, combined with the chord data provided by the user, the extracted tonality, chords, rhythm pattern and rhythm are stored in the form of sequence data as subsequent input data; A training data set collection module is used to collect a training data set, wherein the training data set includes a template sequence consisting of four musical elements, namely, tonality, chords, rhythmic patterns and rhythms, and a melody sequence consisting of corresponding melody tracks; The Structure Transformer model building module is used to input the training data set into the Structure Transformer model based on the Transformer improvement. The model is an encoder-decoder structure. The encoder part is responsible for encoding the input data, i.e., the template sequence data, and converting it into an internal representation. The decoder part generates the corresponding melody sequence data based on these internal representations. The Structure Transformer model introduces the structure recognition and optimization module SIO based on the Transformer. This module is located between position encoding and layer normalization. The specific implementation method is as follows; (1) Let the input of SIO be the sequence X=[x1, x2, x3, …, x N ] R N * Dx , is the sequence after position encoding, where N represents the sequence length and Dx represents the dimension of each music element; first, the input sequence is respectively compared with the weight matrix W q R N * Dk , W k R N * Dk , W v R N * Dv Perform a dot product operation to obtain the query vector matrix Q = [q1, q2, q3, …, q N ], key vector matrix K = [k1, k2, k3, …, k N ] and the value vector matrix V = [v1, v2, v3, …, v N ]; then calculate the self-attention output of X, using the following formula: ； Now Z = [z1, z2, z3, …, z N ] represents the output calculated by the attention mechanism. The output here is the output of a self-attention head. Since a multi-head self-attention mechanism is used, that is, multiple groups of W are used q , W k and W v Get multiple groups of Q, K, and V, then calculate multiple Z matrices respectively, and finally concatenate the multiple Z matrices to get the output of the multi-head self-attention mechanism; (2) Next, we calculate the similarity matrix of the input sequence to help identify the structure of the melody. The similarity matrix is ​​calculated by the Euclidean distance between the elements of the input sequence. i The similarity with other elements finally results in an N*N similarity matrix; the calculation formula is as follows: ； Here, S ij Representation sequence and x j The similarity between them, euclidean_distance(x i , x j ) represents x i and x j The Euclidean distance between is a constant to avoid the situation where the denominator is zero; then, the similarity matrix is ​​normalized; (3) The output of the SIO module is obtained through the dot product operation of the similarity matrix and the self-attention mechanism and the softmax calculation; Model training module, used to design loss function to optimize Structure Transformer model; The melody generation module is used to input the data in the input data acquisition module into the trained StructureTransformer model to generate lyrics-melody.

Images