A melody generation method and device

Abstract

The present disclosure discloses a melody generation method and device. In the method, at least one clause in a song text is obtained; sentiment analysis is performed on each clause to obtain sentiment information of the corresponding clause; after each clause and the sentiment information of the corresponding clause are spliced, the spliced result is input into a trained model based on a self-attention mechanism to obtain a melody vector corresponding to each clause; wherein the model based on the self-attention mechanism is used to process current input data in accordance with a melody vector obtained from previous input data to obtain a melody vector corresponding to the current input data; and a melody of the song text is generated based on the melody vectors of the clauses. In the method, sentiment information corresponding to the song text is obtained by performing text sentiment analysis on the song text under the condition that the song text exists, so that the generated melody is consistent with the song in terms of emotion, and the natural degree of the generated melody and the quality of the generated melody are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for generating melodies. Background Technology

[0002] Music is an important art form. Typically, music comprises two components: melody and lyrics. Melody, a collection of notes arranged sequentially over time, is widely considered a crucial element of music. Currently, methods for generating melodies from lyrics primarily employ text analysis techniques to segment and extract features from the lyrics. These are then compared to lyrics in a custom lyric library for similarity matching. If a match is found, models such as decision trees or random forests are used to predict the corresponding melody.

[0003] However, the method of extracting features from lyrics relies on manually defined rules, which are not comprehensive or refined enough, and the generated melody can only be configured based on existing melodies and cannot be expanded, resulting in low quality of the generated melody. Summary of the Invention

[0004] This disclosure provides a melody generation method and apparatus that solves the aforementioned problems in the prior art and improves the quality of the generated melody. The technical solution of this disclosure is as follows:

[0005] Get at least one clause from the lyrics text;

[0006] Sentiment analysis was performed on each clause to obtain the sentiment information of the corresponding clause;

[0007] After concatenating the emotional information of each clause with the corresponding clause, the concatenation is input into a trained model based on a self-attention mechanism to obtain the melody vector corresponding to each clause; wherein, the model based on a self-attention mechanism is used to process the current input data according to the melody vector obtained from the previous input data to obtain the melody vector corresponding to the current input data.

[0008] The melody of the lyrics text is generated based on the melody vectors of each clause.

[0009] In an optional implementation, the emotional information is an emotional score that measures the type and degree of the emotion;

[0010] Sentiment analysis was performed on each clause to obtain the corresponding sentiment information, including:

[0011] Each clause is input into a preset word embedding model to obtain the word vector of each word in the corresponding clause;

[0012] For each sentence, the word vector of the sentence is sequentially input into the trained sentiment analysis model to obtain the sentiment score of the sentence; the sentiment analysis model is trained based on different lyric sentences and their corresponding sentiment scores.

[0013] In an optional implementation, the clauses are concatenated with their corresponding sentiment information and then input into a trained self-attention-based model, including:

[0014] Based on the number of words in each clause and the corresponding emotional information of the clause, generate the emotional vector of the corresponding clause;

[0015] Based on the word vector of each word in each clause, obtain the clause matrix of the corresponding clause;

[0016] The sentiment vector and the sentence segmentation matrix are concatenated to obtain a concatenated matrix;

[0017] The concatenated matrix is ​​input into the trained model based on the self-attention mechanism.

[0018] In an optional implementation, the sentiment vector is a one-dimensional column vector;

[0019] Based on the number of words in each clause, obtain the sentiment vector corresponding to the sentiment information of the corresponding clause, including:

[0020] The number of words is determined as the number of rows in the sentiment vector;

[0021] The emotional score is determined as a parameter value in the emotional vector.

[0022] In an optional implementation, after generating the sentiment vector corresponding to the corresponding clause based on the number of words in each clause and the sentiment information of the corresponding clause, the method further includes:

[0023] The emotional vectors of each sentence are concatenated according to the order of the lyrics to obtain the concatenated emotional vector.

[0024] After obtaining the sentence segmentation matrix of the corresponding sentence based on the word vector of each word in each sentence, the method further includes:

[0025] According to the order of the lyrics, the sentence matrices of each sentence are concatenated to obtain the concatenated sentence matrix;

[0026] The sentiment vector and the sentence segmentation matrix are concatenated to obtain a concatenated matrix, which includes:

[0027] The concatenated emotion vector is concatenated to the last column of the concatenated sentence matrix to obtain the concatenated matrix.

[0028] In an optional implementation, the melody of the lyrics text is generated based on the melody vectors of each clause, including:

[0029] A preset decoding method is used to decode the melody vector of each phrase to obtain the melody segment corresponding to each phrase;

[0030] The melody of the lyrics is generated based on the order of the lyrics and the melodic fragments of each phrase.

[0031] In an optional implementation, the sentiment analysis model is trained by:

[0032] Obtain the training samples of the lyrics phrases and the corresponding sentiment annotation scores of the training samples of the lyrics phrases;

[0033] Based on the training samples of the lyrics and the sentiment annotation scores of the corresponding training samples of the lyrics, the neural network model is iteratively trained to obtain the sentiment analysis model.

[0034] In an optional implementation, the self-attention-based model is trained by:

[0035] Obtain training samples of sentence segmentation matrices, melody fragments, and emotion vectors for different lyric phrases;

[0036] For each lyric phrase, each note in the melody segment training sample is processed according to a preset note resolution to obtain the melody annotation vector of the melody segment training sample;

[0037] The input training data, consisting of sentence matrix training samples of different lyric phrases and corresponding emotion vector training samples, as well as the corresponding melody annotation vector, is used to iteratively train the initial self-attention mechanism-based model to obtain a self-attention mechanism-based model.

[0038] In an optional implementation, each note in the melody segment training sample is processed according to a preset note resolution to obtain the melody annotation vector of the melody segment training sample, including:

[0039] Based on the melody fragment training samples, obtain the note name and duration of each note in the melody fragment training samples;

[0040] According to the preset encoding method and preset note resolution, the note name and duration of each note are processed to obtain the melody annotation vector of the melody segment training sample.

[0041] According to a second aspect of the present disclosure, a melody generation apparatus is provided, which may include: an acquisition unit, an analysis unit, an input unit, and a generation unit;

[0042] The acquisition unit is used to acquire at least one clause from the lyrics text;

[0043] The analysis unit is used to perform sentiment analysis on each clause to obtain the sentiment information of the corresponding clause;

[0044] The input unit is used to concatenate the emotional information of each clause with the corresponding clause and input it into a trained model based on a self-attention mechanism to obtain the melody vector corresponding to each clause; wherein, the model based on a self-attention mechanism is used to process the current input data according to the melody vector obtained from the previous input data to obtain the melody vector corresponding to the current input data.

[0045] The generation unit is used to generate the melody of the lyrics text based on the melody vector of each clause.

[0046] In an optional implementation, the emotional information is an emotional score that measures the type and degree of the emotion;

[0047] The analysis unit is specifically used to input each clause into a preset word embedding model to obtain the word vector of each word in the corresponding clause;

[0048] For each sentence, the word vector of the sentence is sequentially input into the trained sentiment analysis model to obtain the sentiment score of the sentence; the sentiment analysis model is trained based on different lyric sentences and their corresponding sentiment scores.

[0049] In an optional implementation, the device further includes a splicing unit;

[0050] The generation unit is also used to generate the sentiment vector of the corresponding clause based on the number of words in each clause and the sentiment information of the corresponding clause;

[0051] The acquisition unit is also used to acquire the sentence matrix of the corresponding sentence based on the word vector of each word in each sentence;

[0052] The splicing unit is used to splice the emotion vector and the sentence segmentation matrix to obtain a splicing matrix;

[0053] The input unit is specifically used to input the concatenated matrix into a trained model based on a self-attention mechanism.

[0054] In an optional implementation, the sentiment vector is a one-dimensional column vector;

[0055] The acquisition unit is specifically used to determine the number of words as the number of rows in the sentiment vector;

[0056] Furthermore, the emotional score is determined as a parameter value in the emotional vector.

[0057] In an optional implementation, the splicing unit is further configured to splice the emotion vectors of each clause according to the order of the lyrics text to obtain a spliced ​​emotion vector;

[0058] Furthermore, according to the order of the lyrics text, the sentence matrices of each sentence are concatenated to obtain a concatenated sentence matrix;

[0059] The concatenated emotion vector is concatenated to the last column of the concatenated sentence matrix to obtain the concatenated matrix.

[0060] In an optional implementation, the generation unit is specifically used to decode the melody vector of each phrase using a preset decoding method to obtain the melody segment corresponding to each phrase;

[0061] Furthermore, based on the order of the lyrics text and the melodic fragments of each clause, the melody of the lyrics text is generated.

[0062] In an optional implementation, the apparatus further includes a first training unit;

[0063] The first training unit is configured to perform the following steps:

[0064] Obtain the training samples of the lyrics phrases and the corresponding sentiment annotation scores of the training samples of the lyrics phrases;

[0065] Based on the training samples of the lyrics and the sentiment annotation scores of the corresponding training samples of the lyrics, the neural network model is iteratively trained to obtain the sentiment analysis model.

[0066] In an optional implementation, the apparatus further includes a second training unit;

[0067] The second training unit is used to perform the following steps:

[0068] Obtain training samples of sentence segmentation matrices, melody fragments, and emotion vectors for different lyric phrases;

[0069] For each lyric phrase, each note in the melody segment training sample is processed according to a preset note resolution to obtain the melody annotation vector of the melody segment training sample;

[0070] The input training data, consisting of sentence matrix training samples of different lyric phrases and corresponding emotion vector training samples, as well as the corresponding melody annotation vector, is used to iteratively train the initial self-attention mechanism-based model to obtain a self-attention mechanism-based model.

[0071] In an optional implementation, the second training unit is further configured to obtain the note name and duration of each note in the melody segment training sample based on the melody segment training sample;

[0072] Furthermore, according to a preset encoding method and a preset note resolution, the note name and duration of each note are processed to obtain the melody annotation vector of the melody segment training sample.

[0073] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to said at least one processor, wherein:

[0074] The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is able to perform any of the method steps in the first aspect.

[0075] According to a fourth aspect of the present disclosure, a storage medium is provided that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method steps described in the first aspect.

[0076] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0077] This melody generation method involves acquiring at least one phrase from the lyrics text; performing sentiment analysis on each phrase to obtain its corresponding sentiment information; concatenating each phrase with its corresponding sentiment information and inputting the result into a pre-trained self-attention-based model to obtain a melody vector for each phrase; the self-attention-based model then processes the current input data based on the melody vector obtained from the previous input data to obtain the melody vector corresponding to the current input data; and finally, based on the melody vectors of each phrase, the melody of the lyrics text is generated. This method, given the existence of lyrics text, uses sentiment analysis to obtain the corresponding sentiment information, ensuring that the generated melody is emotionally consistent with the lyrics, thus improving the naturalness and quality of the generated melody.

[0078] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0079] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0080] Figure 1This is a flowchart illustrating a melody generation method according to an exemplary embodiment;

[0081] Figure 2 This is a schematic diagram of the structure of a melody generation device according to an exemplary embodiment;

[0082] Figure 3 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0083] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0084] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0085] For ease of understanding, the musical notes involved in the embodiments of the present invention will be explained below:

[0086] Musical notes are symbols used to record music.

[0087] Different musical notes are marked on different lines and spaces of the staff, which can reflect the pitch and duration.

[0088] Different pitches correspond to different note names. The notes include the note names of 7 basic notes. Each basic note is represented by an English letter: C, D, E, F, G, A and B. C, D, E, F, G, A and B correspond to the solfège names Do, Ruai, Mi, Fa, Sol, La and Xi, respectively.

[0089] The types of musical notes include: whole note, half note, quarter note, eighth note, sixteenth note, thirty-second note, and sixty-fourth note. Different notes represent different durations; that is, the type of note represents the duration (or "time value") of that note.

[0090] The half note, quarter note, eighth note, sixteenth note, thirty-second note, and sixty-fourth note represent time values ​​of 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of a whole note, respectively.

[0091] The melody generation method provided in this embodiment of the invention can be applied on a server or on a terminal.

[0092] The server can be an application server or a cloud server; the terminal can be a user equipment (UE) such as a mobile phone, smartphone, laptop, digital broadcast receiver, personal digital assistant (PDA), tablet computer (PAD), handheld device, vehicle-mounted device, wearable device, computing device or other processing device connected to a wireless modem, mobile station (MS), mobile terminal, etc.

[0093] Compared with the prior art, the melody generation method provided in this embodiment of the invention, under the condition of having lyrics, obtains the emotional information embodied in the lyrics or the emotional information that the lyrics want the listener to experience by performing textual sentiment analysis on the lyrics, thereby making the generated melody consistent with the lyrics in terms of emotion and improving the naturalness of the generated melody.

[0094] It should be noted that the purpose of text sentiment analysis is to identify the speaker's / author's polarized attitudes on certain topics or towards the text. These polarized attitudes include the speaker's / author's emotional state at the time of writing, i.e., the speaker's / author's emotions when making the statements in the text, and the speaker's / author's intentional emotional communication, i.e., the emotions the speaker / author wants the reader / listener to experience.

[0095] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the invention. Furthermore, the embodiments and features described herein can be combined with each other without conflict.

[0096] Figure 1 This is a flowchart illustrating a melody generation method according to an exemplary embodiment. For example... Figure 1 As shown, the method may include:

[0097] In step S110, at least one sentence from the lyrics text is obtained.

[0098] The lyrics text obtained can be text entered by the user in the interactive interface. The length of the lyrics text is not limited; it can be a single line or a paragraph. Alternatively, the interactive interface can display recommended lyrics text, which the user can select from to obtain the selected lyrics text.

[0099] A phrase in the lyrics text is at least one text segment within the lyrics text. For example, the lyrics text of "Twinkle, twinkle, little star, how I wonder what you are" may include two phrases: "Twinkle, twinkle, little star" and "how I wonder what you are," or it may include four phrases: "Twinkle, twinkle," "little star," "how I wonder what you are," and "little star," or the entire lyrics text may be a single phrase. The specific method of phrase division can be determined according to the actual situation, and this embodiment of the invention does not limit it here.

[0100] In step S120, sentiment analysis is performed on each clause to obtain the sentiment information of the corresponding clause.

[0101] Since emotional information can include positive and negative emotional types, negative emotional types include negative emotions such as anger and sadness, while positive emotional types include positive emotions such as joy and happiness, positive and negative emotional types can be represented by positive and negative scores within a preset range to distinguish between them. For example, the preset range can be from -5 to 5, such as [-5, 5]. A positive sign indicates a positive emotional type, and a negative sign indicates a negative emotional type. The scores other than positive and negative signs indicate the degree of emotion of the corresponding emotional type, such as -2 indicating relatively sad and -5 indicating very sad.

[0102] In this embodiment of the disclosure, each clause is input into a preset word embedding model, such as a word embedding model, to obtain the word vector of each word in the corresponding clause.

[0103] In order to obtain accurate emotional information expressed by the lyrics, this disclosure takes each sentence of the lyrics and inputs the word vectors in the sentence matrix into a trained sentiment analysis model, such as the Bret model, to obtain the sentiment information of the sentence. This sentiment information can be a sentiment score that measures the sentiment type and intensity. The sentiment analysis model is trained based on different lyric sentences and corresponding sentiment scores.

[0104] It should be noted that a word is the smallest semantic unit in the lyrics. For example, in the lyrics "Not yet properly feeling the climate of snowflakes blooming," each character is a word. The language of the lyrics is not limited. For Chinese text, a character is a word; for English text, a word is a phrase.

[0105] In step S130, each clause is concatenated with the corresponding emotional information to obtain the current input data.

[0106] To improve the accuracy of melody generation, lyrics can be fused with corresponding emotional information. In this embodiment, the emotional vector of each clause can be generated based on the number of words in each clause and the emotional information of the corresponding clause.

[0107] In practice, the number of words is determined as the number of rows in the sentiment vector, and the sentiment score is determined as the parameter value in the sentiment vector. The sentiment vector can be a one-dimensional column vector.

[0108] For example, if the number of words is 5 and the sentiment score is 3, then the sentiment vector can be represented as a column vector of [3, 3, 3, 3].

[0109] Next, based on the word vectors of each word in each clause, the clause matrix for that clause is obtained. Each row of the clause matrix represents a word vector, and the rows are sorted according to the order of the words in the clause. For example, in a clause matrix t*h, t is the number of rows, and h is the dimension of the word vectors, i.e., the number of hidden layers in the word embedding model.

[0110] Furthermore, the sentiment vector and the sentence segmentation matrix are concatenated to obtain a concatenated matrix, which is then used as the current input data.

[0111] In this embodiment of the disclosure, the emotion vector is concatenated to the last column of the sentence matrix to obtain the concatenation matrix, and the concatenation matrix is ​​determined as the current input data.

[0112] If the sentence matrix is ​​a t*h matrix and the sentiment vector is a t*1 one-dimensional vector, then the sentiment vector is concatenated to the end of the last column of the sentence matrix, resulting in a concatenated matrix of t*(h+1).

[0113] For example, if the clause matrix is ​​represented as: The sentiment vector is represented as: The concatenation matrix can then be represented as:

[0114] In step S140, the current input data is input into the trained model based on the self-attention mechanism to obtain the melody vector corresponding to each phrase.

[0115] After using the concatenated matrix as the current input data, it is fed into a pre-trained Transformer model based on a self-attention mechanism. The self-attention model processes the current input data based on the melody vector obtained from the previous input data to obtain the corresponding melody vector for the current input data.

[0116] The Transformer model based on the self-attention mechanism includes an encoder and a decoder. The encoder encodes the current input data and inputs the processed encoded data into the decoder. The decoder decodes the encoded data of the current input data according to the melody vector obtained when decoding the previous input data, so as to obtain the melody vector corresponding to the current input data.

[0117] This method not only improves the efficiency of melody generation, but also, by combining the melody vector corresponding to the previous input data to obtain the melody vector corresponding to the current input data, makes the transition between the generated current melody vector and the melody vector corresponding to the previous input data more natural and smooth.

[0118] The encoder may include: an encoding data input unit, an encoding position unit, and Nx cyclic encoding subunits. Each cyclic encoding subunit consists of four parts: Multi-HeadAttention, Residual Connection and Normalization Add&Norm, Feed Forward, and Add&Norm. Each cyclic encoding subunit has the same structure, but the internal parameters of each cyclic encoding subunit are different. The input data of the cyclic encoding subunit is the output data of the previous cyclic encoding subunit. The decoder may include: a decoding data input unit, a decoding position unit, and Nx cyclic decoding subunits. Each cyclic decoding unit has Nx cyclic decoding subunits, and each cyclic decoding subunit consists of 6 parts, namely the pseudo multi-head attention mechanism Masked Multi-Head Attention, Add&Norm, Multi-Head Attention, Add&Norm, Feed Forward, and Add&Norm. In addition, it should be noted that the input data of Multi-Head Attention also includes the encoded data result of the encoding part. The structure of each cyclic decoding unit is the same, but the internal parameters of each cyclic decoding subunit are different. The input data of the cyclic decoding subunit is the output data of the previous cyclic decoding subunit.

[0119] For the encoding unit, the current input data is input to the encoding data input unit, the encoding position unit inputs the position code corresponding to the current input data, the current input data and the position code corresponding to the current input data are superimposed and input to the cyclic encoding subunit to obtain the encoded data of the current input data.

[0120] For the decoding unit, the melody vector corresponding to the previous input data is used as the input data of the decoding data input unit, and the position code corresponding to the current input data is used as the input data of the decoding position unit. The position code is superimposed with the position data of the melody vector corresponding to the melody vector of the previous input data, and the encoded data of the current input data, and then input into the loop decoding unit for decoding processing. The melody vector corresponding to the input data is obtained through linear regression and logistic regression Softmax processing.

[0121] In step S150, the melody of the lyrics text is generated based on the melody vector of each clause.

[0122] Using a preset decoding method, such as one-hot encoding, the melody vector of each phrase is decoded to obtain the melody fragment corresponding to each phrase; based on the order of the lyrics text and the melody fragment of each phrase, the melody of the lyrics text is generated, thus enabling automatic melody generation based on the lyrics text.

[0123] In one possible embodiment, the melody fragments of each phrase can be spliced ​​together sequentially according to the order of the lyrics text to generate the melody of the lyrics text.

[0124] As can be seen, the above-mentioned method for generating melodies is automatic, which does not require users to have professional music knowledge to create music based on lyrics. This allows the general public to use the above-mentioned embodiments of the present invention to automatically generate melodies based on lyrics, further promoting the development of digital music.

[0125] In one possible embodiment based on step 130, in order to speed up the melody generation, after generating the emotion vectors of the corresponding sentences, the emotion vectors of each sentence can be concatenated according to the order of the lyrics text to obtain the concatenated emotion vectors corresponding to the lyrics text, and the sentence matrices of each sentence can be concatenated to obtain the concatenated sentence matrix.

[0126] In the lyrics text, adjacent sentences can be separated by delimiters such as “;” or “_”, and the value corresponding to the delimiter is “0”.

[0127] For example, if the sentiment vector corresponding to the first clause "the wind blows" is a column vector of [3, 3, 3], and the sentiment vector corresponding to the second clause "your smile appears" is a column vector of [4, 4, 4, 4, 4, 4], then the concatenated sentiment vector corresponding to the two clauses "the wind blows and your smile appears" is a column vector of [3, 3, 3, 0, 4, 4, 4, 4, 4, 4].

[0128] Next, the concatenated sentiment vector is concatenated to the last column of the concatenated sentence matrix to obtain the concatenated matrix, which is then used as the current input data.

[0129] Based on step 140, the current input data obtained above is input into the trained model based on the self-attention mechanism, thereby directly obtaining the melody vector sequence corresponding to the lyrics text, which includes the melody vector of each phrase.

[0130] Next, the melody vector sequence is decoded according to step 150 to obtain the melody of the lyrics text.

[0131] As can be seen, the above specific implementation method does not require concatenating the melody vectors of each phrase according to the order of the lyrics text, and can directly obtain the melody vector sequence corresponding to the lyrics text, thereby speeding up the generation of the melody.

[0132] Based on any of the above embodiments, in one possible implementation, in order to obtain the emotional information of each clause in the lyrics text, a sentiment analysis model can be trained, and the training method can be as follows:

[0133] Obtain the training samples of the lyrics phrases and the corresponding sentiment annotation scores of the training samples of the lyrics phrases;

[0134] Based on the training samples of lyric phrases and the sentiment annotation scores of the corresponding training samples of lyric phrases, the neural network model, such as the BERT pre-trained model, is iteratively trained to obtain the sentiment analysis model.

[0135] Based on any of the above embodiments, in one possible implementation, the training method for the model based on the self-attention mechanism can be as follows:

[0136] Obtain training samples of sentence segmentation matrices, melody fragments, and emotion vectors for different lyric phrases;

[0137] For each lyric phrase, each note in the melody segment training sample of that lyric phrase is processed according to the preset note resolution to obtain the melody annotation vector of the melody segment training sample.

[0138] The input training data, consisting of sentence matrix training samples with different lyric segments and corresponding emotion vector training samples, as well as the corresponding melody annotation vector, is used to iteratively train the initial self-attention mechanism-based model to obtain a self-attention mechanism-based model.

[0139] Based on the above embodiments, in order to improve the melody effect and accuracy generated by the above model, the musical score can be obtained based on the training samples of the melody fragments, thereby obtaining the note name and duration of each note.

[0140] For example, the note names and durations in the obtained musical score are as follows: E4 eighth note; E4 eighth note; E4 eighth note; E4 eighth note; A4 quarter note; B4 eighth note; C5 eighth note.

[0141] In E4, A4, B4, and C5, the letters represent pitch, and the numbers represent the corresponding pitch range.

[0142] Furthermore, according to the preset encoding method and preset note resolution, each note in the melody segment training sample of the lyric phrase is processed to obtain the melody annotation vector of the melody segment training sample.

[0143] The preset note resolution can be one of 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, and 1 / 64.

[0144] In practice, each note is represented by at least one field containing its note name, according to a preset note resolution, resulting in an extended note. The number of fields indicates the duration of the note. This number of fields is the ratio of the note's duration to its note resolution.

[0145] Since the durations of a half note, quarter note, eighth note, and sixteenth note are 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of a whole note, respectively, let's take a melody fragment with a note resolution of 1 / 16 as an example: E4 eighth note; E4 eighth note; A4 quarter note; C5 eighth note.

[0146] The extended note of the eighth note E4 is E4_, which contains two fields including the note name E4;

[0147] The extended note of the A4 quarter note is A4___, which contains the four fields of the note name A4;

[0148] The extended note of the C5 eighth note is C5_, which contains two fields including the note name C5.

[0149] Then, according to a preset encoding method, such as one-hot encoding, the extended notes of each note are encoded to obtain the melody annotation vector of the training sample of the melody segment. This preset encoding method is the inverse of the preset decoding method in step 140.

[0150] The melody generation method provided in this invention involves obtaining at least one clause from the lyrics text; performing sentiment analysis on each clause to obtain the corresponding sentiment information; concatenating each clause with the corresponding sentiment information and inputting the result into a trained model based on a self-attention mechanism to obtain the melody vector corresponding to each clause; wherein, the model based on the self-attention mechanism is used to process the current input data according to the melody vector obtained from the previous input data to obtain the melody vector corresponding to the current input data; and generating the melody of the lyrics text based on the melody vectors of each clause. This method, under the condition of existing lyrics text, obtains the corresponding sentiment information by performing text sentiment analysis on the lyrics text, ensuring that the generated melody is emotionally consistent with the lyrics, thus improving the naturalness and quality of the generated melody.

[0151] Figure 2 This is a schematic diagram illustrating the structure of a melody generation device according to an exemplary embodiment. (Refer to...) Figure 2 The device may include: an acquisition unit 210, an analysis unit 220, an input unit 230, and a generation unit 240;

[0152] Acquisition unit 210 is used to acquire at least one clause in the lyrics text;

[0153] Analysis unit 220 is used to perform sentiment analysis on each clause to obtain the sentiment information of the corresponding clause;

[0154] Input unit 230 is used to concatenate the emotional information of each clause with the corresponding clause and input it into a trained model based on self-attention mechanism to obtain the melody vector corresponding to each clause; wherein, the model based on self-attention mechanism is used to process the current input data according to the melody vector obtained from the previous input data to obtain the melody vector corresponding to the current input data.

[0155] The generation unit 240 is used to generate the melody of the lyrics text based on the melody vector of each clause.

[0156] In an optional implementation, the emotional information is an emotional score that measures the type and degree of the emotion;

[0157] The analysis unit 220 is specifically used to input each clause into a preset word embedding model to obtain the word vector of each word in the corresponding clause;

[0158] For each sentence, the word vector of the sentence is sequentially input into the trained sentiment analysis model to obtain the sentiment score of the sentence; the sentiment analysis model is trained based on different lyric sentences and their corresponding sentiment scores.

[0159] In an optional implementation, the device further includes a splicing unit 250;

[0160] The generation unit 240 is also used to generate the sentiment vector of the corresponding clause based on the number of words in each clause and the sentiment information of the corresponding clause;

[0161] The acquisition unit 210 is also used to acquire the sentence matrix of the corresponding sentence based on the word vector of each word in each sentence;

[0162] The splicing unit 250 is used to splice the emotion vector and the sentence segmentation matrix to obtain a splicing matrix;

[0163] The input unit 230 is specifically used to input the concatenated matrix into a trained model based on a self-attention mechanism.

[0164] In an optional implementation, the sentiment vector is a one-dimensional column vector;

[0165] The acquisition unit 210 is specifically used to determine the number of words as the number of rows of the sentiment vector;

[0166] Furthermore, the emotional score is determined as a parameter value in the emotional vector.

[0167] In an optional implementation, the splicing unit 250 is also used to splice the emotion vectors of each sentence according to the order of the lyrics text to obtain a spliced ​​emotion vector.

[0168] Furthermore, according to the order of the lyrics text, the sentence matrices of each sentence are concatenated to obtain a concatenated sentence matrix;

[0169] The concatenated emotion vector is concatenated to the last column of the concatenated sentence matrix to obtain the concatenated matrix.

[0170] In an optional implementation, the generation unit 240 is specifically used to decode the melody vector of each phrase using a preset decoding method to obtain the melody segment corresponding to each phrase.

[0171] Furthermore, based on the order of the lyrics text and the melodic fragments of each clause, the melody of the lyrics text is generated.

[0172] In an optional implementation, the apparatus further includes a first training unit 260;

[0173] The first training unit 260 is used to perform the following steps:

[0174] Obtain the training samples of the lyrics phrases and the corresponding sentiment annotation scores of the training samples of the lyrics phrases;

[0175] Based on the training samples of the lyrics and the sentiment annotation scores of the corresponding training samples of the lyrics, the neural network model is iteratively trained to obtain the sentiment analysis model.

[0176] In an optional implementation, the apparatus further includes a second training unit 270;

[0177] The second training unit 270 is used to perform the following steps:

[0178] Obtain training samples of sentence segmentation matrices, melody fragments, and emotion vectors for different lyric phrases;

[0179] For each lyric phrase, each note in the melody segment training sample is processed according to a preset note resolution to obtain the melody annotation vector of the melody segment training sample;

[0180] The input training data, consisting of sentence matrix training samples of different lyric phrases and corresponding emotion vector training samples, as well as the corresponding melody annotation vector, is used to iteratively train the initial self-attention mechanism-based model to obtain a self-attention mechanism-based model.

[0181] In an optional implementation, the second training unit 270 is further configured to obtain the note name and duration of each note in the melody fragment training sample based on the melody fragment training sample;

[0182] Furthermore, according to a preset encoding method and a preset note resolution, the note name and duration of each note are processed to obtain the melody annotation vector of the melody segment training sample.

[0183] The functions of each functional unit of the melody generation device provided in the above embodiments of the present invention can be implemented through the above method steps. Therefore, the specific working process and beneficial effects of each unit in the melody generation device provided in the embodiments of the present invention will not be repeated here.

[0184] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device 300 may be a server, mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0185] Reference Figure 3 The electronic device 300 may include one or more of the following components: processing component 302, memory 304, power supply component 306, multimedia component 308, audio component 310, input / output (I / O) interface 312, sensor component 314, and communication component 316.

[0186] Processing component 302 typically controls the overall operation of electronic device 300, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 302 may include one or more processors 320 to execute instructions to perform... Figure 1 The method shown may include all or part of the steps. Furthermore, the processing component 302 may include one or more modules to facilitate interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate interaction between the multimedia component 308 and the processing component 302.

[0187] Memory 304 is configured to store various types of data to support the operation of electronic device 300. Examples of such data include instructions for any application or method operating on electronic device 300, contact data, phonebook data, messages, pictures, videos, etc. Memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0188] Power supply component 306 provides power to various components of electronic device 300. Power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 300.

[0189] Multimedia component 308 includes a screen that provides an output interface between electronic device 300 and user. In some embodiments, the screen may include a liquid crystal display (LCD) (TP) and a touch panel. If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 308 includes a front-facing camera and / or a rear-facing camera. When electronic device 300 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0190] Audio component 310 is configured to output and / or input audio signals. For example, audio component 310 includes a microphone (MIC) configured to receive external audio signals when electronic device 300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 304 or transmitted via communication component 316. In some embodiments, audio component 310 also includes a speaker for outputting audio signals.

[0191] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0192] Sensor assembly 314 includes one or more sensors for providing state assessments of various aspects of electronic device 300. For example, sensor assembly 314 can detect the on / off state of electronic device 300, the relative positioning of components such as the display and keypad of electronic device 300, changes in position of electronic device 300 or a component of electronic device 300, the presence or absence of user contact with electronic device 300, orientation or acceleration / deceleration of electronic device 300, and temperature changes of electronic device 300. Sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 314 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0193] Communication component 316 is configured to facilitate wired or wireless communication between electronic device 300 and other devices. Electronic device 300 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0194] In one embodiment of this disclosure, the electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0195] In one embodiment of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, which can be executed by a processor 320 of an electronic device 300 to complete the steps of the above method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0196] In one embodiment of this disclosure, an application is also provided that, when executed by the processor of an electronic device, enables the electronic device to perform the steps of the above-described method to achieve the same technical effect.

[0197] In one embodiment of this disclosure, a computer program product is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the steps of the above-described method to achieve the same technical effect.

[0198] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0199] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device / electronic device / storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0200] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0201] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A melody generation method, characterized in that, The method includes: Get at least one clause from the lyrics text; Sentiment analysis was performed on each clause to obtain the sentiment information of the corresponding clause; After concatenating the emotional information of each clause with the corresponding clause, the concatenation is input into a trained model based on a self-attention mechanism to obtain the melody vector corresponding to each clause. The model based on the self-attention mechanism is used to encode the current input data to obtain the encoded data of the current input data, and to decode the encoded data of the current input data according to the melody vector obtained when decoding the previous input data to obtain the melody vector corresponding to the current input data. Based on the melody vectors of each clause, the melody of the lyrics text is generated; The process involves concatenating the sentiment information of each clause with the sentiment information of the corresponding clause, and then inputting the concatenated concatenation into a trained model based on a self-attention mechanism. This includes: generating a sentiment vector for each clause based on the number of words in each clause and the sentiment information of the corresponding clause; obtaining a clause matrix for each clause based on the word vector of each word in each clause, wherein the word vector of each word in each clause is obtained by inputting each clause into a preset word embedding model; concatenating the sentiment vector and the clause matrix to obtain a concatenation matrix; and inputting the concatenation matrix into the trained model based on a self-attention mechanism. The model based on the self-attention mechanism is trained as follows: training samples of sentence matrix, melody fragment, and emotion vector for different lyric phrases are obtained; for each lyric phrase, each note in the melody fragment training sample is processed according to a preset note resolution to obtain the melody annotation vector of the melody fragment training sample; based on the input training data composed of the sentence matrix training samples and corresponding emotion vector training samples, and the corresponding melody annotation vector, the initial model based on the self-attention mechanism is iteratively trained to obtain the model based on the self-attention mechanism.

2. The method as described in claim 1, characterized in that, The emotional information refers to emotional scores that measure the type and degree of emotion. Sentiment analysis was performed on each clause to obtain the corresponding sentiment information, including: Each clause is input into a preset word embedding model to obtain the word vector of each word in the corresponding clause; For each sentence, the word vector of the sentence is sequentially input into the trained sentiment analysis model to obtain the sentiment score of the sentence; the sentiment analysis model is trained based on different lyric sentences and their corresponding sentiment scores.

3. The method as described in claim 1, characterized in that, The emotion vector is a one-dimensional column vector; Based on the number of words in each clause, obtain the sentiment vector corresponding to the sentiment information of the corresponding clause, including: The number of words is determined as the number of rows in the sentiment vector; The emotion score is determined as a parameter value in the emotion vector.

4. The method as described in claim 1, characterized in that, After generating the sentiment vector corresponding to each clause based on the number of words in each clause and the sentiment information of the corresponding clause, the method further includes: The emotional vectors of each sentence are concatenated according to the order of the lyrics to obtain the concatenated emotional vector. After obtaining the sentence segmentation matrix of the corresponding sentence based on the word vector of each word in each sentence, the method further includes: According to the order of the lyrics, the sentence matrices of each sentence are concatenated to obtain the concatenated sentence matrix; The sentiment vector and the sentence segmentation matrix are concatenated to obtain a concatenated matrix, which includes: The concatenated emotion vector is concatenated to the last column of the concatenated sentence matrix to obtain the concatenated matrix.

5. The method as described in claim 1, characterized in that, Based on the melody vectors of each clause, the melody of the lyrics text is generated, including: A preset decoding method is used to decode the melody vector of each phrase to obtain the melody segment corresponding to each phrase; The melody of the lyrics is generated based on the order of the lyrics and the melodic fragments of each phrase.

6. The method as described in claim 2, characterized in that, The sentiment analysis model is trained in the following ways: Obtain the training samples of the lyrics phrases and the corresponding sentiment annotation scores of the training samples of the lyrics phrases; Based on the training samples of the lyrics and the sentiment annotation scores of the corresponding training samples of the lyrics, the neural network model is iteratively trained to obtain the sentiment analysis model.

7. The method as described in claim 1, characterized in that, According to a preset note resolution, each note in the melody fragment training sample is processed to obtain the melody annotation vector of the melody fragment training sample, including: Based on the melody fragment training samples, obtain the note name and duration of each note in the melody fragment training samples; According to the preset encoding method and preset note resolution, the note name and duration of each note are processed to obtain the melody annotation vector of the melody segment training sample.

8. A melody generation device, characterized in that, The device includes: an acquisition unit, an analysis unit, an input unit, and a generation unit; The acquisition unit is used to acquire at least one clause from the lyrics text; The analysis unit is used to perform sentiment analysis on each clause to obtain the sentiment information of the corresponding clause; The input unit is used to concatenate the emotional information of each clause with the corresponding clause, and then input the concatenation into a trained model based on a self-attention mechanism to obtain the melody vector corresponding to each clause; wherein, the model based on a self-attention mechanism is used to encode the current input data to obtain the encoded data of the current input data, and decode the encoded data of the current input data according to the melody vector obtained when decoding the previous input data to obtain the melody vector corresponding to the current input data; The generation unit is used to generate the melody of the lyrics text based on the melody vector of each clause; The device also includes a splicing unit; The generation unit is also used to generate the sentiment vector of the corresponding clause based on the number of words in each clause and the sentiment information of the corresponding clause; The acquisition unit is further configured to acquire the sentence matrix of the corresponding sentence based on the word vector of each word in each sentence, wherein the word vector of each word in each sentence is obtained by inputting each sentence into a preset word embedding model; The splicing unit is used to splice the emotion vector and the sentence segmentation matrix to obtain a splicing matrix; The input unit is specifically used to input the concatenated matrix into a trained model based on a self-attention mechanism; The device further includes a second training unit; The second training unit is used to perform the following steps: Obtain training samples of sentence segmentation matrices, melody fragments, and emotion vectors for different lyric phrases; For each lyric phrase, each note in the melody segment training sample is processed according to a preset note resolution to obtain the melody annotation vector of the melody segment training sample; The input training data, consisting of sentence matrix training samples of different lyric phrases and corresponding emotion vector training samples, as well as the corresponding melody annotation vector, is used to iteratively train the initial self-attention mechanism-based model to obtain a self-attention mechanism-based model.

9. The apparatus as claimed in claim 8, characterized in that, The emotional information refers to emotional scores that measure the type and degree of emotion. The analysis unit is specifically used to input each clause into a preset word embedding model to obtain the word vector of each word in the corresponding clause; For each clause, the word vectors of the clause are sequentially input into the trained sentiment analysis model to obtain the sentiment score of the clause; The sentiment analysis model is trained based on different lyric phrases and their corresponding sentiment scores.

10. The apparatus as claimed in claim 8, characterized in that, The emotion vector is a one-dimensional column vector; The acquisition unit is specifically used to determine the number of words as the number of rows in the sentiment vector; Furthermore, the emotional score is determined as a parameter value in the emotional vector.

11. The apparatus as claimed in claim 8, characterized in that, The splicing unit is also used to splice the emotional vectors of each sentence according to the order of the lyrics text to obtain a spliced ​​emotional vector; Furthermore, according to the order of the lyrics text, the sentence matrices of each sentence are concatenated to obtain a concatenated sentence matrix; The concatenated emotion vector is concatenated to the last column of the concatenated sentence matrix to obtain the concatenated matrix.

12. The apparatus as claimed in claim 8, characterized in that, The generation unit is specifically used to decode the melody vector of each phrase using a preset decoding method to obtain the melody segment corresponding to each phrase; Furthermore, based on the order of the lyrics text and the melodic fragments of each clause, the melody of the lyrics text is generated.

13. The apparatus as claimed in claim 9, characterized in that, The device also includes a first training unit; The first training unit is configured to perform the following steps: Obtain the training samples of the lyrics phrases and the corresponding sentiment annotation scores of the training samples of the lyrics phrases; Based on the training samples of the lyrics and the sentiment annotation scores of the corresponding training samples of the lyrics, the neural network model is iteratively trained to obtain the sentiment analysis model.

14. The apparatus as claimed in claim 8, characterized in that, The second training unit is further configured to obtain the note name and duration of each note in the melody fragment training sample based on the melody fragment training sample; Furthermore, according to a preset encoding method and a preset note resolution, the note name and duration of each note are processed to obtain the melody annotation vector of the melody segment training sample.

15. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to said at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is able to perform the steps of the method as described in any one of claims 1-7.

16. A storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the steps of the method as described in any one of claims 1-7.

Images