Song generation method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a song generation method and device, electronic equipment and storage medium. The method comprises: obtaining an original song selected for a target song to be generated and input lyrics text; extracting score information features of the original song, and determining text statistical information of the lyrics text; inputting the score information features and the text statistical information into a pre-trained score generation model to obtain a target score; the melody line of the target score is the same as that of the original song, and each note in the target score matches each character in the lyrics text; according to the target score, outputting a song audio with the lyrics being the lyrics text to obtain the target song. The present method can automatically adapt the original song to generate a target score based on the input lyrics text, and can obtain a melody that is the same as the melody line of the selected song and meets the specified text, ensuring the harmony of the melody and rhythm, and improving the song generation effect and processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to a song generation method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] With the development of multimedia technology, music and video production are becoming increasingly integrated. Music can help better express the emotions and storytelling conveyed in a video. When users create videos, if songs can be generated based on their input text, the videos can be significantly enhanced.

[0003] In related technologies, when user-inputted text is filled into the selected original score, there may be a mismatch between the user-inputted text and the original score, which cannot guarantee the harmony of the generated melody and rhythm, resulting in poor song generation and low processing efficiency. Summary of the Invention

[0004] This disclosure provides a song generation method, apparatus, electronic device, storage medium, and computer program product to at least solve the problems of poor song generation quality and low processing efficiency in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a song generation method is provided, comprising:

[0006] Obtain the original song selected for the target song to be generated and the input lyrics text;

[0007] Extract the musical score information features of the original song, and determine the textual statistical information of the lyrics;

[0008] The musical score information features and the text statistical information are input into a pre-trained musical score generation model to obtain a target musical score; the melody line of the target musical score is the same as that of the original song, and each note in the target musical score matches each character in the lyrics text;

[0009] Based on the target musical score, the lyrics are output as the audio of the song, thus obtaining the target song.

[0010] In one possible implementation, the step of inputting the musical score information features and the text statistical information into a pre-trained musical score generation model to obtain the target musical score includes:

[0011] The musical score information features and the text statistical information are input into the encoding network of the pre-trained musical score generation model to obtain hidden layer encoding features;

[0012] Obtain the target score features corresponding to the original song, and input the hidden layer encoding features and the target score features into the decoding network of the pre-trained score generation model to obtain the target score; wherein, the target score features are the features corresponding to the parts of the original song that do not need to be predicted.

[0013] In one possible implementation, the step of inputting the hidden layer encoded features and the target musical score features into the decoding network of the pre-trained musical score generation model to obtain the target musical score includes:

[0014] The hidden layer coding features and the target musical score features are processed by the decoding network to obtain predicted note features at N time points; wherein, the predicted note features at the first time point are the result obtained after decoding the hidden layer coding features and the target musical score features, and the predicted note features at the Nth time point are the result obtained after decoding the predicted note features at the (N-1)th time point and the target musical score features, where N is a positive integer greater than or equal to 2;

[0015] The target score is obtained based on the predicted note characteristics at the N time points.

[0016] In one possible implementation, the step of processing the hidden layer encoded features and the target musical score features through the decoding network to obtain predicted note features at N time points includes:

[0017] The hidden layer coding features and the target musical score features are processed by the decoding network to obtain the predicted note information at the first time step; the predicted note information includes pitch information.

[0018] By comparing the melody line extracted from the original song, the pitch information at the first moment is corrected to obtain the corrected pitch information. Based on the corrected pitch information, the predicted note features at the first moment are obtained.

[0019] In one possible implementation, the step of processing the hidden layer encoded features and the target musical score features through the decoding network to obtain predicted note features at N time points includes:

[0020] The predicted note features and the target musical score features at the (N-1)th time point are processed by the decoding network to obtain the predicted note information at the Nth time point; the predicted note information includes pitch information.

[0021] By comparing the melody line extracted from the original song, the pitch information at the Nth time point is corrected to obtain the corrected pitch information. Based on the corrected pitch information, the predicted note features at the Nth time point are obtained.

[0022] In one possible implementation, the extraction of the musical score information features of the original song includes:

[0023] The musical score and text information of the original song are determined; the text information is obtained based on the lyrics of the original song.

[0024] By combining the musical information and the textual information of the musical score, the musical score information features of the original song are obtained.

[0025] In one possible implementation, the pre-trained score generation model is trained using the following method:

[0026] Acquire training data; the training data includes the first sample musical score information features corresponding to some song fragments in the sample songs;

[0027] The first sample score information features and the second sample score information features are input into the score generation model to be trained to obtain the predicted score; the second sample score information features are obtained based on the complete song fragments of the sample songs;

[0028] Based on the predicted score and the sample song, the target loss information is determined, and the model parameters in the score generation model to be trained are adjusted according to the target loss information until the training termination condition is met, thus obtaining the pre-trained score generation model.

[0029] In one possible implementation, the step of outputting the lyrics as the audio of the lyrics text to obtain the target song, based on the target musical score, includes:

[0030] The target song is input into the acoustic model of the vocal synthesis model to obtain the score conversion information;

[0031] The vocoder model in the singing synthesis model is used to restore the music score conversion information to the singing audio, thus obtaining the target song.

[0032] According to a second aspect of the present disclosure, a song generation apparatus is provided, comprising:

[0033] The original song and input lyrics text acquisition unit is configured to acquire the original song and input lyrics text selected for the target song to be generated;

[0034] The unit for obtaining musical score information features and text statistics is configured to extract the musical score information features of the original song and determine the text statistics of the lyrics text;

[0035] The target score generation unit is configured to input the score information features and the text statistical information into a pre-trained score generation model to obtain the target score; the melody line of the target score is the same as the melody line of the original song, and each note in the target score matches each character in the lyrics text;

[0036] The target song acquisition unit is configured to execute a singing audio that outputs lyrics as the lyrics text based on the target musical score, thereby obtaining the target song.

[0037] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0038] processor;

[0039] Memory used to store the processor's executable instructions;

[0040] The processor is configured to execute the instructions to implement the song generation method as described in any of the preceding embodiments.

[0041] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the song generation method as described in any of the preceding claims.

[0042] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the song generation method as described in any of the preceding claims.

[0043] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0044] The disclosed solution obtains the original song and input lyrics text selected for the target song to be generated. It then extracts the musical score features of the original song and determines the textual statistics of the lyrics text. These features are then input into a pre-trained score generation model to obtain the target score. The melody line of the target score is identical to that of the original song, and each note in the target score matches each character in the lyrics text. Based on the target score, the lyrics are output as an audio recording of the lyrics text, thus obtaining the target song. In this way, a target score can be automatically generated from the original song to adapt to the user-input lyrics text, and the audio recording of the song can be output to obtain the target song. This ensures that the melody is identical to the selected song's melody line and conforms to the specified text, guaranteeing the harmony of melody and rhythm, and improving the song generation effect and processing efficiency.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0047] Figure 1 This is a flowchart illustrating a song generation method according to an exemplary embodiment.

[0048] Figure 2 This is a schematic diagram illustrating a song generation process according to an exemplary embodiment.

[0049] Figure 3 This is a schematic diagram illustrating a music score generation model processing flow according to an exemplary embodiment.

[0050] Figure 4 This is a schematic diagram illustrating a model training process according to an exemplary embodiment.

[0051] Figure 5 This is a flowchart illustrating another song generation method according to an exemplary embodiment.

[0052] Figure 6 This is a block diagram illustrating a song generation apparatus according to an exemplary embodiment.

[0053] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0055] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure.

[0056] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0057] Figure 1 This is a flowchart illustrating a song generation method according to an exemplary embodiment. This embodiment uses the application of this method to a terminal as an example for illustration. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] In step S110, the original song selected for the target song to be generated and the input lyrics text are obtained;

[0059] In practical applications, in response to a song generation operation, the system can obtain the original song selected for the target song to be generated, and receive the lyrics text input for the target song to be generated, for example, such as... Figure 2 As shown, the user-selected song can be used as the original song, and the user-inputted text can be used as the lyrics text, so as to further analyze and process based on the original song and the input lyrics text.

[0060] In step S120, the musical score information features of the original song are extracted, and the textual statistical information of the lyrics is determined;

[0061] Among them, the musical score information features can be a feature sequence organized in compound-word form based on the musical score and textual information of the original song. This feature sequence can include multiple compound-word structural features of different feature types and can be arranged based on the temporal sequence of the original song.

[0062] As an example, text statistics may include, but are not limited to, the number of words in the lyrics text, sentence segmentation information, etc.

[0063] In practical implementation, for the original song, the musical score and text information of the original song can be extracted. Then, by combining the musical score and text information, the musical score information features of the original song can be obtained. For the input lyrics text, text statistics such as the number of words and sentence segmentation information can be obtained. In order to generate a new song based on the musical score information features and the text statistics of the lyrics text.

[0064] In one example, by analyzing the rhythm and lyrics of the original song, musical information of the score can be obtained based on the musical content of the original song, and musical text information of the score can be obtained based on the lyrics of the original song. For example, the segmentation information of the singing phrases and the number of words corresponding to each singing phrase can be counted. The pitch and duration of each note and the note corresponding to each lyric can be obtained. The chord information in the score can be simplified into triads, etc. Other musical information and musical text information of the score can also be obtained. No specific limitations are made in this embodiment.

[0065] In another example, for the original song, features can be organized in compound-word form according to the obtained musical and textual information of the score, resulting in multiple compound-word structural features of different feature types. The compound-word structural feature sequence obtained by arranging them in chronological order can then be used as the score information feature.

[0066] In step S130, the musical score information features and text statistical information are input into the pre-trained musical score generation model to obtain the target musical score;

[0067] The target score has the same melody line as the original song, and each note in the target score matches each character in the lyrics. For example, it can generate a melody with the same melody line as the original melody fragment and a specified number of words, so as to ensure the harmony of melody and rhythm to a certain extent.

[0068] After obtaining the characteristics of the musical score and the statistical information of the text, such as Figure 2 As shown, the features of the musical score and the statistical information of the text can be input into the pre-trained musical score generation model to obtain the generated musical score (i.e. the target musical score). By generating a musical score with a specified number of words and the same melody line as the input melody, the problem of missing data in traditional song search-based methods can be avoided, as well as the reliance on music domain expert knowledge in traditional rule-based text filling methods.

[0069] Specifically, the pre-trained music score generation model can be a compound-word-based transformer model. Through processing by the pre-trained music score generation model, the music score of the original song can be adapted to fit the lyrics text input by the user. For example, in order to ensure that the duration of the original song remains unchanged during the adaptation, based on the lyrics text input by the user, the number of notes, the duration of the note corresponding to each word, the pitch of the notes, etc. in the original song can be adjusted according to the increase or decrease of the number of words in the lyrics text. In this way, the melody line of the generated melody can be the same as the melody line of the original melody and meet the specified number of words. That is, the melody line of the target music score is the same as the melody line of the original song, and each note in the target music score matches each character in the lyrics text.

[0070] In step S140, based on the target score, the audio of the song with lyrics as text is output to obtain the target song.

[0071] After obtaining the target sheet music, such as Figure 2 As shown, the target song can be input into the acoustic model of the vocal synthesis model to obtain the score conversion information. Then, the score conversion information can be restored into vocal audio through the vocoder model in the vocal synthesis model, so as to output the vocal audio of the lyrics as the lyrics text, thus obtaining the target song.

[0072] For example, by analyzing the rhythm and lyrics of the original song and combining it with the lyrics text input by the user, a brand new musical score can be generated. Furthermore, based on a vocal synthesis model, an AI singer can generate the audio recording of the song, thus obtaining the target song.

[0073] Compared to traditional methods, the technical solution in this embodiment generates target scores by using a pre-trained score generation model. It is data-driven, flexible in its generation method, and does not require extensive knowledge from music experts or a large score library for searching and matching. Furthermore, based on a score generation model trained using neural network technology that generates melodies from melody fragments, it can be used to generate melodies with the same melody line as the original melody fragments and that meet the specified number of words, thus ensuring the harmony of melody and rhythm to a certain extent.

[0074] In the aforementioned song generation method, the original song and input lyrics text are obtained for the target song to be generated. Then, the musical score features of the original song and the textual statistics of the lyrics text are extracted. These features and statistics are then input into a pre-trained musical score generation model to obtain the target score. Based on the target score, the audio recording of the lyrics text is output, thus obtaining the target song. In this way, the target score can be automatically generated from the original song to match the user-input lyrics text, and the audio recording of the song can be output to obtain the target song. This method can obtain a melody that is identical to the selected song's melody line and conforms to the specified text, ensuring the harmony of melody and rhythm, and improving the song generation effect and processing efficiency.

[0075] In one exemplary embodiment, the process of inputting musical score information features and textual statistical information into a pre-trained musical score generation model to obtain a target musical score includes: inputting musical score information features and textual statistical information into the encoding network of the pre-trained musical score generation model to obtain hidden layer encoding features; obtaining the target musical score features corresponding to the original song; and inputting the hidden layer encoding features and the target musical score features into the decoding network of the pre-trained musical score generation model to obtain the target musical score.

[0076] As an example, the target score features can be features in the original song that do not need to be predicted, such as chords, measure positions, and other features in the original song's score that do not need to be predicted.

[0077] In practical applications, such as Figure 3 As shown, the characteristics of the musical score and textual statistical information (such as...) can be combined. Figure 3 The input features are fed into the encoding network of the pre-trained music score generation model to obtain hidden layer encoding features. Then, features that do not need to be predicted, such as chords and measure positions, in the original song's score (i.e., target score features) can be obtained. The hidden layer encoding features and the features that do not need to be predicted are then fed into the decoding network of the pre-trained music score generation model. By predicting the note features at each time point, a brand new score can be generated, that is, the target score is obtained.

[0078] The technical solution of this embodiment obtains hidden layer coding features by inputting the musical score information features and text statistical information into the encoding network of the pre-trained musical score generation model, thereby obtaining the target musical score features corresponding to the original song. The hidden layer coding features and the target musical score features are then input into the decoding network of the pre-trained musical score generation model to obtain the target musical score. Based on the pre-trained musical score generation model, the target musical score can be flexibly generated with data as the driving force.

[0079] In one exemplary embodiment, the hidden layer coding features and the target musical score features are input into the decoding network of a pre-trained musical score generation model to obtain the target musical score, including: processing the hidden layer coding features and the target musical score features through the decoding network to obtain predicted note features at N time points; and obtaining the target musical score based on the predicted note features at the N time points.

[0080] The predicted note features at the first time step can be derived from the hidden layer encoding features and the target score features (such as...). Figure 3 The result obtained after decoding the features that do not need to be predicted is that the predicted note features at time N are the same as the predicted note features at time N-1 (e.g., ...). Figure 3 The result obtained by decoding the predicted note features and the target musical score features, where N is a positive integer greater than or equal to 2;

[0081] In one example, such as Figure 3 As shown, the hidden layer coding features and target musical score features can be processed by the decoding network to obtain the predicted note information at the first time step (e.g., Figure 3 The note information at time t_0 is predicted, and then compared with the melody line extracted from the original song to correct the pitch information at time 1 (e.g., ...). Figure 3By correcting the pitch information at time t_0, the predicted note information at each time can be predicted one by one and the corresponding pitch information can be corrected to obtain the predicted note features at N times. Based on the predicted note features at these N times, a brand new score (i.e., the target score) can be generated.

[0082] The technical solution of this embodiment processes the hidden layer coding features and the target score features through a decoding network to obtain the predicted note features at N time points. Then, based on the predicted note features at N time points, the target score is obtained. The predicted note features at each time point can be predicted one by one, providing data support for obtaining the target score.

[0083] In an exemplary embodiment, the method of processing hidden layer coding features and target musical score features through a decoding network to obtain predicted note features at N time points includes: processing hidden layer coding features and target musical score features through a decoding network to obtain predicted note information at the first time point; comparing the pitch information at the first time point with the melody line extracted from the original song to obtain corrected pitch information; and obtaining predicted note features at the first time point based on the corrected pitch information.

[0084] The predicted note information may include pitch information, such as the note duration, pitch, and correspondence between the note and lyrics at each moment.

[0085] In specific implementations, such as Figure 3 As shown, by processing the hidden layer encoded features and the target musical score features through the decoding network, the predicted note information at the first time step (such as...) can be obtained. Figure 3 The note information at time t_0 is predicted, and then compared with the melody line extracted from the original song to correct the pitch information at time 1 (e.g., ...). Figure 3 (Correct the pitch information at time t_0) to obtain the predicted note features at time 1.

[0086] In one alternative embodiment, the melody line can be extracted from the score of the original song. Then, by comparing the melody line of the generated melody with that of the original melody, corrections can be made based on the pitch trend in the melody line. This can avoid pitch abnormalities in the generated melody line, ensuring that the melody line of the target score is the same as that of the original song.

[0087] The technical solution of this embodiment processes the hidden layer coding features and the target score features through a decoding network to obtain the predicted note information at the first moment. Then, it compares the pitch information at the first moment with the melody line extracted from the original song to obtain the corrected pitch information. Based on the corrected pitch information, the predicted note features at the first moment are obtained, which provides data support for obtaining the predicted note features at the subsequent N moments.

[0088] In one exemplary embodiment, the method of processing hidden layer coding features and target musical score features through a decoding network to obtain predicted note features at N time points includes: processing the predicted note features and target musical score features at the (N-1)th time point through a decoding network to obtain predicted note information at the Nth time point; the predicted note information includes pitch information; comparing the pitch information at the Nth time point with the melody line extracted from the original song to obtain corrected pitch information; and obtaining the predicted note features at the Nth time point based on the corrected pitch information.

[0089] The predicted note information may include pitch information, such as the note duration, pitch, and correspondence between the note and lyrics at each moment.

[0090] In practical applications, notes can be predicted sequentially, moment by moment. For example, the features of already predicted notes can be used to predict notes at the next moment. Figure 3 As shown, by processing the predicted note features and target score features at time N-1 through a decoding network, the predicted note information at time N can be obtained (e.g., Figure 3 The note information at time t_i is predicted, and compared with the melody line extracted from the original song to correct the pitch information at time N (e.g., Figure 3 (Correct the pitch information at time t_i) to obtain the predicted note features at time N.

[0091] For example, the melody line can be extracted from the original song's score. By comparing the generated melody line with the original melody line, and making corrections based on the pitch trend in the melody line, the corresponding pitch information can be corrected for each note at each moment to avoid abnormal pitches in the predicted notes at each moment.

[0092] The technical solution of this embodiment processes the predicted note features and target score features at the (N-1)th time step through a decoding network to obtain the predicted note information at the Nth time step. Then, it compares the pitch information at the Nth time step with the melody line extracted from the original song to obtain the corrected pitch information. Based on the corrected pitch information, the predicted note features at the Nth time step are obtained. This allows for the prediction and correction of notes at each time step, avoiding pitch anomalies and improving the song generation effect.

[0093] In an exemplary embodiment, extracting the sheet music information features of the original song includes: determining the sheet music music information and sheet music text information of the original song; and combining the sheet music music information and sheet music text information to obtain the sheet music information features of the original song.

[0094] The musical score text information can be obtained from the lyrics text of the original song.

[0095] In practical implementation, the rhythm and lyrics of the original song can be analyzed to obtain musical information based on the musical content of the original song, and musical information based on the lyrics. For example, the segmentation information of the singing phrases and the number of words corresponding to each singing phrase can be counted. The pitch and duration of each note and the note corresponding to each lyric can be obtained. The chord information in the musical score can be simplified into triads. Then, based on the obtained musical and musical information, features can be organized in compound-word form to obtain multiple compound-word structural features of different feature types. The compound-word structural feature sequence arranged in chronological order can be used as the musical score information feature of the original song.

[0096] For example, compound-word structural features of different feature types can be represented as follows:

[0097] type 0EOS: Termination information;

[0098] The vector is [0,0,0,0,0,0,0,0,0,0,0,0]

[0099] type 1GLOBAL: phrase, tempo, key_mode;

[0100] The vector is [0,phrase,tempo,key_mode,0,0,0,0,0,0,0]

[0101] type 2PHRASE:phrase_word_nums;

[0102] The vector is [0,0,0,0,phrase_word_nums,0,0,0,0,0,0]

[0103] type 3METRIC: bar / beat,chord;

[0104] The vector is [0,0,0,0,0,bar / beat,chord,0,0,0,0]

[0105] type 4NOTES: pitch, duration, word_index, word_nums;

[0106] The vector is [0,0,0,0,0,0,0,pitch,duration,word_index,word_nums]

[0107] The technical solution of this embodiment determines the musical information and textual information of the original song, and then combines the musical information and textual information to obtain the musical information features of the original song, providing data support for further generating a brand new musical score.

[0108] In an exemplary embodiment, the pre-trained music score generation model is trained by the following method: acquiring training data; inputting the first sample music score information features and the second sample music score information features into the music score generation model to be trained to obtain a predicted music score; determining the target loss information based on the predicted music score and the sample songs, and adjusting the model parameters in the music score generation model to be trained based on the target loss information until the training termination condition is met, thereby obtaining the pre-trained music score generation model.

[0109] As an example, the training data may include first sample sheet music information features corresponding to partial song segments in the sample songs; second sample sheet music information features may be obtained based on complete song segments of the sample songs, and the predicted sheet music may be the sheet music corresponding to the complete song segments of the sample songs.

[0110] In one example, the music score generation model to be trained can adopt a compound-word-based transformer structure, such as... Figure 4As shown, the features of the compound-word structure (i.e., the first sample score information features corresponding to a part of the song fragment) can be input into the transformer-encoder to obtain the hidden layer encoding features. Then, the hidden layer encoding features and the complete score features (i.e., the second sample score information features obtained based on the complete song fragments of the sample song) can be input into the transformer-decoder to obtain the output predicted score. Then, the loss function can be calculated based on the predicted score and the real score (i.e., the sample song) to train the score generation model.

[0111] For example, the features input during model training can be the features of a missing melody segment in a sample song. Then, by predicting the missing melody segment, the predicted score corresponding to the complete song segment can be obtained, and the loss function can be calculated based on the predicted score and the real score.

[0112] In an optional embodiment, the music score generation model to be trained can also adopt a MIDI-like transformer structure or a REMI-based transformer structure. Compared with these two model structures, using music score features based on compound-word representation can make the music score feature length shorter, which is beneficial for handling long-distance dependencies in music. Moreover, adding different types of tokens will not increase the sequence length, which is highly scalable. It also supports differentiated processing of different types of tokens, such as different prediction heads, different embedding sizes, and different sampling methods.

[0113] The technical solution of this embodiment acquires training data, then inputs the first sample score information features and the second sample score information features into the score generation model to be trained to obtain a predicted score. Then, based on the predicted score and the sample song, the target loss information is determined, and the model parameters in the score generation model to be trained are adjusted according to the target loss information until the training termination condition is met, thus obtaining a pre-trained score generation model. It can train a score generation model from melody fragment to melody fragment based on neural network technology, so as to generate a melody with the same melody line as the original melody fragment and conforming to the specified number of words, thus ensuring the harmony of melody and rhythm.

[0114] In one exemplary embodiment, the process of obtaining a target song by outputting lyrics as lyrics text and singing audio based on a target score includes: inputting the target song into the acoustic model in a singing synthesis model to obtain score conversion information; and restoring the score conversion information into singing audio through the vocoder model in the singing synthesis model to obtain the target song.

[0115] In one example, an acoustic model based on FastSpeech can be used to convert the score of the target song into a mel score (i.e., score conversion information). Then, a vocoder model based on GAN can be used to restore the mel score to the vocal audio, thus obtaining the target song.

[0116] In an optional embodiment, an acoustic model based on tacotron, such as the ByteSing model, can also be used, and no specific limitation is made in this embodiment; for the vocoder model in the singing synthesis model, a signal-based vocoder or an RNN-based vocoder, such as WaveRNN, LPCNet, etc., can also be used, and no specific limitation is made in this embodiment.

[0117] The technical solution of this embodiment obtains the score conversion information by inputting the target song into the acoustic model of the singing synthesis model, and then restores the score conversion information into singing audio through the vocoder model in the singing synthesis model to obtain the target song. It can generate singing audio through AI singer, thereby improving the song generation effect and processing efficiency.

[0118] Figure 5 This is a flowchart illustrating another song generation method according to an exemplary embodiment, such as... Figure 5 As shown, this method is used in computer devices such as terminals and includes the following steps.

[0119] In step S510, training data is acquired; the training data includes the first sample score information features corresponding to some song segments in the sample songs; the first sample score information features and the second sample score information features are input into the score generation model to be trained to obtain the predicted score. In step S520, based on the predicted score and the sample songs, the target loss information is determined, and the model parameters in the score generation model to be trained are adjusted according to the target loss information until the training termination condition is met, thus obtaining the pre-trained score generation model. In step S530, the original song selected for the target song to be generated and the input lyrics text are acquired. In step S540, the score music information and score text information of the original song are determined; the score music information and score text information are combined to obtain the score information features of the original song, and the text statistical information of the lyrics text is determined. In step S550, the score information features and text statistical information are input into the encoding network of the pre-trained score generation model to obtain the hidden layer encoding features. In step S560, the target score features corresponding to the original song are obtained. The hidden layer coding features and the target score features are processed by a decoding network to obtain predicted note features at N time points. Based on the predicted note features at N time points, the target score is obtained. In step S570, the target song is input into the acoustic model of the vocal synthesis model to obtain score conversion information. The score conversion information is then restored to vocal audio using the vocoder model in the vocal synthesis model to obtain the target song. It should be noted that the specific limitations of the above steps can be found in the specific limitations of a song generation method described above, and will not be repeated here.

[0120] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0121] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0122] Based on the same inventive concept, this disclosure also provides a song generation apparatus for implementing the song generation method described above.

[0123] Figure 6 This is a block diagram illustrating a song generation apparatus according to an exemplary embodiment. (Refer to...) Figure 6 The device includes:

[0124] The original song and input lyrics text acquisition unit 601 is configured to acquire the original song and input lyrics text selected for the target song to be generated;

[0125] The music score information feature and text statistical information obtaining unit 602 is configured to extract the music score information features of the original song and determine the text statistical information of the lyrics text;

[0126] The target score generation unit 603 is configured to input the score information features and the text statistical information into a pre-trained score generation model to obtain a target score; the melody line of the target score is the same as the melody line of the original song, and each note in the target score matches each character in the lyrics text;

[0127] The target song acquisition unit 604 is configured to execute a singing audio that outputs lyrics as the lyrics text according to the target score, thereby obtaining the target song.

[0128] In one possible implementation, the target score generation unit 603 is specifically configured to: input the score information features and the text statistical information into the encoding network of the pre-trained score generation model to obtain hidden layer encoding features; obtain the target score features corresponding to the original song; and input the hidden layer encoding features and the target score features into the decoding network of the pre-trained score generation model to obtain the target score; wherein the target score features are the features corresponding to the parts of the original song that do not need to be predicted.

[0129] In one possible implementation, the target score generation unit 603 is specifically configured to process the hidden layer coding features and the target score features through the decoding network to obtain predicted note features at N time points; wherein, the predicted note features at the first time point are the result obtained after decoding the hidden layer coding features and the target score features, and the predicted note features at the Nth time point are the result obtained after decoding the predicted note features at the (N-1)th time point and the target score features, where N is a positive integer greater than or equal to 2; the target score is obtained based on the predicted note features at the N time points.

[0130] In one possible implementation, the target score generation unit 603 is specifically configured to process the hidden layer coding features and the target score features through the decoding network to obtain the predicted note information at the first time step; the predicted note information includes pitch information; the pitch information at the first time step is corrected by comparing it with the melody line extracted from the original song to obtain the corrected pitch information; and the predicted note features at the first time step are obtained based on the corrected pitch information.

[0131] In one possible implementation, the target score generation unit 603 is specifically configured to process the predicted note features and the target score features at the (N-1)th time step through the decoding network to obtain the predicted note information at the Nth time step; the predicted note information includes pitch information; the pitch information at the Nth time step is corrected by comparing it with the melody line extracted from the original song to obtain corrected pitch information; and the predicted note features at the Nth time step are obtained based on the corrected pitch information.

[0132] In one possible implementation, the music score information feature and text statistical information obtaining unit 602 is specifically configured to perform the following: determine the music score information and music score text information of the original song; the music score text information is obtained based on the lyrics text of the original song; combine the music score information and the music score text information to obtain the music score information features of the original song.

[0133] In one possible implementation, the song generation apparatus further includes:

[0134] The training data acquisition unit is specifically configured to acquire training data; the training data includes the first sample musical score information features corresponding to some song segments in the sample songs.

[0135] The predicted score unit is specifically configured to input the first sample score information features and the second sample score information features into the score generation model to be trained, and obtain the predicted score; the second sample score information features are obtained based on the complete song fragments of the sample songs;

[0136] The model training unit is specifically configured to determine the target loss information based on the predicted score and the sample song, and adjust the model parameters in the score generation model to be trained based on the target loss information until the training termination condition is met, thereby obtaining the pre-trained score generation model.

[0137] In one possible implementation, the target song obtaining unit 604 is specifically configured to execute an acoustic model that inputs the target song into a vocal synthesis model to obtain music score conversion information; and then, through a vocoder model in the vocal synthesis model, restore the music score conversion information to the vocal audio to obtain the target song.

[0138] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0139] Each module in the aforementioned song generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0140] Figure 7 This is a block diagram illustrating an electronic device 700 for implementing a song generation method according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0141] Reference Figure 7 The electronic device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0142] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0143] Memory 704 is configured to store various types of data to support the operation of electronic device 700. Examples of such data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene memory.

[0144] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0145] Multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0146] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0147] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0148] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 can detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or its components, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0149] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0150] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0151] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0152] In an exemplary embodiment, a computer program product is also provided, which includes instructions that can be executed by a processor 720 of an electronic device 700 to perform the above-described method.

[0153] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0154] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0155] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A song generation method characterized by comprising: The method includes: Obtain the original song selected for the target song to be generated and the input lyrics text; Extract the musical score information features of the original song, and determine the textual statistical information of the lyrics; The musical score information features and the text statistical information are input into the encoding network of the pre-trained musical score generation model to obtain the hidden layer encoding features; The target score features corresponding to the original song are obtained, and the hidden layer encoding features and the target score features are input into the decoding network of the pre-trained score generation model to obtain the target score. The target score features are the features corresponding to the parts of the original song that do not need to be predicted. The melody line of the target score is the same as that of the original song, and each note in the target score matches each character in the lyrics text. Based on the target musical score, the lyrics are output as the audio of the song, thus obtaining the target song.

2. The method according to claim 1, characterized in that, The step of inputting the hidden layer encoded features and the target musical score features into the decoding network of the pre-trained musical score generation model to obtain the target musical score includes: The hidden layer coding features and the target musical score features are processed by the decoding network to obtain predicted note features at N time points; wherein, the predicted note features at the first time point are the result obtained after decoding the hidden layer coding features and the target musical score features, and the predicted note features at the Nth time point are the result obtained after decoding the predicted note features at the (N-1)th time point and the target musical score features, where N is a positive integer greater than or equal to 2; The target score is obtained based on the predicted note characteristics at the N time points.

3. The method according to claim 2, characterized in that, The process of processing the hidden layer encoded features and the target musical score features through the decoding network to obtain predicted note features at N time points includes: The hidden layer coding features and the target musical score features are processed by the decoding network to obtain the predicted note information at the first time step; the predicted note information includes pitch information. By comparing the melody line extracted from the original song, the pitch information at the first moment is corrected to obtain the corrected pitch information. Based on the corrected pitch information, the predicted note features at the first moment are obtained.

4. The method according to claim 2, characterized in that, The process of processing the hidden layer encoded features and the target musical score features through the decoding network to obtain predicted note features at N time points includes: The predicted note features and the target musical score features at the (N-1)th time point are processed by the decoding network to obtain the predicted note information at the Nth time point; the predicted note information includes pitch information. By comparing the melody line extracted from the original song, the pitch information at the Nth time point is corrected to obtain the corrected pitch information. Based on the corrected pitch information, the predicted note features at the Nth time point are obtained.

5. The method according to claim 1, characterized in that, The extraction of the musical score information features of the original song includes: The musical score and text information of the original song are determined; the text information is obtained based on the lyrics of the original song. By combining the musical information and the textual information of the musical score, the musical score information features of the original song are obtained.

6. The method according to claim 1, characterized in that, The pre-trained music score generation model was trained using the following method: Acquire training data; the training data includes the first sample musical score information features corresponding to some song fragments in the sample songs; The first sample score information features and the second sample score information features are input into the score generation model to be trained to obtain the predicted score; the second sample score information features are obtained based on the complete song fragments of the sample songs; Based on the predicted score and the sample song, the target loss information is determined, and the model parameters in the score generation model to be trained are adjusted according to the target loss information until the training termination condition is met, thus obtaining the pre-trained score generation model.

7. The method according to any one of claims 1 to 6, characterized in that, The step of outputting the lyrics as the lyrics text and obtaining the target song based on the target musical score includes: The target song is input into the acoustic model of the vocal synthesis model to obtain the score conversion information; The vocoder model in the singing synthesis model is used to restore the music score conversion information to the singing audio, thus obtaining the target song.

8. A song generation device, characterized in that, The device includes: The original song and input lyrics text acquisition unit is configured to acquire the original song and input lyrics text selected for the target song to be generated; The unit for obtaining musical score information features and text statistics is configured to extract the musical score information features of the original song and determine the text statistics of the lyrics text; The target score generation unit is configured to input the score information features and the text statistical information into the encoding network of a pre-trained score generation model to obtain hidden layer encoding features; obtain the target score features corresponding to the original song; input the hidden layer encoding features and the target score features into the decoding network of the pre-trained score generation model to obtain the target score; the target score features are the features corresponding to the parts of the original song that do not need to be predicted; the melody line of the target score is the same as the melody line of the original song, and each note in the target score matches each character in the lyrics text; The target song acquisition unit is configured to execute a singing audio that outputs lyrics as the lyrics text based on the target musical score, thereby obtaining the target song.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the song generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the song generation method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the song generation method as described in any one of claims 1 to 7.

Images