Song generation method, song generation model training method, device and equipment

Abstract

The present disclosure relates to a song generation method and device, electronic equipment and storage medium, belonging to the technical field of computers. The method comprises: performing feature extraction on a lyric text to obtain first lyric features; mapping the first lyric features to first transformation parameters; performing feature transformation on first noise features based on the first transformation parameters to obtain second noise features, the first noise features being randomly generated noise features; and generating a target song matching the lyric text based on the first lyric features and the second noise features. The present disclosure does not require the knowledge of musical scores, improves the convenience of generating songs, can more comprehensively capture and utilize lyric features by dynamically adjusting the transformation parameters, and is conducive to ensuring the quality of the generated songs.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a song generation method, a training method, device and equipment of a song generation model. BACKGROUND

[0002] With the development of voice technology and artificial intelligence, the production methods of songs are more and more diversified. In addition to recording the singing voice of a singer and synthesizing the singing voice with accompaniment to generate a song, some song generation methods based on artificial intelligence technology are currently provided.

[0003] In the related art, a song generation model is used to generate a corresponding song according to lyrics and score information, but this method depends on professional score information and is not simple enough for intelligent song generation. SUMMARY

[0004] The present disclosure provides a song generation method, a training method, device and equipment of a song generation model, which can improve the convenience and quality of song generation.

[0005] According to an aspect of an embodiment of the present disclosure, a song generation method is provided, and the method comprises:

[0006] performing feature extraction on a lyrics text to obtain first lyrics features;

[0007] mapping the first lyrics features to first transformation parameters;

[0008] performing feature transformation on first noise features based on the first transformation parameters to obtain second noise features, wherein the first noise features are randomly generated noise features;

[0009] generating a target song matching the lyrics text based on the first lyrics features and the second noise features.

[0010] Optionally, the generating the target song matching the lyrics text based on the first lyrics features and the second noise features comprises:

[0011] performing cross processing on the first lyrics features and the second noise features to obtain second lyrics features by fusion;

[0012] generating the target song matching the lyrics text based on the second lyrics features.

[0013] Optionally, the mapping the first lyrics features to first transformation parameters comprises:

[0014] mapping the first lyrics features to the first transformation parameters and second transformation parameters;

[0015] The generating, based on the second lyric feature, a target song matching the lyric text comprises:

[0016] The feature transformation is performed on the second lyric feature based on the second transformation parameter to obtain a third lyric feature.

[0017] The third lyric feature is decoded to obtain the target song matching the lyric text.

[0018] Optionally, the method further comprises:

[0019] determining a current time step feature, the time step feature representing a position of a current audio frame to be generated in the target song, the target song comprising a plurality of audio frames;

[0020] The first lyric feature is fused with the time step feature to obtain a first fused feature.

[0021] The first fused feature is mapped to the first transformation parameter, or the first fused feature is mapped to the first transformation parameter and the second transformation parameter.

[0022] Optionally, the feature extraction on the lyric text to obtain a first lyric feature comprises:

[0023] The lyric text is converted into a phoneme sequence, the phoneme sequence comprising phonemes of a plurality of words in the lyric text.

[0024] The phoneme sequence is subjected to feature extraction to obtain the first lyric feature.

[0025] Optionally, the cross processing of the first lyric feature and the second noise feature to obtain a second lyric feature comprises:

[0026] A song attribute feature of a reference song is obtained, the song attribute feature comprising at least one of a song style feature or a song duration feature.

[0027] The first lyric feature and the song attribute feature are fused to obtain a second fused feature.

[0028] The second fused feature and the second noise feature are cross processed to obtain the second lyric feature.

[0029] The song attribute of the target song is consistent with the song attribute of the reference song.

[0030] Optionally, the cross processing of the second fused feature and the second noise feature to obtain the second lyric feature comprises:

[0031] The first mapping matrix is used for feature mapping of the second fusion feature to obtain a query feature;

[0032] The second mapping matrix is used for feature mapping of the second noise feature to obtain a key feature and a value feature;

[0033] The query feature, the key feature and the value feature are fused to obtain the second lyrics feature.

[0034] Optionally, the steps of generating the first transformation parameter, the second noise feature and the second lyrics feature are performed by k adaptive transformation networks, and the k adaptive transformation networks are connected in sequence; the mapping of the first lyrics feature into the first transformation parameter comprises:

[0035] The first lyrics feature is mapped into k first transformation parameters through the k adaptive transformation networks respectively;

[0036] The second noise feature is obtained by performing feature transformation on the first noise feature based on the first transformation parameter, and the method comprises:

[0037] The first noise feature is transformed into the first second noise feature based on the first first transformation parameter through the first adaptive transformation network;

[0038] The m-1 third lyrics feature is transformed into the m second noise feature based on the m first transformation parameter through the m adaptive transformation network; wherein the m-1 third lyrics feature is the third lyrics feature output by the m-1 adaptive transformation network, and m is an integer greater than 1 and not greater than k.

[0039] Optionally, the first lyrics feature and the second noise feature are cross-processed to obtain the second lyrics feature, and the method comprises:

[0040] The first lyrics feature and the first second noise feature are cross-processed through the first adaptive transformation network to obtain the first second lyrics feature;

[0041] The first lyrics feature and the m second noise feature are cross-processed through the m adaptive transformation network to obtain the m second lyrics feature.

[0042] Optionally, the k adaptive transformation networks further generate k second transformation parameters; and the target song matched with the lyrics text is generated based on the second lyrics feature, and the method comprises:

[0043] The first second lyric feature is subjected to feature transformation based on the first second transformation parameter through the first adaptive transformation network, and the first third lyric feature is obtained.

[0044] The mth second lyric feature is subjected to feature transformation based on the mth second transformation parameter through the mth adaptive transformation network, and the mth third lyric feature is obtained.

[0045] The kth third lyric feature is subjected to decoding through the decoding network, and the target song is obtained.

[0046] According to still another aspect of the embodiments of the present disclosure, a training method of a song generation model is provided, and the method comprises:

[0047] A sample lyric text and a sample song matched with the sample lyric text are obtained.

[0048] The sample lyric text is subjected to feature extraction through the second song generation model, and a first sample lyric feature is obtained.

[0049] The first sample lyric feature is mapped to a first sample transformation parameter through the second song generation model.

[0050] The first sample noise feature is subjected to feature transformation based on the first sample transformation parameter through the second song generation model, and a second sample noise feature is obtained, wherein the first sample noise feature is a randomly generated noise feature.

[0051] The sample song is subjected to encoding through the second song generation model, and a target sample lyric feature is obtained.

[0052] The second song generation model is trained based on the first sample lyric feature, the second sample noise feature and the target sample lyric feature, and a first song generation model is obtained, wherein the first song generation model is used to generate a song matched with any lyric text.

[0053] Optionally, the model parameters of the second song generation model are trained based on the first sample lyric feature, the second sample noise feature and the target sample lyric feature, and the first song generation model is obtained, comprising:

[0054] The first sample lyric feature and the second sample noise feature are subjected to cross processing through the second song generation model, and a second sample lyric feature is obtained.

[0055] A loss parameter is determined based on the second sample lyric feature and the target sample lyric feature.

[0056] Train the second song generation model based on the loss parameter to obtain the first song generation model.

[0057] Optionally, the second song generation model comprises a feature extraction network, an adaptive transformation network and an encoding network, the step of generating the first sample lyric feature is performed by the feature extraction network; the steps of generating the first sample transformation parameter, the second sample noise feature and the second sample lyric feature are performed by the adaptive transformation network; and the step of generating the target sample lyric feature is performed by the encoding network.

[0058] The training of the second song generation model based on the loss parameter to obtain the first song generation model comprises:

[0059] Adjusting model parameters of the second song generation model based on the loss parameter to obtain a third song generation model.

[0060] Removing the encoding network in the third song generation model and adding a trained decoding network in the third song generation model to obtain the first song generation model.

[0061] According to still another aspect of the embodiments of the present disclosure, a song generation device is provided, which comprises:

[0062] a feature extraction unit configured to perform feature extraction on a lyric text to obtain a first lyric feature;

[0063] a feature processing unit configured to perform mapping of the first lyric feature to a first transformation parameter;

[0064] The feature processing unit is further configured to perform feature transformation on a first noise feature based on the first transformation parameter to obtain a second noise feature, the first noise feature being a randomly generated noise feature.

[0065] The feature processing unit is further configured to perform generation of a target song matching the lyric text based on the first lyric feature and the second noise feature.

[0066] Optionally, the feature processing unit is configured to perform:

[0067] cross processing of the first lyric feature and the second noise feature to obtain a second lyric feature by fusion;

[0068] Generation of a target song matching the lyric text based on the second lyric feature.

[0069] Optionally, the feature processing unit is configured to perform mapping the first lyric feature into the first transformation parameter and the second transformation parameter.

[0070] The feature processing unit is further configured to perform feature transformation on the second lyric feature based on the second transformation parameter to obtain a third lyric feature.

[0071] The feature processing unit is further configured to perform decoding on the third lyric feature to obtain a target song matching the lyric text.

[0072] Optionally, the feature processing unit is further configured to perform:

[0073] determining a current time step feature, the time step feature representing a position of a current audio frame to be generated in the target song, the target song including a plurality of audio frames;

[0074] fusing the first lyric feature and the time step feature to obtain a first fused feature;

[0075] mapping the first fused feature into the first transformation parameter, or mapping the first fused feature into the first transformation parameter and the second transformation parameter.

[0076] Optionally, the feature extraction unit is configured to perform:

[0077] converting the lyric text into a phoneme sequence, the phoneme sequence including phonemes of a plurality of words in the lyric text;

[0078] extracting features from the phoneme sequence to obtain the first lyric feature.

[0079] Optionally, the feature processing unit is configured to perform:

[0080] obtaining a song attribute feature of a reference song, the song attribute feature including at least one of a song style feature or a song duration feature;

[0081] fusing the first lyric feature and the song attribute feature to obtain a second fused feature;

[0082] cross-processing the second fused feature and the second noise feature to obtain the second lyric feature;

[0083] wherein a song attribute of the target song is consistent with a song attribute of the reference song.

[0084] Optionally, the feature processing unit is configured to perform:

[0085] The first mapping matrix is used for performing feature mapping on the second fusion feature to obtain a query feature;

[0086] The second mapping matrix is used for performing feature mapping on the second noise feature to obtain a key feature and a value feature;

[0087] The query feature, the key feature and the value feature are fused to obtain the second lyrics feature.

[0088] Optionally, the number of adaptive transformation networks is k, and the k adaptive transformation networks are connected in sequence; the feature processing unit is configured to perform:

[0089] The first lyrics feature is mapped into k first transformation parameters through the k adaptive transformation networks respectively;

[0090] The feature processing unit is further configured to perform:

[0091] The first noise feature is transformed into the first second noise feature based on the first first transformation parameter through the first adaptive transformation network;

[0092] The mth third lyrics feature is transformed into the mth second noise feature based on the mth first transformation parameter through the mth adaptive transformation network; wherein the mth-1 third lyrics feature is the third lyrics feature output by the mth-1 adaptive transformation network, and m is an integer greater than 1 and not greater than k.

[0093] Optionally, the feature processing unit is configured to perform:

[0094] The first lyrics feature and the first second noise feature are cross-processed through the first adaptive transformation network to obtain the first second lyrics feature;

[0095] The first lyrics feature and the mth second noise feature are cross-processed through the mth adaptive transformation network to obtain the mth second lyrics feature.

[0096] Optionally, the k adaptive transformation networks further generate k second transformation parameters; the feature processing unit is configured to perform feature transformation on the first second lyrics feature based on the first second transformation parameter through the first adaptive transformation network to obtain the first third lyrics feature;

[0097] The feature processing unit is further configured to perform feature transformation on the mth second lyric feature based on the mth second transformation parameter through the mth adaptive transformation network to obtain the mth third lyric feature.

[0098] The feature processing unit is further configured to perform decoding on the kth third lyric feature through a decoding network to obtain the target song.

[0099] According to still another aspect of the embodiments of the present disclosure, a device for training a song generation model is provided, and the device comprises:

[0100] A obtaining unit is configured to perform obtaining a sample lyric text and a sample song matched with the sample lyric text.

[0101] A feature extraction unit is configured to perform feature extraction on the sample lyric text through a second song generation model to obtain a first sample lyric feature.

[0102] A feature processing unit is configured to perform mapping of the first sample lyric feature to a first sample transformation parameter through the second song generation model.

[0103] The feature processing unit is further configured to perform feature transformation on the first sample noise feature based on the first sample transformation parameter through the second song generation model to obtain a second sample noise feature, the first sample noise feature being a randomly generated noise feature.

[0104] The feature processing unit is further configured to perform encoding of the sample song through the second song generation model to obtain a target sample lyric feature.

[0105] A model training unit is configured to perform training of the second song generation model based on the first sample lyric feature, the second sample noise feature and the target sample lyric feature to obtain a first song generation model, the first song generation model being used for generating a song matched with any lyric text.

[0106] Optionally, the model training unit is configured to perform:

[0107] cross processing of the first sample lyric feature and the second sample noise feature through the second song generation model to obtain a second sample lyric feature;

[0108] determining a loss parameter based on the second sample lyric feature and the target sample lyric feature;

[0109] training the second song generation model based on the loss parameter to obtain the first song generation model.

[0110] Optionally, the second song generation model comprises the feature extraction network, the adaptive transformation network and an encoding network, the step of generating the first sample lyric feature is performed by the feature extraction network, the steps of generating the first sample transformation parameter, the second sample noise feature and the second sample lyric feature are performed by the adaptive transformation network, and the step of generating the target sample lyric feature is performed by the encoding network.

[0111] The model training unit is configured to perform:

[0112] Based on the loss parameter, adjust the model parameter of the second song generation model to obtain a third song generation model.

[0113] Remove the encoding network in the third song generation model, add a trained decoding network in the third song generation model to obtain the first song generation model.

[0114] According to still another aspect of the embodiments of the present disclosure, an electronic device is provided, which comprises:

[0115] a processor;

[0116] a memory for storing instructions executable by the processor;

[0117] The processor is configured to execute the instructions to implement the song generation method according to the above aspect, or implement the training method of the song generation model according to the above aspect.

[0118] According to still another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, instructions in the computer readable storage medium are executed by a processor of an electronic device, so that the electronic device can execute the song generation method according to the above aspect, or execute the training method of the song generation model according to the above aspect.

[0119] According to still another aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises a computer program, the computer program is executed by a processor to implement the song generation method according to the above aspect, or implement the training method of the song generation model according to the above aspect.

[0120] The scheme provided by the embodiment of the present disclosure introduces a randomly generated noise feature when generating a target song matched with a lyric text, and the lyric text provides semantic information for the generation process of the target song, the random noise feature provides the potential ability to generate songs with different rhythms, and the transformation of the noise feature according to the transformation parameter generated according to the lyric feature can flexibly adjust the rhythm of the song according to the lyric text, without relying on music score knowledge, thereby improving the convenience of generating songs. By dynamically adjusting the transformation parameter, the lyric feature can be more comprehensively captured and utilized, which is beneficial to guarantee the quality of the generated song.

[0121] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0122] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0123] Figure 1 is a schematic diagram of a computer system according to an exemplary embodiment.

[0124] Figure 2 is a flowchart of a song generation method according to an exemplary embodiment.

[0125] Figure 3 is a structural schematic diagram of a first song generation model according to an exemplary embodiment.

[0126] Figure 4 is a structural schematic diagram of another first song generation model according to an exemplary embodiment.

[0127] Figure 5 is a structural schematic diagram of an adaptive transformation network according to an exemplary embodiment.

[0128] Figure 6 is a flowchart of another song generation method according to an exemplary embodiment.

[0129] Figure 7 is a flowchart of a lyric feature generation method according to an exemplary embodiment.

[0130] Figure 8 is a flowchart of still another song generation method according to an exemplary embodiment.

[0131] Figure 9 is a flowchart of yet another song generation method according to an exemplary embodiment.

[0132] Figure 10is a flowchart of a training method of a song generation model according to an exemplary embodiment.

[0133] Figure 11 is a flowchart of another training method of a song generation model according to an exemplary embodiment.

[0134] Figure 12 is a structural schematic diagram of a second song generation model according to an exemplary embodiment.

[0135] Figure 13 is a block diagram of a song generation device according to an exemplary embodiment.

[0136] Figure 14 is a block diagram of a training device of a song generation model according to an exemplary embodiment.

[0137] Figure 15 is a block diagram of a terminal according to an exemplary embodiment.

[0138] Figure 16 is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0139] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0140] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above description of the drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present disclosure are all authorized by the user or fully authorized by all parties. The collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the lyrics text, lyrics features, target songs, etc. involved in the present disclosure are obtained under full authorization.

[0142] For ease of understanding, the concepts related to the embodiments of the present disclosure are explained below.

[0143] TTS (Text-To-Speech, speech synthesis) is one of the core technologies of intelligent voice interaction. By converting the received text sequence into a natural and realistic voice waveform, it is fed back to the user. Speech synthesis technology directly affects the actual use effect of human-computer interaction. Speech synthesis technology involves speech signal processing, pattern recognition, natural language processing, acoustics, linguistics and other disciplines, and is an indispensable key technology in the field of information processing.

[0144] AIGC (Artificial Intelligence Generated Content) refers to various types of content created using artificial intelligence technology, such as text, images, music and videos.

[0145] G2P (Grapheme-to-Phoneme, grapheme-to-phoneme conversion) refers to the output of the pinyin or phoneme annotation of the text, usually using a rule-based dictionary mapping method. In the process of grapheme-to-phoneme conversion, there is a problem of multi-phonetic word recognition, and the grapheme-to-phoneme conversion result has a great influence on the pronunciation of the synthesis system.

[0146] VAE (Variational Autoencoder, Variational Autoencoder) refers to transforming real samples into an ideal data distribution through an encoder network, and the data distribution is passed to a decoder network to obtain generated samples. Further variational processing is done on the autoencoder model, so that the output result of the encoder can correspond to the mean and variance of the target distribution.

[0147] The embodiments of the present disclosure provide a song generation method and a training method of a song generation model, and the execution subject is an electronic device. Illustratively, the electronic device is a terminal, and the terminal is a computer, a mobile phone, a tablet computer or other terminals. Illustratively, the electronic device is a server, and the server is a background server or a cloud server that provides cloud computing and cloud storage services, etc. Among them, the electronic device can generate a target song matched with a lyric text based on the lyric text. Or, the electronic device can train a first song generation model, and the first song generation model is used to generate a song matched with any lyric text.

[0148] Figure 1 is a schematic diagram of a computer system according to an exemplary embodiment, see Figure 1 The computer system includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wireless or wired network.

[0149] The terminal 101 is installed and runs a client, which can be a social application client, an online payment client, an online shopping client, a game client, a medical service client, a video client, and the like. When the terminal 101 runs the client, a user interface of the client is displayed on a screen of the terminal 101. The terminal 101 is a terminal used by a user.

[0150] Optionally, the terminal 101 can refer to one of a plurality of terminals, and those skilled in the art can know that the number of the terminals can be more or less. For example, the terminals can be only one, or 6 or 8 or more. The number of terminals and the type of equipment are not limited in the embodiments of the present disclosure.

[0151] The server 102 includes at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. The server 102 is used to provide background services for the client. Optionally, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing.

[0152] In an embodiment, the terminal 101 sends a song generation request carrying a lyrics text to the server 102, and the server 102 obtains the lyrics text after receiving the song generation request. The server 102 generates a target song matched with the lyrics text based on the lyrics text, and returns the target song to the terminal 101, which can display the target song to the user. For example, the server 102 trains a first song generation model for generating a song, and generates the target song matched with the lyrics text through the first song generation model after receiving the song generation request.

[0153] In another embodiment, the server 102 trains a first song generation model for generating a song, and distributes the first song generation model to the terminal 101, which generates the target song matched with the lyrics text through the first song generation model.

[0154] It should be noted that the above implementation environment is only an example, and the method provided by the embodiments of the present disclosure can also be executed by the terminal 101 or the server 102 alone, or by other computer devices, and the embodiments of the present disclosure do not limit this.

[0155] Figure 2 is a flowchart of a song generation method according to an exemplary embodiment, see Figure 2 The method is executed by an electronic device, and includes the following steps:

[0156] In step 201, the electronic device extracts features of the lyrics text to obtain first lyrics features.

[0157] The electronic device obtains lyrics text, and the purpose of the embodiment of the present disclosure is to intelligently generate a target song matched with the lyrics text. The electronic device extracts features of the obtained lyrics text to obtain first lyrics features. The first lyrics features are used to reflect the semantics and rhythm that the lyrics text wants to express.

[0158] In step 202, the electronic device maps the first lyrics features to first transformation parameters.

[0159] After obtaining the first lyrics features, the electronic device adaptively generates first transformation parameters based on the first lyrics features, and the first transformation parameters are used to transform first noise features randomly generated in step 203 in the process of generating a song.

[0160] In step 203, the electronic device transforms first noise features based on the first transformation parameters to obtain second noise features, and the first noise features are randomly generated noise features.

[0161] The electronic device obtains first noise features, which can be randomly generated noise features, so the first noise features have a certain randomness. The electronic device transforms the first noise features based on the first transformation parameters to obtain second noise features. Since the first transformation parameters are obtained based on the first lyrics features, the second noise features embed features related to the lyrics text, adaptively reduce the randomness of the noise features, and enhance the relevance of the noise features and the lyrics text.

[0162] In step 204, the electronic device generates a target song matched with the lyrics text based on the first lyrics features and the second noise features.

[0163] After determining the first lyrics features and the second noise features, the electronic device generates a target song based on the first lyrics features and the second noise features, and the target song is matched with the lyrics text. That is, the lyrics of the target song are the lyrics text.

[0164] The method provided by the embodiment of the present disclosure introduces randomly generated noise features when intelligently generating a target song matched with lyrics text, and the lyrics text provides semantic information for the generation process of the target song. The random noise features provide the potential ability to generate songs with different rhythms, and the transformation of the noise features by the transformation parameters generated according to the lyrics features can flexibly adjust the rhythm of the song according to the lyrics text, without relying on music score knowledge, thereby improving the convenience of generating songs. By dynamically adjusting the transformation parameters, the lyrics features can be more comprehensively captured and utilized, which is conducive to ensuring the quality of the generated song.

[0165] The above Figure 2 The embodiments described herein are merely brief illustrations of song generation methods. In some embodiments, the electronic device may also perform the process of generating a target song that matches the lyrics text through a first song generation model. This first song generation model includes a feature extraction network, an adaptive transformation network, and a decoding network.

[0166] Figure 3 This is a schematic diagram illustrating the structure of a first song generation model according to an exemplary embodiment, such as... Figure 3 As shown, the first song generation model includes a feature extraction network, an adaptive transformation network, and a decoding network. The feature extraction network is connected to the adaptive transformation network, and the output of the feature extraction network serves as the input to the adaptive transformation network. The adaptive transformation network is connected to the decoding network, and the output of the adaptive transformation network serves as the input to the decoding network.

[0167] Among them, the feature extraction network is used to extract features from the lyrics text, the adaptive transformation network is used to transform and fuse the lyrics features and noise features, and the decoding network is used to decode the lyrics features to generate the song.

[0168] In some embodiments, such as Figure 4 As shown, the first song generation model includes k sequentially connected adaptive transformation networks, and the feature extraction network includes a first feature extraction network, a second feature extraction network, and a third feature extraction network. The first feature extraction network, the second feature extraction network, and the third feature extraction network are respectively connected to the k adaptive transformation networks.

[0169] The first feature extraction network is used to extract features from the lyrics text to obtain the first lyrics features. For example, such as... Figure 4 As shown, the first feature extraction network comprises two parts: a first part consisting of G2P and an embedding layer, and a second part consisting of a Transformer (a type of neural network layer). The first part converts the lyrics text into a phoneme sequence, and the second part extracts features from the phoneme sequence to obtain the first lyric features. The second feature extraction network extracts the song style features and song duration features of the reference song. In this embodiment, the first lyric features, song style features, and song duration features output by the first, second, and third feature extraction networks are fused to obtain a second fused feature. This second fused feature is then input into k adaptive transformation networks for processing.

[0170] The input of the first adaptive transformation network is the second fusion feature and the first noise feature, and the output is a first third lyric feature. The input of the second adaptive transformation network is the second fusion feature and the first third lyric feature, and the output is a second third lyric feature. The input of the third adaptive transformation network is the second fusion feature and the second third lyric feature, and the output is a third third lyric feature. In this way, the input of the kth adaptive transformation network (i.e., the last adaptive transformation network) is the second fusion feature and the (k-1)th third lyric feature, and the output is a kth third lyric feature.

[0171] In some embodiments, as shown in Figure 5 , the adaptive transformation network includes a normalization layer, a linear transformation layer, an attention layer, a scale transformation layer, and a multi-layer perception. The input of the multi-layer perception is a first fusion feature obtained by fusing a time step feature and a first lyric feature, and the output is a first transformation parameter and a second transformation parameter. The input of the normalization layer is a first noise feature or an output of a previous adaptive transformation network. The input of the linear transformation layer is an output of the normalization layer and the first transformation parameter, and the output is a second noise feature. The input of the attention layer is the first lyric feature and an output of the linear transformation layer, and the output is a second lyric feature. The input of the scale transformation layer is the second lyric feature and the second transformation parameter, and the output is a third lyric feature.

[0172] The input of the decoding network is the kth third lyric feature, and the output is a target song.

[0173] The specific process of the electronic device generating a song through the first song generation model is described below Figure 6 in the embodiments. Figure 6 is a flowchart of another song generation method according to an example embodiment, which is performed by an electronic device, and is described below Figure 6 , the method includes the following steps:

[0174] In step 601, the electronic device extracts features of the lyric text through a feature extraction network in the first song generation model to obtain a first lyric feature.

[0175] The electronic device obtains the lyric text, inputs the lyric text into the feature extraction network in the first song generation model, processes the lyric text by the feature extraction network, and outputs the first lyric feature.

[0176] In some embodiments, the electronic device performs the following steps through the feature extraction network: converting the lyric text into a phoneme sequence, the phoneme sequence including phonemes of a plurality of words in the lyric text; and extracting features of the phoneme sequence to obtain the first lyric feature.

[0177] Wherein, the phoneme is the smallest unit or the smallest speech fragment constituting a syllable, or the phoneme is the smallest speech unit divided according to the natural attributes of speech, each phoneme has its own unique sound characteristics, for example, vowel phonemes a, e, i, o, u, and consonant phonemes such as b, p, m, f, etc., in addition, tone also belongs to phoneme.

[0178] As shown in the example, Figure 4 The feature extraction network includes a first part (G2P and embedder) and a second part (Transformer). The first part is used to convert the lyrics text into a phoneme sequence, and the second part is used to extract features from the phoneme sequence to obtain the first lyrics feature. Wherein, the G2P can be a G2P based on conditional random field or a BERT-based G2P, etc., which is not limited in the embodiments of the present disclosure.

[0179] Figure 7 According to an example embodiment, a lyrics feature generation method is shown in a flow chart, as shown in Figure 7 After obtaining the lyrics text, the lyrics text is preprocessed and normalized, the preprocessed and normalized lyrics text is segmented and tokenized, the word pronunciation conversion is performed based on the segmentation and tokenization, the phoneme sequence of the lyrics text is output, and the content encoding is performed on the phoneme sequence to obtain the first lyrics feature.

[0180] In the embodiments of the present disclosure, the lyrics text is converted into a phoneme sequence, the pronunciation features of the lyrics can be extracted from the phonetic level, the prosody, syllable and other information are captured, the speech feature expression ability of the lyrics is enhanced, and the generated song can better reflect the speech characteristics of the lyrics. And based on the phoneme sequence, the first lyrics feature is extracted, the pronunciation rules of the lyrics can be more closely combined with the music melody, and the generated target song is more in line with the prosody and rhythm requirements of the lyrics in rhythm and melody.

[0181] In step 602, the electronic device maps the first lyrics feature to the first transformation parameter and the second transformation parameter through the adaptive transformation network in the first song generation model.

[0182] After obtaining the first lyrics feature, the computer device maps the first lyrics feature to the first transformation parameter and the second transformation parameter through the adaptive transformation network in the first song generation model. Wherein, the first transformation parameter is used to transform the noise feature randomly generated in the following step 603 in the process of generating the song, and the second transformation parameter is used to transform the second lyrics feature generated in the following step 604 in the process of generating the song.

[0183] In some embodiments, the electronic device performs the following steps by the adaptive transformation network: determining a current time step feature, the time step feature representing a position of a current audio frame to be generated in a target song, the target song including a plurality of audio frames; fusing the first lyric feature and the time step feature to obtain a first fused feature; and mapping the first fused feature into the first transformation parameter and the second transformation parameter.

[0184] In the embodiments of the present disclosure, the target song generated by the song generation model includes a plurality of continuous audio frames, which are generated in a time sequence, and in the process of generating a current audio frame, the electronic device obtains a time step feature representing a position of the current audio frame to be generated in the target song by the song generation model, and generates the first transformation parameter and the second transformation parameter based on the first lyric feature and the time step feature.

[0185] For example, the electronic device splices the first lyric feature and the time step feature by the adaptive transformation network to obtain the first fused feature.

[0186] For example, as shown in Figure 4 The adaptive transformation network includes a multi-layer perceptron, and the electronic device inputs the first fused feature into the multi-layer perceptron, and the multi-layer perceptron outputs the first transformation parameter and the second transformation parameter.

[0187] It should be noted that in the embodiments of the present disclosure, not only the first noise feature is transformed by the first transformation parameter, but also the second lyric feature is transformed by the second transformation parameter, so the first fused feature needs to be mapped into the first transformation parameter and the second transformation parameter. In another embodiment, only the first noise feature can be transformed by a transformation parameter, and the second lyric feature does not need to be transformed, so only the first fused feature needs to be mapped into the first transformation parameter.

[0188] In the embodiments of the present disclosure, by introducing the time step feature and fusing it with the first lyric feature, the time sequence relationship between the lyric text and each audio frame in the target song can be effectively established, so that the position of each audio frame can be accurately modeled, avoiding the situation that rhythm dislocation or audio frame and lyric content mismatch may occur in the generation process, thereby improving the accuracy and naturalness of the target song.

[0189] In step 603, the electronic device performs feature transformation on the first noise feature based on the first transformation parameter by the adaptive transformation network in the first song generation model to obtain a second noise feature, and the first noise feature is a randomly generated noise feature.

[0190] The electronic device obtains a first noise feature, which can be a randomly generated noise feature, and thus has a certain randomness. The electronic device performs feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature. Since the first transformation parameter is obtained based on the first lyric feature, the second noise feature has features related to the lyric text embedded therein, and the randomness of the noise feature is adaptively reduced, and the relevance of the noise feature to the lyric text is enhanced.

[0191] In some embodiments, as shown in Figure 5 The adaptive transformation network includes a normalization layer and a linear transformation layer. The electronic device inputs the first noise feature into the normalization layer, and the normalization layer outputs a normalized first noise feature. The electronic device inputs the normalized first noise feature and the first transformation parameter into the linear transformation layer, and the linear transformation layer outputs the second noise feature. For example, the normalization layer can be a Layer Norm layer, and the linear transformation layer can be a Scale&Shift layer.

[0192] In step 604, the electronic device performs cross processing on the first lyric feature and the second noise feature through the adaptive transformation network in the first song generation model to obtain a second lyric feature.

[0193] After obtaining the first lyric feature and the second noise feature, the electronic device performs cross processing on the first lyric feature and the second noise feature to obtain a second lyric feature. Since the second lyric feature fuses the first lyric feature and the second noise feature, the second lyric feature not only reflects the features of the lyric text, but also has a certain randomness, providing potential ability to generate songs with different rhythms. Therefore, combining the two text guiding mechanisms of cross processing and dynamically adjusting the transformation parameter can more comprehensively capture and utilize the lyric features, which is conducive to ensuring the quality of the generated songs.

[0194] In some embodiments, the electronic device obtains a song attribute feature of a reference song, and the song attribute feature includes at least one of a song style feature or a song length feature. The electronic device fuses the first lyric feature and the song attribute feature. The electronic device performs the following steps through the adaptive transformation network: cross processing the second fused feature and the second noise feature to obtain a second lyric feature. The song attribute of the target song is consistent with the song attribute of the reference song.

[0195] The reference song can be any type of song, and the song attribute of the reference song is used as a reference to participate in the generation process of the target song, which can guide the song generation model to generate a target song consistent with the song attribute of the reference song.

[0196] For example, as shown in Figure 4As shown, the feature extraction network of the song generation model includes a first feature extraction network, a second feature extraction network, and a third feature extraction network. The electronic device inputs the reference song to the second feature extraction network, and the second feature extraction network outputs a song style feature. The electronic device inputs the reference song length of the reference song to the third feature extraction network, and the third feature extraction network outputs a song length feature.

[0197] For example, the electronic device concatenates the first lyric feature, the song style feature, and the song length feature to obtain a second fusion feature.

[0198] In the embodiments of the present disclosure, by introducing the song attribute feature of the reference song and fusing it with the first lyric feature, the generated target song can inherit the specific attributes of the reference song, and precise control of the style of the target song can be achieved. On the basis of introducing the second noise feature, the embodiments ensure that the target song maintains the consistency of the attributes of the reference song while having a certain uniqueness.

[0199] Optionally, the electronic device performs the following steps through the adaptive transformation network: using a first mapping matrix to perform feature mapping on the second fusion feature to obtain a query feature; using a second mapping matrix to perform feature mapping on the second noise feature to obtain a key feature and a value feature; and fusing the query feature, the key feature, and the value feature to obtain a second lyric feature. The key feature, the value feature, and the query feature in the embodiments of the present disclosure belong to different feature spaces respectively.

[0200] The electronic device obtains a first mapping matrix, which is used for spatial transformation of the second fusion feature. The electronic device multiplies the second fusion feature by the first mapping matrix to obtain a first spatial matrix, and splits out the query feature in the first spatial matrix. For example, the first mapping matrix is a 3-dimensional mapping matrix, the first spatial matrix obtained by multiplying the second fusion feature by the first mapping matrix is a 3-dimensional first spatial matrix, and the electronic device takes the feature in a preset dimension of the first spatial matrix as the query feature.

[0201] The electronic device obtains a second mapping matrix, which is used for spatial transformation of the second noise feature. The electronic device multiplies the second noise feature by the second mapping matrix to obtain a second spatial matrix, and splits out the key feature and the value feature in the second spatial matrix. For example, the second mapping matrix is a 3-dimensional mapping matrix, the second spatial matrix obtained by multiplying the second noise feature by the second mapping matrix is a 3-dimensional second spatial matrix, and the electronic device takes the feature in a preset dimension of the second spatial matrix as the key feature and the value feature.

[0202] Optionally, the first mapping matrix and the second mapping matrix are model parameters of an adaptive transformation network in the song generation model, and the first mapping matrix and the second mapping matrix are obtained by training the song generation model.

[0203] Optionally, the electronic device normalizes a product of the query feature, the transpose of the key feature, and the scaling factor to obtain a normalized feature, and multiplies the normalized feature and the value feature to obtain the second lyric feature.

[0204] For example, as shown in FIG. 6, the adaptive transformation network in the song generation model includes an attention layer, and model parameters of the attention layer include a first mapping matrix and a second mapping matrix. Figure 4 The electronic device inputs the second fused feature and the second noise feature into the attention layer, and the attention layer outputs the second lyric feature. For example, the attention layer can be a cross-attention layer, a forward-attention, a GMM-attention, a location-attention, etc., which are not limited in the embodiments of the present disclosure.

[0205] In the embodiments of the present disclosure, by mapping the second fused feature into the query feature and mapping the second noise feature into the key feature and the value feature, the attention mechanism is introduced to ensure that important information is focused during feature fusion, avoid interference of redundant or irrelevant features, and further strengthen the comprehensive expression ability of multi-dimensional information such as rhythm and semantics in the lyric feature, so that the generated second lyric feature is more in line with the requirements of the target song, which is conducive to improving the accuracy and naturalness of song generation.

[0206] In addition, by introducing the cross-attention mechanism, although the expression ability of the song generation model is retained, the computational burden of the song generation model is significantly reduced. By removing the cross-attention mask, it is conducive to speeding up the iteration speed of the song generation model and improving the overall training efficiency of the song generation model.

[0207] In step 605, the electronic device performs feature transformation on the second lyric feature based on a second transformation parameter through an adaptive transformation network in the first song generation model to obtain a third lyric feature.

[0208] The electronic device performs feature transformation on the second lyric feature based on the second transformation parameter to obtain a third lyric feature. Since the second transformation parameter is obtained based on the first lyric feature, the third lyric feature embeds features related to the lyric text, further enhancing the relevance of the third lyric feature to the lyric text.

[0209] In some embodiments, as shown in FIG. 7, the adaptive transformation network in the song generation model includes an attention layer, and model parameters of the attention layer include a first mapping matrix and a second mapping matrix. Figure 4As shown, the adaptive transformation network includes a scale transformation layer, and the electronic device inputs the second lyric feature and the second transformation parameter to the scale transformation layer, and the scale transformation layer outputs a third lyric feature. Exemplarily, the scale transformation layer can be a Scale layer.

[0210] In step 606, the electronic device decodes the third lyric feature through a decoding network in the first song generation model to obtain a target song matched with the lyric text.

[0211] The electronic device inputs the third lyric feature to the decoding network in the first song generation model, decodes the third lyric feature through the decoding network, and obtains a target song matched with the lyric text. That is, the lyrics of the target song are the lyric text.

[0212] Exemplarily, the decoding network can be a decoding network in a VAE model architecture.

[0213] In the embodiments of the present disclosure, the transformation parameter generated according to the first lyric feature is used to transform the second lyric feature obtained after the cross processing, which can effectively extract the potential features of the lyrics, so that the third lyric feature after the transformation can more accurately represent the information such as the emotion and rhythm of the lyric text, providing a high-quality input for subsequent decoding, which is conducive to improving the matching degree of the target song and the lyric text obtained by decoding.

[0214] Figure 8 is a flowchart of another song generation method according to an exemplary embodiment, as shown in Figure 8 As shown, the song generation process can be divided into two parts of lyric text analysis in the front end and song synthesis in the back end. The front end part provides a lyric analysis function for the input lyric text, and the back end part synthesizes a song based on the analysis result of the lyric text. Among them, the song generation method of the embodiments of the present disclosure can be applied to any scene where a song needs to be generated, such as song creation, film music, game background music, etc.

[0215] The method provided by the embodiments of the present disclosure introduces a randomly generated noise feature when generating a target song matched with the lyric text, provides semantic information for the generation process of the target song, provides potential ability to generate songs with different rhythms for the random noise feature, and transforms the noise feature according to the transformation parameter generated according to the lyric feature, which can flexibly adjust the rhythm of the song according to the lyric text, without relying on music knowledge, improving the convenience of generating songs. Moreover, the present disclosure combines the two text guiding mechanisms of feature cross processing and dynamic adjustment of transformation parameters, which can more comprehensively capture and utilize lyric features, and is conducive to ensuring the quality of the generated song.

[0216] And, the process of generating the target song based on the lyrics text is performed by the first song generation model, which is beneficial to improve the convenience and efficiency of the song generation process.

[0217] It should be noted that the embodiments of the present disclosure only take the electronic device performing the song generation method of steps 601-606 by the first song generation model as an example for description. In another embodiment, the electronic device can also directly perform the song generation method of steps 601-606 without relying on the first song generation model.

[0218] Based on the above Figure 6 embodiments, as Figure 4 indicated, the number of adaptive transformation networks in the first song generation model is k, and the k adaptive transformation networks are connected in sequence. Then, the detailed process of generating the target song is described in the following Figure 9 embodiments. Figure 9 is a flowchart of another song generation method according to an example embodiment, which is performed by an electronic device, as described in Figure 9 , and the method includes the following steps:

[0219] In step 901, the electronic device extracts features of the lyrics text by a feature extraction network in the first song generation model to obtain first lyrics features.

[0220] The process of extracting the first lyrics features in this step 901 is the same as the process of extracting the first lyrics features in the above step 601, and will not be repeated here.

[0221] In step 902, the electronic device maps the first lyrics features to a first first transformation parameter and a first second transformation parameter by a first adaptive transformation network.

[0222] As Figure 5 indicated, the first adaptive transformation network includes a multi-layer perceptron, and the electronic device inputs the first lyrics features and the time step features into the multi-layer perceptron in the first adaptive transformation network, and the multi-layer perceptron in the first adaptive transformation network outputs the first first transformation parameter and the first second transformation parameter.

[0223] The process of generating the first first transformation parameter and the first second transformation parameter in this step 902 is the same as the process of generating the first transformation parameter and the second transformation parameter in the above step 602, and will not be repeated here.

[0224] In step 903, the electronic device performs feature transformation on the first noise features based on the first first transformation parameter by the first adaptive transformation network to obtain a first second noise feature.

[0225] As Figure 5As shown, the first adaptive transformation network includes a normalization layer and a linear transformation layer. The electronic device inputs the first noise feature into the normalization layer to obtain a normalized first noise feature. The electronic device inputs the normalized first noise feature and the first first transformation parameter output by the multi-layer perception in the first adaptive transformation network into the linear transformation layer, and the linear transformation layer outputs the first second noise feature.

[0226] The process of generating the first second noise feature in step 903 is the same as the process of generating the second noise feature in step 603, which will not be repeated here.

[0227] In step 904, the electronic device cross-processes the first lyric feature and the first second noise feature through the first adaptive transformation network to obtain a first second lyric feature.

[0228] As shown, the first adaptive transformation network includes an attention layer. The electronic device inputs the first lyric feature and the first second noise feature output by the linear transformation layer in the first adaptive transformation network into the attention layer, and the attention layer outputs the first second lyric feature. Figure 5 The process of generating the first second lyric feature in step 904 is the same as the process of generating the second lyric feature in step 604, which will not be repeated here.

[0229] In step 905, the electronic device performs feature transformation on the first second lyric feature based on the first second transformation parameter through the first adaptive transformation network to obtain a first third lyric feature.

[0230] As shown, the first adaptive transformation network includes a scale transformation layer. The electronic device inputs the first second lyric feature output by the attention layer in the first adaptive transformation network and the first second transformation parameter output by the multi-layer perception in the first adaptive transformation network into the scale transformation layer, and the scale transformation layer outputs the first third lyric feature.

[0231] Figure 5 The first third lyric feature is used to be input into a second adaptive transformation network, as shown. The first third lyric feature is input into a normalization layer in the second adaptive transformation network.

[0232] Figure 5 The process of generating the first third lyric feature in step 905 is the same as the process of generating the third lyric feature in step 605, which will not be repeated here.

[0233] The process of generating the first third lyric feature in step 905 is the same as the process of generating the third lyric feature in step 605, which will not be repeated here.

[0234] ​In step 906, the electronic device generates the mth first transformation parameter and the mth second transformation parameter based on the first lyric feature through the mth adaptive transformation network, m is an integer greater than 1 and not greater than k.

[0235] For example, as shown in the case where m is equal to 2, Figure 5 the second adaptive transformation network includes a multi-layer perception, the electronic device inputs the first lyric feature and the time step feature into the multi-layer perception in the second adaptive transformation network, and the multi-layer perception in the second adaptive transformation network outputs the second first transformation parameter and the second second transformation parameter.

[0236] For example, as shown in the case where m is equal to k, Figure 5 the kth adaptive transformation network includes a multi-layer perception, the electronic device inputs the first lyric feature and the time step feature into the multi-layer perception in the kth adaptive transformation network, and the multi-layer perception in the kth adaptive transformation network outputs the kth first transformation parameter and the kth second transformation parameter.

[0237] In step 907, the electronic device performs feature transformation on the m-1th third lyric feature based on the mth first transformation parameter through the mth adaptive transformation network to obtain the mth second noise feature, wherein the m-1th third lyric feature is the third lyric feature output by the m-1th adaptive transformation network.

[0238] For example, as shown in the case where m is equal to 2, Figure 5 the second adaptive transformation network includes a normalization layer and a linear transformation layer, the electronic device inputs the first third lyric feature into the normalization layer to obtain the normalized first third lyric feature, inputs the normalized first third lyric feature and the second first transformation parameter output by the multi-layer perception in the second adaptive transformation network into the linear transformation layer, and the linear transformation layer performs feature transformation on the normalized first third lyric feature based on the second first transformation parameter to output the second second noise feature.

[0239] For example, as shown in the case where m is equal to k, Figure 5 the kth adaptive transformation network includes a normalization layer and a linear transformation layer, the electronic device inputs the k-1th third lyric feature into the normalization layer to obtain the normalized k-1th third lyric feature, inputs the normalized k-1th third lyric feature and the kth first transformation parameter output by the multi-layer perception in the kth adaptive transformation network into the linear transformation layer, and the linear transformation layer performs feature transformation on the normalized k-1th third lyric feature based on the kth first transformation parameter to output the kth second noise feature.

[0240] In step 908, the electronic device cross-processes the first lyric feature and the mth second noise feature through the mth adaptive transformation network, and fuses to obtain an mth second lyric feature.

[0241] For example, as shown in the following Figure 5 , the second adaptive transformation network includes an attention layer, and the electronic device inputs the first lyric feature and the second second noise feature output by the linear transformation layer in the second adaptive transformation network into the attention layer, and the attention layer outputs the second second lyric feature.

[0242] For example, as shown in the following Figure 5 , the kth adaptive transformation network includes an attention layer, and the electronic device inputs the first lyric feature and the kth second noise feature output by the linear transformation layer in the kth adaptive transformation network into the attention layer, and the attention layer outputs the kth second lyric feature.

[0243] In step 909, the electronic device performs feature transformation on the mth second lyric feature based on the mth second transformation parameter through the mth adaptive transformation network to obtain an mth third lyric feature.

[0244] For example, as shown in the following Figure 5 , the second adaptive transformation network includes a scale transformation layer, and the electronic device inputs the second second lyric feature output by the attention layer in the second adaptive transformation network and the second second transformation parameter output by the multi-layer perception in the second adaptive transformation network into the scale transformation layer, and the scale transformation layer performs feature transformation on the second second lyric feature based on the second second transformation parameter to output the second third lyric feature.

[0245] For example, as shown in the following Figure 5 , the kth adaptive transformation network includes a scale transformation layer, and the electronic device inputs the kth second lyric feature output by the attention layer in the kth adaptive transformation network and the kth second transformation parameter output by the multi-layer perception in the kth adaptive transformation network into the scale transformation layer, and the scale transformation layer performs feature transformation on the kth second lyric feature based on the kth second transformation parameter to output the kth third lyric feature.

[0246] In step 910, the electronic device decodes the kth third lyric feature through the decoding network to obtain the target song.

[0247] The electronic device obtains a kth third lyric feature, which is a third lyric feature output by the kth adaptive network, i.e., a third lyric feature output by the last adaptive network. The electronic device inputs the kth third lyric feature into the decoding network to obtain a target song output by the decoding network, which matches the lyric text.

[0248] The method provided by the embodiments of the present disclosure introduces randomly generated noise features when generating a target song matching the lyric text, provides semantic information for the generation process of the target song, provides potential ability to generate songs with different rhythms, and can flexibly adjust the rhythm of the song according to the lyric text by transforming the noise features according to the transformation parameters generated according to the lyric features, without relying on music score knowledge, thereby improving the convenience of generating songs. Moreover, the present disclosure combines the two text-guided mechanisms of feature cross processing and dynamic adjustment of transformation parameters, which can more comprehensively capture and utilize lyric features, and is conducive to ensuring the quality of the generated song.

[0249] Moreover, through the layer-by-layer processing of the k adaptive transformation networks, the first lyric features and the noise features are transformed at multiple levels, the feature performance is gradually refined, irrelevant information in the noise features is gradually eliminated, and features useful for generating the target song are retained and strengthened.

[0250] Moreover, through the cross processing of the first lyric features and noise features of different levels by the multi-layer adaptive transformation network, the feature fusion effect is optimized layer by layer, which helps to retain the semantic and rhythm features of the lyrics, and each layer of cross processing operation can capture deeper relationships between the lyric features and the noise features, thereby enriching the expression ability of the second lyric features.

[0251] Moreover, through the multi-layer adaptive transformation network, the second lyric features obtained after the cross processing are transformed according to the second transformation parameters generated by each layer, to obtain third lyric features after transformation, the feature transformation effect is optimized layer by layer, the potential features of the lyrics can be effectively extracted, the third lyric features after transformation can more accurately represent the information such as emotion and rhythm of the lyric text, and the last third lyric feature output by the last adaptive transformation network is used as a high-quality input of the decoding network, which is conducive to improving the matching degree of the target song obtained by decoding and the lyric text.

[0252] Moreover, since the number of adaptive transformation networks can be flexibly adjusted, the complexity of the generation process can be controlled, thereby significantly improving the performance of the model, the multi-level network structure supports flexible adjustment for different application scenarios, adapts to more song styles and specific task requirements, and widens the application range of the model.

[0253] The training process of the first song generation model in the above embodiments can be referred to in the followingFigure 10 Examples of implementations. Figure 10 This is a flowchart illustrating a training method for a song generation model according to an exemplary embodiment, the method being performed by an electronic device, see [link to flowchart]. Figure 10 The method includes the following steps:

[0254] In step 1001, the electronic device acquires sample lyrics text and sample songs that match the sample lyrics text.

[0255] In this context, "matching the sample song with the sample lyrics text" means that the lyrics in the sample song are the same as the sample lyrics text.

[0256] In step 1002, the electronic device extracts features from the sample lyrics text using the second song generation model to obtain the first sample lyrics features.

[0257] In step 1003, the electronic device maps the lyric features of the first sample to the transformation parameters of the first sample through the second song generation model.

[0258] The first sample transformation parameter is used to transform the first sample noise features randomly generated in step 1004. In some embodiments, the electronic device generates the first sample transformation parameter and the second sample transformation parameter based on the first sample lyrics features using a second song generation model.

[0259] In step 1004, the electronic device uses the second song generation model to perform feature transformation on the first sample noise features based on the first sample transformation parameters to obtain the second sample noise features. The first sample noise features are randomly generated noise features.

[0260] The process of generating the first sample lyrics feature and the second sample noise feature in steps 1002-1005 is the same as the process of generating the first lyrics feature and the second noise feature in steps 601-604, and will not be repeated here.

[0261] In step 1005, the electronic device encodes the sample song using the second song generation model to obtain the target sample lyrics features.

[0262] In this embodiment of the disclosure, the electronic device encodes the sample song using a second song generation model to obtain target sample lyrics features. These target sample lyrics features are the true features of the sample song.

[0263] In step 1006, the electronic device trains the second song generation model based on the first sample lyrics features, the second sample noise features, and the target sample lyrics features to obtain the first song generation model. The first song generation model is used to generate a song that matches any lyrics text.

[0264] The first sample lyrics feature and the second sample noise feature are used to generate a song matched with the sample lyrics text, and the target sample lyrics feature is a real feature of the song matched with the sample lyrics text, in other words, the target sample lyrics feature can be used to accurately generate a song matched with the sample lyrics text. Therefore, training the second song generation model based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature can make the first song generation model generated more accurate first sample lyrics features and second sample noise features, so as to generate a song matched with the sample lyrics text more accurately using the more accurate first sample lyrics features and second sample noise features, thereby ensuring the accuracy of the first song generation model.

[0265] In the embodiments of the present disclosure, when generating a target song matched with the lyrics text intelligently, a randomly generated noise feature is introduced, the lyrics text provides semantic information for the generation process of the target song, the random noise feature provides potential ability to generate songs with different rhythms, and the transformation of the noise feature according to the transformation parameter generated according to the lyrics feature can realize flexible adjustment of the rhythm of the song according to the lyrics text, without relying on music score knowledge, thereby improving the generation ability of the song generation model.

[0266] On the basis of the above Figure 10 embodiments, as shown in Figure 11 , step 1006 includes steps 1016-1036.

[0267] In step 1016, the electronic device cross-processes the first sample lyrics feature and the second sample noise feature by the second song generation model to obtain a second sample lyrics feature by fusion.

[0268] The step 1016 is the same as the above step 604, and will not be repeated here.

[0269] In step 1026, the electronic device determines a loss parameter based on the second sample lyrics feature and the target sample lyrics feature.

[0270] The target sample lyrics feature is a real feature of the sample song, and the second sample lyrics feature is obtained by the second song generation model. The smaller the difference between the second sample lyrics feature and the target sample lyrics feature generated by the song generation model, the smaller the difference between the decoded song based on the second sample lyrics feature and the sample song, that is, the more accurate the song generation model. Therefore, based on the second sample lyrics feature and the target sample lyrics feature, the second song generation model is trained, and the training target is to reduce the difference between the second sample lyrics feature and the target sample lyrics feature, thereby improving the accuracy of the model. In the embodiment of the present disclosure, the electronic device determines the loss parameter for training the model based on the second sample lyrics feature and the target sample lyrics feature. For example, the electronic device determines the loss parameter based on the difference between the second sample lyrics feature and the target sample lyrics feature, and the loss parameter is positively correlated with the difference, and then the training target is to reduce the loss parameter.

[0271] In some embodiments, as described in step 1003 above, the second song generation model also generates a second sample transformation parameter, and then step 1026 includes: the electronic device performs feature transformation on the second sample lyrics feature based on the second sample transformation parameter by the second song generation model to obtain a third sample lyrics feature; and determines the loss parameter based on the third sample lyrics feature and the target sample lyrics feature.

[0272] Wherein, the process of generating the third sample lyrics feature is the same as the process of generating the third lyrics feature in step 605 described above, and will not be repeated here. The third sample lyrics feature is used to decode a song that matches the sample lyrics text, and therefore the electronic device trains the second song generation model based on the third sample lyrics feature and the target sample lyrics feature, and the training target is to reduce the difference between the third sample lyrics feature and the target sample lyrics feature, thereby improving the accuracy of the model. In the embodiment of the present disclosure, the electronic device determines the loss parameter for training the model based on the third sample lyrics feature and the target sample lyrics feature. For example, the electronic device determines the loss parameter based on the difference between the third sample lyrics feature and the target sample lyrics feature, and the loss parameter is positively correlated with the difference, and then the training target is to reduce the loss parameter.

[0273] In the embodiment of the present disclosure, the sample transformation parameter generated according to the first sample lyrics feature is used to transform the second sample lyrics feature obtained after cross processing, which can effectively extract the potential features of the sample lyrics, so that the third sample lyrics feature after transformation can more accurately represent the information such as emotion and rhythm of the sample lyrics text, which is conducive to improving the accuracy of the third sample lyrics feature and speeding up the model training speed.

[0274] In step 1036, the electronic device trains the second song generation model based on the loss parameter to obtain the first song generation model.

[0275] The electronic device trains the second song generation model based on the loss parameter after determining the loss parameter, a training target being to reduce the loss parameter, so as to improve the accuracy of the model, and obtain the trained first song generation model.

[0276] In some embodiments, the second song generation model comprises a feature extraction network, an adaptive transformation network and an encoding network, the step of generating the first sample lyric feature being performed by the feature extraction network; the steps of generating the first sample transformation parameter, the second sample noise feature and the second sample lyric feature being performed by the adaptive transformation network; and the step of generating the target sample lyric feature being performed by the encoding network.

[0277] The step 1007 comprises: adjusting model parameters of the second song generation model based on the loss parameter to obtain a third song generation model; removing the encoding network in the third song generation model and adding the trained decoding network to the third song generation model to obtain the first song generation model.

[0278] For example, the encoding network in the first song generation model is a trained encoding network, and when the model parameters of the second song generation model are adjusted, the model parameters of the encoding network do not need to be adjusted. For example, the trained encoding network and the trained decoding network belong to a VAE model architecture, and the encoding network and the decoding network in the VAE model architecture are synchronously trained to obtain the trained encoding network and the trained decoding network.

[0279] Figure 12 FIG. 2 is a structural schematic diagram of a second song generation model according to an example embodiment, as shown in FIG. 2, the second song generation model comprises a feature extraction network, k adaptive transformation networks and an encoding network. Figure 12 The second song generation model is trained to obtain a third song generation model, the encoding network in the third song generation model is removed, and a trained decoding network is added to obtain a first song generation model as shown in FIG. 3. Figure 4

[0280] In the embodiments of the present disclosure, by determining the loss parameter based on the difference between the second sample lyric feature and the target sample lyric feature, the gap between the model output and the target can be quantified, which ensures that the model can gradually approach the feature distribution of the target sample in the generation process, and helps to improve the feature learning ability of the model.

[0281] ​And, the encoding network is used to learn the feature distribution of the song in the model training stage, and the decoding network is used to generate the song in the model using stage, the functions of the encoding network and the decoding network are separated, the model is more flexible and easy to expand, the network structure can be adjusted according to different requirements, different types of generation tasks are supported, and great flexibility is provided for expanding new generation strategies or processing different data.

[0282] And, the feature cross processing and the dynamic adjustment of the transformation parameter are combined, the two text guiding mechanisms are combined, the lyrics features can be more comprehensively captured and utilized, and the accuracy of the song generation model can be guaranteed. And, through the diversified text guiding mechanism, the song generation model can more comprehensively utilize the feature information, so that the training number of rounds required by the song generation model when reaching the optimal solution is greatly reduced, time and resources are effectively saved, and the training efficiency is improved.

[0283] Figure 13 is a block diagram of a song generation device according to an example embodiment. Referring to Figure 13 The device includes a feature extraction unit 1301 and a feature processing unit 1302.

[0284] The feature extraction unit 1301 is configured to perform feature extraction on the lyrics text to obtain first lyrics features;

[0285] The feature processing unit 1302 is configured to perform mapping of the first lyrics features to first transformation parameters;

[0286] The feature processing unit 1302 is further configured to perform feature transformation on first noise features based on the first transformation parameters to obtain second noise features, the first noise features being randomly generated noise features;

[0287] The feature processing unit 1302 is further configured to perform generation of a target song matching the lyrics text based on the first lyrics features and the second noise features.

[0288] The device provided by the embodiment of the present disclosure introduces randomly generated noise features when generating a target song matching the lyrics text, the lyrics text provides semantic information for the generation process of the target song, the random noise features provide potential ability to generate songs with different rhythms, and the transformation of the noise features based on the transformation parameters generated according to the lyrics features can flexibly adjust the rhythm of the song according to the lyrics text, without relying on music score knowledge, and the convenience of generating the song is improved. The dynamic adjustment of the transformation parameter can more comprehensively capture and utilize the lyrics features, which is beneficial to guarantee the quality of the generated song.

[0289] In a possible implementation manner, referring to Figure 13, the feature processing unit 1302 is configured to perform:

[0290] cross processing the first lyric feature and the second noise feature to obtain a second lyric feature;

[0291] generating a target song matching the lyric text based on the second lyric feature.

[0292] In a possible implementation, referring to Figure 13 , the feature processing unit 1302 is configured to perform mapping the first lyric feature to the first transformation parameter and the second transformation parameter;

[0293] The feature processing unit 1302 is further configured to perform feature transformation on the second lyric feature based on the second transformation parameter to obtain a third lyric feature;

[0294] The song generation unit 1303 is configured to perform decoding the third lyric feature to obtain a target song matching the lyric text.

[0295] In a possible implementation, referring to Figure 13 , the feature processing unit 1302 is further configured to perform:

[0296] determining a current time step feature, the time step feature representing a position of a current audio frame to be generated in the target song, the target song including a plurality of audio frames;

[0297] fusing the first lyric feature and the time step feature to obtain a first fused feature;

[0298] mapping the first fused feature to the first transformation parameter, or mapping the first fused feature to the first transformation parameter and the second transformation parameter.

[0299] In a possible implementation, referring to Figure 13 , the feature extraction unit 1301 is configured to perform:

[0300] converting the lyric text into a phoneme sequence, the phoneme sequence including phonemes of a plurality of words in the lyric text;

[0301] extracting features from the phoneme sequence to obtain the first lyric feature.

[0302] In a possible implementation, referring to Figure 13 , the feature processing unit 1302 is configured to perform:

[0303] obtaining a song attribute feature of a reference song, the song attribute feature including at least one of a song style feature or a song duration feature;

[0304] The first lyric feature and the song attribute feature are fused to obtain a second fusion feature;

[0305] The second fusion feature and the second noise feature are cross-processed to obtain a second lyric feature through fusion.

[0306] The song attribute of the target song is consistent with the song attribute of the reference song.

[0307] In a possible implementation manner, referring to Figure 13 , the feature processing unit 1302 is configured to perform:

[0308] The second fusion feature is mapped through a first mapping matrix to obtain a query feature.

[0309] The second noise feature is mapped through a second mapping matrix to obtain a key feature and a value feature.

[0310] The query feature, the key feature, and the value feature are fused to obtain a second lyric feature.

[0311] In a possible implementation manner, the number of adaptive transformation networks is k, and the k adaptive transformation networks are connected in sequence; the feature processing unit 1302 is configured to perform:

[0312] The first lyric feature is mapped into k first transformation parameters through the k adaptive transformation networks respectively.

[0313] The feature processing unit 1302 is further configured to perform:

[0314] The first noise feature is transformed through the first adaptive transformation network based on the first transformation parameter to obtain a first second noise feature.

[0315] The (m-1) th third lyric feature is transformed through the m th adaptive transformation network based on the m th transformation parameter to obtain an m th second noise feature; the (m-1) th third lyric feature is the third lyric feature output by the (m-1) th adaptive transformation network, and m is an integer greater than 1 and less than or equal to k.

[0316] In a possible implementation manner, referring to Figure 13 , the feature processing unit 1302 is configured to perform:

[0317] The first lyric feature and the first second noise feature are cross-processed through the first adaptive transformation network to obtain a first second lyric feature through fusion.

[0318] The first lyric feature and the m th second noise feature are cross-processed through the m th adaptive transformation network to obtain an m th second lyric feature through fusion.

[0319] In a possible implementation, referring to Figure 14 , the k adaptive transformation networks further generate k second transformation parameters; the feature processing unit 1302 is configured to perform feature transformation on the first second lyric feature based on the first second transformation parameter through the first adaptive transformation network to obtain a first third lyric feature;

[0320] The feature processing unit 1302 is further configured to perform feature transformation on the mth second lyric feature based on the mth second transformation parameter through the mth adaptive transformation network to obtain an mth third lyric feature;

[0321] The feature processing unit 1302 is further configured to perform decoding on the kth third lyric feature through the decoding network to obtain the target song.

[0322] As to the apparatus in the above embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.

[0323] Figure 14 is a block diagram of a song generation model training apparatus according to an example embodiment. Referring to Figure 14 , the apparatus includes an acquisition unit 1401, a feature extraction unit 1402, a feature processing unit 1403, and a model training unit 1404.

[0324] The acquisition unit 1401 is configured to perform acquisition of sample lyric text and sample songs matched with the sample lyric text;

[0325] The feature extraction unit 1402 is configured to perform feature extraction on the sample lyric text through the second song generation model to obtain first sample lyric features;

[0326] The feature processing unit 1403 is configured to perform mapping of the first sample lyric features to first sample transformation parameters through the second song generation model;

[0327] The feature processing unit 1403 is further configured to perform feature transformation on the first sample noise features based on the first sample transformation parameters through the second song generation model to obtain second sample noise features, the first sample noise features being randomly generated noise features;

[0328] The feature processing unit 1403 is further configured to perform encoding on the sample songs through the second song generation model to obtain target sample lyric features;

[0329] The model training unit 1404 is configured to train the second song generation model based on the first sample lyric feature, the second sample noise feature, and the target sample lyric feature to obtain the first song generation model, the first song generation model being used to generate a song matching any lyric text.

[0330] In a possible implementation, referring to Figure 14 The model training unit 1404 is configured to perform the following operations:

[0331] cross-processing the first sample lyric feature and the second sample noise feature by the second song generation model to obtain a second sample lyric feature;

[0332] determining a loss parameter based on the second sample lyric feature and the target sample lyric feature;

[0333] training the second song generation model based on the loss parameter to obtain the first song generation model.

[0334] In a possible implementation, referring to Figure 15 The second song generation model includes a feature extraction network, an adaptive transformation network, and an encoding network, the step of generating the first sample lyric feature is performed by the feature extraction network; the steps of generating the first sample transformation parameter, the second sample noise feature, and the second sample lyric feature are performed by the adaptive transformation network; and the step of generating the target sample lyric feature is performed by the encoding network.

[0335] The model training unit 1404 is configured to perform the following operations:

[0336] adjusting a model parameter of the second song generation model based on the loss parameter to obtain a third song generation model;

[0337] removing the encoding network in the third song generation model, and adding the trained decoding network in the third song generation model to obtain the first song generation model.

[0338] As to the apparatus in the above-described embodiments, the specific manners in which various units perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0339] The embodiments of the present disclosure provide an electronic device, which includes a processor and a memory for storing instructions executable by the processor. Wherein the processor is configured to execute the instructions to implement the song generation method or the training method of the song generation model as described above.

[0340] In some embodiments, the electronic device includes a terminal. Figure 15is a structural block diagram of a terminal 1500 according to an exemplary embodiment. The terminal 1500 can be a portable mobile terminal such as a smartphone, a tablet, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook, or a desktop computer. The terminal 1500 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0341] The terminal 1500 includes a processor 1501 and a memory 1502.

[0342] The processor 1501 can include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1501 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1501 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. The memory 1502 can include one or more computer-readable storage media, which can be non-transitory. The memory 1502 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1502 is used to store at least one program code for being executed by the processor 1501 to implement a song generation method or a training method of a song generation model provided by the song generation method of the method embodiments in the present disclosure.

[0343] In some embodiments, terminal 1500 can also optionally include a peripheral device interface 1503 and at least one peripheral device. The processor 1501, the memory 1502, and the peripheral device interface 1503 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1503 through a bus, a signal line, or a circuit board. The peripheral devices include at least one of a radio frequency circuit 1504, a display screen 1505, a camera component 1506, an audio circuit 1507, and a power supply 1508.

[0344] The peripheral device interface 1503 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1501 and the memory 1502. In some embodiments, the processor 1501, the memory 1502, and the peripheral device interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1501, the memory 1502, and the peripheral device interface 1503 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0345] The radio frequency circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1504 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1504 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1504 can also include NFC (Near Field Communication) related circuitry, and the present disclosure is not limited in this regard.

[0346] The display screen 1505 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1505 is a touch display screen, the display screen 1505 is further configured to capture touch signals on or above the surface of the display screen 1505. The touch signals can be input to the processor 1501 as control signals for processing. In this case, the display screen 1505 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1505 can be one, disposed on the front panel of the electronic device 1500; in other embodiments, the display screen 1505 can be at least two, respectively disposed on different surfaces of the electronic device 1500 or in a folding design; in other embodiments, the display screen 1505 can be a flexible display screen, disposed on a curved surface or a folding surface of the electronic device 1500. Even, the display screen 1505 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 1505 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0347] The camera assembly 1506 is configured to capture images or videos. Optionally, the camera assembly 1506 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the electronic device, and the rear-facing camera is disposed on the back of the electronic device. In some embodiments, the rear-facing camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function of the main camera and the depth-of-field camera, the panorama shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1506 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0348] The audio circuit 1507 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1501 for processing, or input to the radio frequency circuit 1504 to realize voice communication. The microphone can be multiple for the purpose of stereo sound collection or noise reduction, and is arranged at different parts of the terminal 1500. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert an electrical signal from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker can be a conventional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert an electrical signal into a sound wave audible to humans, but also can convert an electrical signal into an inaudible sound wave to humans for ranging purposes. In some embodiments, the audio circuit 1507 can also include a headphone jack.

[0349] The power supply 1508 is used to supply power to various components in the electronic device 1500. The power supply 1508 can be alternating current, direct current, disposable batteries or rechargeable batteries. When the power supply 1508 includes rechargeable batteries, the rechargeable batteries can be wired charging batteries or wireless charging batteries. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0350] Those skilled in the art can understand that the structure shown in FIG. 16 does not constitute a limitation on the terminal 1500, and can include more or fewer components than shown, or combine certain components, or use different component arrangements. Figure 16 Those skilled in the art can understand that the structure shown in FIG. 16 does not constitute a limitation on the terminal 1500, and can include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0351] In some embodiments, the electronic device includes a server. ​ FIG. 16 is a structural schematic diagram of a server according to an exemplary embodiment. The server 1600 can have a large difference due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1601 and one or more memories 1602, wherein the memory 1602 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1601 to implement the method provided by each method embodiment. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input and output interface, etc. to input and output, and can include other components for realizing device functions, which are not described here.

[0352] In an exemplary embodiment, there is also provided a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the steps in the song generation method or the training method of the song generation model described above. For example, the computer-readable storage medium can be a ROM (ReadOnly Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0353] In an exemplary embodiment, there is also provided a computer program product including a computer program that, when executed by a processor of an electronic device, implements the steps in the song generation method or the training method of the song generation model described above.

[0354] Other embodiments of the present disclosure will be readily apparent to those skilled in the art in view of the disclosure herein. The present disclosure is intended to embrace all such alterations, modifications, and variations of the present disclosure that fall within the scope of the present disclosure. The above specification, examples and data

[0355] It is to be understood that the present disclosure is not limited to the precise construction described above and shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A song generation method characterized by comprising: The method comprises: characteristic extraction is performed on the lyrics text to obtain first lyrics characteristics; the first lyrics characteristics are mapped to first transformation parameters; based on the first transformation parameters, first noise characteristics are transformed in characteristics to obtain second noise characteristics, the first noise characteristics being randomly generated noise characteristics; the first lyrics characteristics and the second noise characteristics are input into an attention layer in an adaptive transformation network of a first song generation model to obtain second lyrics characteristics output by the attention layer, the attention layer comprising a first mapping matrix and a second mapping matrix; based on the second lyrics characteristics, a target song matching the lyrics text is generated.

2. The song generation method according to claim 1, characterized by, The first lyrics characteristics are mapped to the first transformation parameters, comprising: the first lyrics characteristics are mapped to the first transformation parameters and second transformation parameters; based on the second lyrics characteristics, a target song matching the lyrics text is generated, comprising: based on the second transformation parameters, the second lyrics characteristics are transformed in characteristics to obtain third lyrics characteristics; the third lyrics characteristics are decoded to obtain a target song matching the lyrics text.

3. The song generation method according to claim 1 or 2, characterized by, The method further comprises: determining a current time step feature, the time step feature representing a position of a current audio frame to be generated in the target song, the target song comprising a plurality of audio frames; the first lyrics characteristics and the time step feature are fused to obtain first fused characteristics; the first fused characteristics are mapped to the first transformation parameters, or the first fused characteristics are mapped to the first transformation parameters and the second transformation parameters.

4. The song generation method of claim 1, wherein, The first lyrics characteristics are obtained by performing characteristic extraction on the lyrics text, comprising: the lyrics text is converted into a phoneme sequence, the phoneme sequence comprising phonemes of a plurality of words in the lyrics text; the first lyrics characteristics are obtained by performing characteristic extraction on the phoneme sequence.

5. The song generation method of claim 1, wherein, The method further comprises: obtaining song attribute characteristics of a reference song, the song attribute characteristics comprising at least one of song style characteristics or song duration characteristics; the first lyrics characteristics and the song attribute characteristics are fused to obtain second fused characteristics; the second fused characteristics and the second noise characteristics are input into the attention layer to obtain the second lyrics characteristics output by the attention layer; wherein the song attributes of the target song are consistent with the song attributes of the reference song.

6. The song generation method according to claim 5, characterized by, The second fused characteristics and the second noise characteristics are input into the attention layer to obtain the second lyrics characteristics output by the attention layer, comprising: by the attention layer, the first mapping matrix is used to perform characteristic mapping on the second fused characteristics to obtain query characteristics; by the attention layer, the second mapping matrix is used to perform characteristic mapping on the second noise characteristics to obtain key characteristics and value characteristics; by the attention layer, the query characteristics, the key characteristics and the value characteristics are fused to obtain the second lyrics characteristics.

7. The song generation method of claim 1, wherein, The step of generating the first transformation parameter, the second noise feature and the second lyrics feature is performed by the adaptive transformation network, the number of the adaptive transformation network is k, and the k adaptive transformation networks are connected in sequence. The first lyrics feature is mapped into the first transformation parameter by the k adaptive transformation networks. The first noise feature is transformed based on the first transformation parameter to obtain the second noise feature. The first noise feature is transformed based on the first transformation parameter to obtain the second noise feature. The first noise feature is transformed based on the first transformation parameter to obtain the second noise feature.

8. The song generation method according to claim 7, characterized by, The first lyrics feature is mapped into the first transformation parameter by the k adaptive transformation networks. The first lyrics feature is mapped into the first transformation parameter by the k adaptive transformation networks. The first lyrics feature is mapped into the first transformation parameter by the k adaptive transformation networks.

9. The song generation method according to claim 8, characterized by, The k adaptive transformation networks also generate k second transformation parameters. The first noise feature is transformed based on the first transformation parameter to obtain the second noise feature. The first noise feature is transformed based on the first transformation parameter to obtain the second noise feature. The first noise feature is transformed based on the first transformation parameter to obtain the second noise feature. 10.A method for training a song generation model, comprising: The method comprises: Obtaining sample lyrics text and sample songs matched with the sample lyrics text; Extracting features of the sample lyrics text by a second song generation model to obtain first sample lyrics features; Mapping the first sample lyrics features into first sample transformation parameters by the second song generation model; Transforming first sample noise features based on the first sample transformation parameters by the second song generation model to obtain second sample noise features, wherein the first sample noise features are randomly generated noise features; Encoding the sample songs by the second song generation model to obtain target sample lyrics features; inputting the first sample lyric feature and the second sample noise feature into an attention layer in an adaptive transformation network of the second song generation model to obtain a second sample lyric feature output by the attention layer, the attention layer comprising a first mapping matrix and a second mapping matrix; determining a loss parameter based on the second sample lyric feature and the target sample lyric feature, and training the second song generation model based on the loss parameter to obtain a first song generation model, the first song generation model being configured to generate a song matching any lyric text. 11.The method of claim 10, wherein, The second song generation model comprises a feature extraction network, the adaptive transformation network and an encoding network, the step of generating the first sample lyric feature being performed by the feature extraction network, the steps of generating the first sample transformation parameter, the second sample noise feature and the second sample lyric feature being performed by the adaptive transformation network, and the step of generating the target sample lyric feature being performed by the encoding network. The training of the second song generation model based on the loss parameter to obtain a first song generation model comprises: adjusting model parameters of the second song generation model based on the loss parameter to obtain a third song generation model; removing the encoding network in the third song generation model and adding a trained decoding network in the third song generation model to obtain the first song generation model.

12. A song generating apparatus characterized by comprising: The device comprises: a feature extraction unit configured to perform feature extraction on a lyric text to obtain a first lyric feature; a feature processing unit configured to perform mapping of the first lyric feature into a first transformation parameter; the feature processing unit is further configured to perform feature transformation on a first noise feature based on the first transformation parameter to obtain a second noise feature, the first noise feature being a randomly generated noise feature; the feature processing unit is further configured to perform inputting of the first lyric feature and the second noise feature into an attention layer in an adaptive transformation network of a first song generation model to obtain a second lyric feature output by the attention layer, the attention layer comprising a first mapping matrix and a second mapping matrix; the feature processing unit is further configured to perform generation of a target song matching the lyric text based on the second lyric feature.

13. A device for training a song generation model, the device comprising: The device comprises: an acquisition unit configured to perform acquisition of a sample lyric text and a sample song matching the sample lyric text; a feature extraction unit configured to perform feature extraction on the sample lyric text by a second song generation model to obtain a first sample lyric feature; a feature processing unit configured to perform mapping of the first sample lyric feature into a first sample transformation parameter by the second song generation model; the feature processing unit is further configured to perform feature transformation on a first sample noise feature based on the first sample transformation parameter by the second song generation model to obtain a second sample noise feature, the first sample noise feature being a randomly generated noise feature; The feature processing unit is further configured to perform encoding of the sample song by the second song generation model to obtain a target sample song lyric feature; The model training unit is configured to perform inputting of the first sample song lyric feature and the second sample noise feature into an attention layer in an adaptive transformation network of the second song generation model to obtain a second sample song lyric feature output by the attention layer, the attention layer comprising a first mapping matrix and a second mapping matrix; The model training unit is further configured to perform determining of a loss parameter based on the second sample song lyric feature and the target sample song lyric feature, and training of the second song generation model based on the loss parameter to obtain a first song generation model, the first song generation model being used to generate a song matching any song lyric text.

14. An electronic device, comprising: The electronic device comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the song generation method according to any one of claims 1 to 9, or implement the training method of the song generation model according to any one of claims 10 to 11.

15. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the song generation method according to any one of claims 1 to 9, or perform the training method of the song generation model according to any one of claims 10 to 11.

16. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the song generation method according to any one of claims 1 to 9, or implements the training method of the song generation model according to any one of claims 10 to 11.

Images