Song generation method, song generation model training method, device and equipment

Abstract

The invention relates to a song generation method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. The method comprises the steps of performing feature extraction on a lyric text to obtain a first lyric feature; mapping the first lyric feature into a first transformation parameter; performing feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature, the first noise feature being a randomly generated noise feature; and generating a target song matched with the lyric text based on the first lyric feature and the second noise feature. The method does not need to depend on music score knowledge, the convenience of song generation is improved, the lyric features can be captured and utilized more comprehensively by dynamically adjusting and changing the parameters, and the quality of the generated song can be guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a song generation method, a training method, a device and equipment for a song generation model. Background Art

[0002] With the development of voice technology and artificial intelligence, the ways of making songs are becoming more and more diverse. In addition to recording the singer's voice and synthesizing the voice with the accompaniment, some song generation methods based on artificial intelligence technology are currently provided.

[0003] In the related art, a song generation model is used to generate corresponding songs according to lyrics and music score information. However, this method relies on professional music score information, and the method of intelligently generating songs is not simple enough. Summary of the invention

[0004] The present disclosure provides a song generation method, a training method, a device and a equipment for a song generation model, which can improve the convenience and quality of song generation.

[0005] According to one aspect of an embodiment of the present disclosure, a song generation method is provided, the method comprising:

[0006] Perform feature extraction on the lyrics text to obtain a first lyrics feature;

[0007] Mapping the first lyrics feature into a first transformation parameter;

[0008] Based on the first transformation parameter, a first noise feature is transformed to obtain a second noise feature, where the first noise feature is a randomly generated noise feature;

[0009] Based on the first lyrics feature and the second noise feature, a target song matching the lyrics text is generated.

[0010] Optionally, generating a target song matching the lyrics text based on the first lyrics feature and the second noise feature includes:

[0011] Cross-processing the first lyrics feature and the second noise feature to obtain a second lyrics feature;

[0012] Based on the second lyrics feature, a target song matching the lyrics text is generated.

[0013] Optionally, mapping the first lyrics feature to a first transformation parameter includes:

[0014] Mapping the first lyrics feature into the first transformation parameter and the second transformation parameter;

[0015] The step of generating a target song matching the lyrics text based on the second lyrics feature includes:

[0016] Based on the second transformation parameter, performing feature transformation on the second lyrics feature to obtain a third lyrics feature;

[0017] The third lyrics feature is decoded to obtain a target song that matches the lyrics text.

[0018] Optionally, the method further comprises:

[0019] Determine a current time step feature, wherein the time step feature represents a position of a current audio frame to be generated in the target song, wherein the target song includes a plurality of audio frames;

[0020] Fusing the first lyrics feature with the time step feature to obtain a first fused feature;

[0021] The first fused feature is mapped to the first transformation parameter, or the first fused feature is mapped to the first transformation parameter and the second transformation parameter.

[0022] Optionally, the extracting features of the lyrics text to obtain a first lyrics feature includes:

[0023] Converting the lyrics text into a phoneme sequence, wherein the phoneme sequence includes the phonemes of a plurality of words in the lyrics text;

[0024] Perform feature extraction on the phoneme sequence to obtain the first lyrics feature.

[0025] Optionally, cross-processing the first lyrics feature and the second noise feature to obtain a second lyrics feature by fusing the first lyrics feature and the second noise feature includes:

[0026] Acquire song attribute features of a reference song, wherein the song attribute features include at least one of a song style feature or a song duration feature;

[0027] Fusing the first lyrics feature and the song attribute feature to obtain a second fused feature;

[0028] Cross-processing the second fusion feature and the second noise feature to obtain the second lyrics feature;

[0029] The song attributes of the target song are consistent with the song attributes of the reference song.

[0030] Optionally, cross-processing the second fusion feature and the second noise feature to obtain the second lyrics feature by fusing the second fusion feature includes:

[0031] Using the first mapping matrix, performing feature mapping on the second fused features to obtain query features;

[0032] Using a second mapping matrix, performing feature mapping on the second noise feature to obtain a key feature and a value feature;

[0033] The query feature, the key feature and the value feature are fused to obtain the second lyrics feature.

[0034] Optionally, the step of generating the first transformation parameter, the second noise feature and the second lyrics feature is performed by an adaptive transformation network, the number of the adaptive transformation networks is k, and the k adaptive transformation networks are connected in sequence; the step of mapping the first lyrics feature to the first transformation parameter includes:

[0035] Mapping the first lyrics features into k first transformation parameters respectively through k said adaptive transformation networks;

[0036] The step of performing feature transformation on the first noise feature based on the first transformation parameter to obtain the second noise feature includes:

[0037] Performing feature transformation on the first noise feature based on the first first transformation parameter through the first adaptive transformation network to obtain the first second noise feature;

[0038] Through the mth adaptive transformation network, based on the mth first transformation parameter, feature transformation is performed on the m-1th third lyrics feature to obtain the mth second noise feature; wherein the m-1th third lyrics feature is the third lyrics feature output by the m-1th adaptive transformation network, and m is an integer greater than 1 and not greater than k.

[0039] Optionally, cross-processing the first lyrics feature and the second noise feature to obtain a second lyrics feature by fusing the first lyrics feature and the second noise feature includes:

[0040] Through the first adaptive transformation network, the first lyrics feature and the first second noise feature are cross-processed to obtain the first second lyrics feature by fusing;

[0041] The first lyrics feature and the mth second noise feature are cross-processed through the mth adaptive transformation network to obtain the mth second lyrics feature.

[0042] Optionally, the k adaptive transformation networks further generate k second transformation parameters; and the step of generating a target song matching the lyrics text based on the second lyrics feature comprises:

[0043] By using the first of the adaptive transformation networks, based on the first of the second transformation parameters, the first of the second lyrics features is subjected to feature transformation to obtain the first of the third lyrics features;

[0044] By using the mth adaptive transformation network, based on the mth second transformation parameter, the mth second lyrics feature is subjected to feature transformation to obtain the mth third lyrics feature;

[0045] The kth third lyrics feature is decoded through a decoding network to obtain the target song.

[0046] According to another aspect of the embodiments of the present disclosure, a method for training a song generation model is provided, the method comprising:

[0047] Acquire sample lyrics text and a sample song that matches the sample lyrics text;

[0048] Using the second song generation model, feature extraction is performed on the sample lyrics text to obtain first sample lyrics features;

[0049] Mapping the first sample lyrics feature into a first sample transformation parameter through the second song generation model;

[0050] By using the second song generation model, based on the first sample transformation parameter, a feature transformation is performed on the first sample noise feature to obtain a second sample noise feature, wherein the first sample noise feature is a randomly generated noise feature;

[0051] The sample song is encoded by the second song generation model to obtain target sample lyrics features;

[0052] Based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature, the second song generation model is trained to obtain a first song generation model, and the first song generation model is used to generate a song that matches any lyrics text.

[0053] Optionally, the training of the model parameters of the second song generation model based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature to obtain the first song generation model includes:

[0054] The first sample lyrics feature and the second sample noise feature are cross-processed by the second song generation model to obtain a second sample lyrics feature;

[0055] Determining a loss parameter based on the second sample lyrics feature and the target sample lyrics feature;

[0056] Based on the loss parameter, the second song generation model is trained to obtain the first song generation model.

[0057] Optionally, the second song generation model includes a feature extraction network, an adaptive transformation network and an encoding network, the step of generating the first sample lyrics feature is performed by the feature extraction network; the step of generating the first sample transformation parameter, the second sample noise feature and the second sample lyrics feature is performed by the adaptive transformation network; the step of generating the target sample lyrics feature is performed by the encoding network;

[0058] The step of training the second song generation model based on the loss parameter to obtain the first song generation model comprises:

[0059] Based on the loss parameter, adjusting the model parameters of the second song generation model to obtain a third song generation model;

[0060] The encoding network in the third song generation model is removed, and the trained decoding network is added to the third song generation model to obtain the first song generation model.

[0061] According to another aspect of the embodiments of the present disclosure, a song generation device is provided, the device comprising:

[0062] A feature extraction unit is configured to perform feature extraction on the lyrics text to obtain a first lyrics feature;

[0063] A feature processing unit, configured to perform mapping of the first lyrics feature into a first transformation parameter;

[0064] The feature processing unit is further configured to perform feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature, wherein the first noise feature is a randomly generated noise feature;

[0065] The feature processing unit is further configured to generate a target song matching the lyrics text based on the first lyrics feature and the second noise feature.

[0066] Optionally, the feature processing unit is configured to execute:

[0067] Cross-processing the first lyrics feature and the second noise feature to obtain a second lyrics feature;

[0068] Based on the second lyrics feature, a target song matching the lyrics text is generated.

[0069] Optionally, the feature processing unit is configured to perform mapping of the first lyrics feature into the first transformation parameter and the second transformation parameter;

[0070] The feature processing unit is further configured to perform feature transformation on the second lyrics feature based on the second transformation parameter to obtain a third lyrics feature;

[0071] The feature processing unit is further configured to decode the third lyrics feature to obtain a target song that matches the lyrics text.

[0072] Optionally, the feature processing unit is further configured to execute:

[0073] Determine a current time step feature, wherein the time step feature represents a position of a current audio frame to be generated in the target song, wherein the target song includes a plurality of audio frames;

[0074] Fusing the first lyrics feature with the time step feature to obtain a first fused feature;

[0075] The first fused feature is mapped to the first transformation parameter, or the first fused feature is mapped to the first transformation parameter and the second transformation parameter.

[0076] Optionally, the feature extraction unit is configured to perform:

[0077] Converting the lyrics text into a phoneme sequence, wherein the phoneme sequence includes the phonemes of a plurality of words in the lyrics text;

[0078] Perform feature extraction on the phoneme sequence to obtain the first lyrics feature.

[0079] Optionally, the feature processing unit is configured to execute:

[0080] Acquire song attribute features of a reference song, wherein the song attribute features include at least one of a song style feature or a song duration feature;

[0081] Fusing the first lyrics feature and the song attribute feature to obtain a second fused feature;

[0082] Cross-processing the second fusion feature and the second noise feature to obtain the second lyrics feature;

[0083] The song attributes of the target song are consistent with the song attributes of the reference song.

[0084] Optionally, the feature processing unit is configured to execute:

[0085] Using the first mapping matrix, performing feature mapping on the second fused features to obtain query features;

[0086] Using a second mapping matrix, performing feature mapping on the second noise feature to obtain a key feature and a value feature;

[0087] The query feature, the key feature and the value feature are fused to obtain the second lyrics feature.

[0088] Optionally, the number of the adaptive transformation networks is k, and the k adaptive transformation networks are connected in sequence; the feature processing unit is configured to perform:

[0089] Mapping the first lyrics features into k first transformation parameters respectively through k said adaptive transformation networks;

[0090] The feature processing unit is further configured to execute:

[0091] Performing feature transformation on the first noise feature based on the first first transformation parameter through the first adaptive transformation network to obtain the first second noise feature;

[0092] Through the mth adaptive transformation network, based on the mth first transformation parameter, feature transformation is performed on the m-1th third lyrics feature to obtain the mth second noise feature; wherein the m-1th third lyrics feature is the third lyrics feature output by the m-1th adaptive transformation network, and m is an integer greater than 1 and not greater than k.

[0093] Optionally, the feature processing unit is configured to execute:

[0094] Through the first adaptive transformation network, the first lyrics feature and the first second noise feature are cross-processed to obtain the first second lyrics feature by fusing;

[0095] The first lyrics feature and the mth second noise feature are cross-processed through the mth adaptive transformation network to obtain the mth second lyrics feature.

[0096] Optionally, the k adaptive transformation networks further generate k second transformation parameters; the feature processing unit is configured to perform feature transformation on the first second lyrics feature through the first adaptive transformation network based on the first second transformation parameter to obtain the first third lyrics feature;

[0097] The feature processing unit is further configured to perform feature transformation on the mth second lyrics feature based on the mth second transformation parameter through the mth adaptive transformation network to obtain the mth third lyrics feature;

[0098] The feature processing unit is further configured to perform decoding on the kth third lyrics feature through a decoding network to obtain the target song.

[0099] According to another aspect of the embodiments of the present disclosure, a training device for a song generation model is provided, the device comprising:

[0100] An acquisition unit is configured to acquire a sample lyrics text and a sample song matching the sample lyrics text;

[0101] A feature extraction unit is configured to perform feature extraction on the sample lyrics text through a second song generation model to obtain first sample lyrics features;

[0102] a feature processing unit configured to execute mapping of the first sample lyrics feature into a first sample transformation parameter through the second song generation model;

[0103] The feature processing unit is further configured to perform feature transformation on the first sample noise feature based on the first sample transformation parameter through the second song generation model to obtain a second sample noise feature, wherein the first sample noise feature is a randomly generated noise feature;

[0104] The feature processing unit is further configured to perform encoding of the sample song through the second song generation model to obtain target sample lyrics features;

[0105] The model training unit is configured to train the second song generation model based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature to obtain a first song generation model, wherein the first song generation model is used to generate a song that matches any lyrics text.

[0106] Optionally, the model training unit is configured to execute:

[0107] The first sample lyrics feature and the second sample noise feature are cross-processed by the second song generation model to obtain a second sample lyrics feature;

[0108] Determining a loss parameter based on the second sample lyrics feature and the target sample lyrics feature;

[0109] Based on the loss parameter, the second song generation model is trained to obtain the first song generation model.

[0110] Optionally, the second song generation model includes the feature extraction network, the adaptive transformation network and the encoding network, the step of generating the first sample lyrics feature is performed by the feature extraction network; the step of generating the first sample transformation parameter, the second sample noise feature and the second sample lyrics feature is performed by the adaptive transformation network; the step of generating the target sample lyrics feature is performed by the encoding network;

[0111] The model training unit is configured to execute:

[0112] Based on the loss parameter, adjusting the model parameters of the second song generation model to obtain a third song generation model;

[0113] The encoding network in the third song generation model is removed, and the trained decoding network is added to the third song generation model to obtain the first song generation model.

[0114] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0115] processor;

[0116] a memory for storing instructions executable by the processor;

[0117] Among them, the processor is configured to execute the instructions to implement the song generation method as described in the above aspects, or to implement the training method of the song generation model as described in the above aspects.

[0118] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the song generation method as described in the above aspects, or execute the song generation model training method as described in the above aspects.

[0119] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the song generation method as described in the above aspects, or implements the song generation model training method as described in the above aspects.

[0120] The solution provided by the disclosed embodiment introduces randomly generated noise features when intelligently generating a matching target song through lyrics text. The lyrics text provides semantic information for the generation process of the target song, and the random noise features provide the potential ability to generate songs with different rhythms. By transforming the noise features according to the transformation parameters generated according to the lyrics features, the rhythm of the song can be flexibly adjusted according to the lyrics text without relying on music score knowledge, thereby improving the convenience of generating songs. By dynamically adjusting the transformation parameters, the lyrics features can be captured and utilized more comprehensively, which is conducive to ensuring the quality of the generated songs.

[0121] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0123] Figure 1 The diagram is a schematic diagram of a computer system according to an exemplary embodiment.

[0124] Figure 2 The present invention is a flowchart of a song generating method according to an exemplary embodiment.

[0125] Figure 3 It is a structural diagram of a first song generation model according to an exemplary embodiment.

[0126] Figure 4 It is a structural diagram of another first song generation model according to an exemplary embodiment.

[0127] Figure 5 It is a schematic diagram showing the structure of an adaptive transformation network according to an exemplary embodiment.

[0128] Figure 6 is a flow chart of another song generating method according to an exemplary embodiment.

[0129] Figure 7 The figure is a flow chart of a method for generating lyrics features according to an exemplary embodiment.

[0130] Figure 8 is a flowchart of yet another song generating method according to an exemplary embodiment.

[0131] Fig. 9 is a flowchart of yet another song generating method according to an exemplary embodiment.

[0132] Fig.10It is a flowchart of a method for training a song generation model according to an exemplary embodiment.

[0133] Fig.11 It is a flowchart of another method for training a song generation model according to an exemplary embodiment.

[0134] Fig.12 It is a structural diagram of a second song generation model according to an exemplary embodiment.

[0135] Fig.13 It is a block diagram of a song generating device according to an exemplary embodiment.

[0136] Fig.14 It is a block diagram of a training device for a song generation model according to an exemplary embodiment.

[0137] Fig.15 It is a block diagram of a terminal according to an exemplary embodiment.

[0138] Fig.16 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0139] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0140] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned figure descriptions are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties. The collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the lyrics text, lyrics features, target songs, etc. involved in this disclosure are all obtained with full authorization.

[0142] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0143] TTS (Text-To-Speech) is one of the core technologies of intelligent voice interaction. It converts the received text sequence into a natural and realistic voice waveform and transmits the feedback to the user. Speech synthesis technology directly affects the actual use effect of human-computer interaction. Speech synthesis technology involves multiple disciplines such as speech signal processing, pattern recognition, natural language processing, acoustics, linguistics, etc. It is an indispensable key technology in the field of information processing.

[0144] AIGC (Artificial Intelligence Generated Content) refers to various types of content created using artificial intelligence technology, such as text, images, music, and videos.

[0145] G2P (Grapheme-to-Phoneme) refers to the pinyin or phoneme annotation of the output text, usually using a rule-based dictionary mapping method. There is a problem of polyphonetic character recognition in the process of grapheme-to-phoneme conversion, and the result of grapheme-to-phoneme conversion has a great impact on the pronunciation of the synthesis system.

[0146] VAE (Variational Autoencoder) refers to transforming real samples into an ideal data distribution through an encoder network, and passing the data distribution to a decoder network to obtain generated samples. Further variational processing is performed on the autoencoder model so that the output of the encoder can correspond to the mean and variance of the target distribution.

[0147] The disclosed embodiments provide a song generation method and a song generation model training method, and the execution subject is an electronic device. Exemplarily, the electronic device is a terminal, and the terminal is a computer, a mobile phone, a tablet computer or other terminals. Exemplarily, the electronic device is a server, and the server is a background server or a cloud server that provides cloud computing and cloud storage services. Among them, the electronic device can generate a target song that matches the lyrics text based on the lyrics text. Alternatively, the electronic device can train a first song generation model, and the first song generation model is used to generate a song that matches any lyrics text.

[0148] Figure 1 is a schematic diagram of a computer system according to an exemplary embodiment. Figure 1 , the computer system includes: a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0149] The terminal 101 has a client installed and running, and the client can be a social application client, an online payment client, an online shopping client, a game client, a medical service client, a video client, etc. When the terminal 101 runs the client, the user interface of the client is displayed on the screen of the terminal 101. The terminal 101 is a terminal used by a user.

[0150] Optionally, terminal 101 may generally refer to one of a plurality of terminals. Those skilled in the art may know that the number of the terminals may be more or less. For example, the terminal may be only one, or the number of the terminals may be 6, 8 or more. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0151] The server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 is used to provide background services for the client. Optionally, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing.

[0152] In one embodiment, the terminal 101 sends a song generation request carrying lyrics text to the server 102. After receiving the song generation request, the server 102 obtains the lyrics text. Based on the lyrics text, the server 102 generates a target song that matches the lyrics text. The server 102 returns the target song to the terminal 101, and the terminal 101 can display the target song to the user. Exemplarily, the server 102 trains a first song generation model for generating songs. After receiving the song generation request, the server 102 generates a target song that matches the lyrics text through the first song generation model.

[0153] In another embodiment, the server 102 trains a first song generation model for generating songs, and sends the first song generation model to the terminal 101. The terminal 101 generates a target song that matches the lyrics text through the first song generation model.

[0154] It should be noted that the above implementation environment is only an example. The method provided in the embodiment of the present disclosure can also be executed by the terminal 101 or the server 102 alone, or by other computer devices, and the embodiment of the present disclosure does not limit this.

[0155] Figure 2 is a flowchart of a song generation method according to an exemplary embodiment. Figure 2 The method is performed by an electronic device and comprises the following steps:

[0156] In step 201, the electronic device extracts features from the lyrics text to obtain a first lyrics feature.

[0157] The electronic device obtains a lyric text, and the purpose of the embodiment of the present disclosure is to intelligently generate a target song that matches the lyric text. The electronic device extracts features from the obtained lyric text to obtain a first lyric feature. The first lyric feature is used to reflect the semantics of the lyric text and the rhythm to be expressed.

[0158] In step 202, the electronic device maps the first lyrics feature into a first transformation parameter.

[0159] After acquiring the first lyrics feature, the electronic device adaptively generates a first transformation parameter based on the first lyrics feature. The first transformation parameter is used to transform the first noise feature randomly generated in the following step 203 during the process of generating a song.

[0160] In step 203, the electronic device performs feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature, where the first noise feature is a randomly generated noise feature.

[0161] The electronic device obtains a first noise feature, which may be a randomly generated noise feature, so the first noise feature has a certain degree of randomness. The electronic device performs feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature. Since the first transformation parameter is obtained based on the first lyrics feature, the second noise feature embeds features related to the lyrics text, adaptively reducing the randomness of the noise feature and enhancing the relevance of the noise feature to the lyrics text.

[0162] In step 204, the electronic device generates a target song that matches the lyrics text based on the first lyrics feature and the second noise feature.

[0163] After determining the first lyrics feature and the second noise feature, the electronic device generates a target song based on the first lyrics feature and the second noise feature, and the target song matches the lyrics text. That is, the lyrics of the target song are the lyrics text.

[0164] The method provided by the embodiment of the present disclosure introduces randomly generated noise features when intelligently generating a matching target song through lyrics text. The lyrics text provides semantic information for the generation process of the target song, and the random noise features provide the potential ability to generate songs with different rhythms. By transforming the noise features according to the transformation parameters generated according to the lyrics features, the rhythm of the song can be flexibly adjusted according to the lyrics text without relying on the knowledge of the musical score, thereby improving the convenience of generating songs. By dynamically adjusting the transformation parameters, the lyrics features can be more comprehensively captured and utilized, which is conducive to ensuring the quality of the generated songs.

[0165] Above Figure 2 The embodiment is only a brief description of the song generation method. In some embodiments, the electronic device can also perform the process of generating a target song matching the lyrics text through a first song generation model. The first song generation model includes a feature extraction network, an adaptive transformation network and a decoding network.

[0166] Figure 3 is a structural diagram of a first song generation model according to an exemplary embodiment. Figure 3 As shown, the first song generation model includes a feature extraction network, an adaptive transformation network and a decoding network. The feature extraction network is connected to the adaptive transformation network, and the output of the feature extraction network is the input of the adaptive transformation network. The adaptive transformation network is connected to the decoding network, and the output of the adaptive transformation network is the input of the decoding network.

[0167] Among them, the feature extraction network is used to extract the features of the lyrics text, the adaptive transformation network is used to transform and fuse the lyrics features and noise features, and the decoding network is used to decode the lyrics features to generate songs.

[0168] In some embodiments, Figure 4 As shown, the first song generation model includes k adaptive transformation networks connected in sequence, the feature extraction network includes a first feature extraction network, a second feature extraction network and a third feature extraction network, and the first feature extraction network, the second feature extraction network and the third feature extraction network are respectively connected to the k adaptive transformation networks.

[0169] The first feature extraction network is used to extract features from the lyrics text to obtain the first lyrics feature. For example, Figure 4 As shown, the first feature extraction network includes two parts, the first part is G2P and embedder (embedding layer), and the second part is Transformer (a neural network layer). Among them, the first part is used to convert the lyrics text into a phoneme sequence, and the second part is used to extract features of the phoneme sequence to obtain the first lyrics feature. Among them, the second feature extraction network is used to extract the song style features of the reference song, and the second feature extraction network is used to extract the song duration features of the reference song. In the embodiment of the present disclosure, the first lyrics features, song style features, and song duration features output by the first feature extraction network, the second feature extraction network, and the third feature extraction network are fused to obtain a second fused feature, and the second fused features are respectively input into k adaptive transformation networks for processing.

[0170] Among them, the input of the first adaptive transformation network is the second fusion feature and the first noise feature, and the output is the first third lyrics feature. The input of the second adaptive transformation network is the second fusion feature and the first third lyrics feature, and the output is the second third lyrics feature. The input of the third adaptive transformation network is the second fusion feature and the second third lyrics feature, and the output is the third third lyrics feature. And so on, the input of the kth adaptive transformation network (that is, the last adaptive transformation network) is the second fusion feature and the k-1th third lyrics feature, and the output is the kth third lyrics feature.

[0171] In some embodiments, Figure 5 As shown, the adaptive transformation network includes a normalization layer, a linear transformation layer, an attention layer, a scale transformation layer and a multi-layer perceptron. The input of the multi-layer perceptron is a first fusion feature obtained by fusing the time step feature and the first lyrics feature, and the output is a first transformation parameter and a second transformation parameter. The input of the normalization layer is the first noise feature or the output of the previous adaptive transformation network. The input of the linear transformation layer is the output of the normalization layer and the first transformation parameter, and the output is the second noise feature. The input of the attention layer is the first lyrics feature and the output of the linear transformation layer, and the output is the second lyrics feature. The input of the scale transformation layer is the second lyrics feature and the second transformation parameter, and the output is the third lyrics feature.

[0172] Among them, the input of the decoding network is the kth third lyrics feature, and the output is the target song.

[0173] The specific process of the electronic device generating songs through the first song generation model is as follows Figure 6 Embodiment of the invention. Figure 6 is a flowchart of another song generation method according to an exemplary embodiment, the method is executed by an electronic device, see Figure 6 , the method comprises the following steps:

[0174] In step 601, the electronic device extracts features from the lyrics text through a feature extraction network in a first song generation model to obtain a first lyrics feature.

[0175] The electronic device obtains a lyric text, inputs the lyric text into a feature extraction network in a first song generation model, and processes the lyric text with the feature extraction network to output a first lyric feature.

[0176] In some embodiments, the electronic device performs the following steps through a feature extraction network: converting the lyrics text into a phoneme sequence, where the phoneme sequence includes the phonemes of multiple words in the lyrics text; and performing feature extraction on the phoneme sequence to obtain a first lyrics feature.

[0177] Among them, phonemes are the smallest units or smallest speech fragments that constitute syllables, or in other words, phonemes are the smallest speech units divided according to the natural properties of speech. Each phoneme has its own unique sound characteristics, such as vowel phonemes a, e, i, o, u, and consonant phonemes such as b, p, m, f, etc. In addition, tones also belong to phonemes.

[0178] For example, Figure 4 As shown, the feature extraction network includes a first part (G2P and embedder) and a second part (Transformer). The first part is used to convert the lyrics text into a phoneme sequence, and the second part is used to extract features from the phoneme sequence to obtain the first lyrics feature. Among them, G2P can be a G2P based on a conditional random field or a G2P based on BERT, etc., which is not limited in the embodiments of the present disclosure.

[0179] Figure 7 is a flowchart of a method for generating lyrics features according to an exemplary embodiment. Figure 7 As shown, after obtaining the lyrics text, the lyrics text is preprocessed and normalized, the preprocessed and normalized lyrics text is divided into sentences and words, the pronunciation conversion is performed based on the sentence division and word segmentation, the phoneme sequence of the lyrics text is output, the phoneme sequence is content encoded, and the first lyrics feature is obtained.

[0180] In the disclosed embodiment, the lyrics text is converted into a phoneme sequence, which can extract the pronunciation features of the lyrics from the phonetics level, capture its rhythm, syllables and other information, enhance the ability of the lyrics to express the voice features, and enable the generated song to better reflect the voice characteristics of the lyrics. In addition, by extracting the first lyrics feature based on the phoneme sequence, the pronunciation rules of the lyrics can be more closely combined with the music melody, and the generated target song is more in line with the rhythm and rhythm requirements of the lyrics in terms of rhythm and melody.

[0181] In step 602, the electronic device maps the first lyrics feature into a first transformation parameter and a second transformation parameter through an adaptive transformation network in a first song generation model.

[0182] After acquiring the first lyrics feature, the computer device maps the first lyrics feature into a first transformation parameter and a second transformation parameter through an adaptive transformation network in the first song generation model. The first transformation parameter is used to transform the noise feature randomly generated in the following step 603 during the song generation process, and the second transformation parameter is used to transform the second lyrics feature generated in the following step 604 during the song generation process.

[0183] In some embodiments, the electronic device performs the following steps through an adaptive transformation network: determining a current time step feature, the time step feature represents the position of a current audio frame to be generated in a target song, the target song includes multiple audio frames; fusing the first lyrics feature with the time step feature to obtain a first fused feature; mapping the first fused feature into a first transformation parameter and a second transformation parameter.

[0184] In the disclosed embodiment, the target song generated by the song generation model includes a plurality of continuous audio frames, and the plurality of continuous audio frames are generated in sequence in a chronological order. Then, in the process of generating the current audio frame, the electronic device obtains a time step feature for representing the position of the current audio frame to be generated in the target song through the song generation model, and generates a first transformation parameter and a second transformation parameter based on the first lyrics feature and the time step feature.

[0185] Exemplarily, the electronic device concatenates the first lyrics feature with the time step feature through an adaptive transformation network to obtain a first fusion feature.

[0186] For example, Figure 4 As shown, the adaptive transformation network includes a multi-layer perceptron, the electronic device inputs the first fusion feature into the multi-layer perceptron, and the multi-layer perceptron outputs a first transformation parameter and a second transformation parameter.

[0187] It should be noted that in the disclosed embodiment, not only the first noise feature is transformed by the first transformation parameter, but also the second lyrics feature is transformed by the second transformation parameter, so it is necessary to map the first fusion feature to the first transformation parameter and the second transformation parameter. In another embodiment, the first noise feature can be transformed only by a transformation parameter, and the second lyrics feature does not need to be transformed, so it is only necessary to map the first fusion feature to the first transformation parameter.

[0188] In the disclosed embodiment, by introducing the time step feature and fusing it with the first lyrics feature, the temporal relationship between the lyrics text and each audio frame in the target song can be effectively established, so that the position of each audio frame can be accurately modeled, avoiding the rhythm dislocation or mismatch between the audio frame and the lyrics content that may occur during the generation process, thereby improving the accuracy and naturalness of the target song.

[0189] In step 603, the electronic device performs feature transformation on the first noise feature based on the first transformation parameter through the adaptive transformation network in the first song generation model to obtain a second noise feature, where the first noise feature is a randomly generated noise feature.

[0190] The electronic device obtains a first noise feature, which may be a randomly generated noise feature, so the first noise feature has a certain degree of randomness. The electronic device performs feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature. Since the first transformation parameter is obtained based on the first lyrics feature, the second noise feature embeds features related to the lyrics text, adaptively reducing the randomness of the noise feature and enhancing the relevance of the noise feature to the lyrics text.

[0191] In some embodiments, Figure 5 As shown, the adaptive transformation network includes a normalization layer and a linear transformation layer. The electronic device inputs the first noise feature into the normalization layer, and the normalization layer outputs the normalized first noise feature. The electronic device inputs the normalized first noise feature and the first transformation parameter into the linear transformation layer, and the linear transformation layer outputs the second noise feature. Exemplarily, the normalization layer can be a Layer Norm layer, and the linear transformation layer can be a Scale&Shift layer.

[0192] In step 604, the electronic device cross-processes the first lyrics feature and the second noise feature through the adaptive transformation network in the first song generation model to obtain the second lyrics feature by fusing them.

[0193] After acquiring the first lyrics feature and the second noise feature, the electronic device cross-processes the first lyrics feature and the second noise feature to obtain the second lyrics feature. Since the second lyrics feature combines the first lyrics feature and the second noise feature, the second lyrics feature can not only reflect the characteristics of the lyrics text, but also has a certain degree of randomness, providing the potential ability to generate songs with different rhythms. Therefore, combining the two text-guiding mechanisms of feature cross-processing and dynamic adjustment of transformation parameters can more comprehensively capture and utilize lyrics features, which is conducive to ensuring the quality of the generated songs.

[0194] In some embodiments, the electronic device obtains a song attribute feature of a reference song, the song attribute feature includes at least one of a song style feature or a song duration feature, and fuses the first lyrics feature and the song attribute feature. The electronic device performs the following steps through an adaptive transformation network: cross-processing the second fusion feature and the second noise feature to obtain a second lyrics feature. The song attribute of the target song is consistent with the song attribute of the reference song.

[0195] The reference song may be any type of song. The song attributes of the reference song are used as a reference to participate in the generation process of the target song, which can guide the song generation model to generate a target song that is consistent with the song attributes of the reference song.

[0196] For example, Figure 4As shown, the feature extraction network of the song generation model includes a first feature extraction network, a second feature extraction network and a third feature extraction network. The electronic device inputs the reference song into the second feature extraction network, and the second feature extraction network outputs the song style feature. The electronic device inputs the reference song duration of the reference song into the third feature extraction network, and the third feature extraction network outputs the song duration feature.

[0197] Exemplarily, the electronic device combines the first lyrics feature, the song style feature and the song duration feature to obtain the second fusion feature.

[0198] In the disclosed embodiment, by introducing the song attribute feature of the reference song and fusing it with the first lyrics feature, the generated target song can inherit the specific attributes of the reference song, thereby achieving precise control of the style of the target song. Based on the introduction of the second noise feature, this embodiment ensures that the target song has a certain uniqueness while maintaining the consistency of the reference song attributes.

[0199] Optionally, the electronic device performs the following steps through an adaptive transformation network: using a first mapping matrix to perform feature mapping on the second fused feature to obtain a query feature; using a second mapping matrix to perform feature mapping on the second noise feature to obtain a key feature and a value feature; fusing the query feature, the key feature and the value feature to obtain a second lyric feature. The key feature (key), value feature (value) and query feature (query) in the disclosed embodiment belong to different feature spaces respectively.

[0200] The electronic device obtains a first mapping matrix, which is used to perform spatial transformation on the second fused feature, multiplies the second fused feature by the first mapping matrix to obtain a first spatial matrix, and separates the query feature from the first spatial matrix. For example, the first mapping matrix is ​​a 3-dimensional mapping matrix, and the first spatial matrix obtained by multiplying the second fused feature by the first mapping matrix is ​​a 3-dimensional first spatial matrix, and the electronic device uses the features of the preset dimensions in the first spatial matrix as query features.

[0201] The electronic device obtains a second mapping matrix, which is used to perform spatial transformation on the second noise feature, multiplies the second noise feature by the second mapping matrix to obtain a second spatial matrix, and separates the key feature and the value feature from the second spatial matrix. For example, the second mapping matrix is ​​a 3-dimensional mapping matrix, and the second spatial matrix obtained by multiplying the second noise feature by the second mapping matrix is ​​a 3-dimensional second spatial matrix, and the electronic device uses the features on the preset dimensions in the second spatial matrix as key features and value features.

[0202] Optionally, the first mapping matrix and the second mapping matrix are model parameters of an adaptive transformation network in a song generation model, and the first mapping matrix and the second mapping matrix are obtained by training the song generation model.

[0203] Optionally, the electronic device normalizes the product of the query feature, the transposition of the key feature and the scaling factor to obtain a normalized feature, and multiplies the normalized feature by the value feature to obtain a second lyric feature.

[0204] For example, Figure 4 As shown, the adaptive transformation network in the song generation model includes an attention layer, the model parameters of the attention layer include a first mapping matrix and a second mapping matrix, the electronic device inputs the second fusion feature and the second noise feature into the attention layer, and the attention layer outputs the second lyrics feature. For example, the attention layer can be a cross-attention layer, forward-attention, GMM-attention, location-attention, etc., which is not limited in the embodiments of the present disclosure.

[0205] In the disclosed embodiment, the second fusion feature is mapped to the query feature, and the second noise feature is mapped to the key feature and the value feature, and an attention mechanism is introduced to ensure that important information is focused on during the feature fusion process and to avoid interference from redundant or irrelevant features. This further enhances the comprehensive expression capability of multi-dimensional information such as rhythm and semantics in the lyrics features, making the generated second lyrics features more in line with the target song requirements, and is conducive to improving the accuracy and naturalness of song generation.

[0206] Moreover, by introducing the cross-attention mechanism, the computational burden of the song generation model is significantly reduced, although the expressive power of the song generation model is retained. By removing the cross-attention mask, it is helpful to speed up the iteration speed of the song generation model and improve the overall training efficiency of the song generation model.

[0207] In step 605, the electronic device performs feature transformation on the second lyrics feature based on the second transformation parameter through the adaptive transformation network in the first song generation model to obtain a third lyrics feature.

[0208] The electronic device performs feature transformation on the second lyrics feature based on the second transformation parameter to obtain a third lyrics feature. Since the second transformation parameter is obtained based on the first lyrics feature, the third lyrics feature embeds features related to the lyrics text, further enhancing the relevance of the third lyrics feature to the lyrics text.

[0209] In some embodiments, Figure 4As shown, the adaptive transformation network includes a scale transformation layer, the electronic device inputs the second lyrics feature and the second transformation parameter into the scale transformation layer, and the scale transformation layer outputs the third lyrics feature. Exemplarily, the scale transformation layer may be a Scale layer.

[0210] In step 606, the electronic device decodes the third lyrics feature through the decoding network in the first song generation model to obtain a target song that matches the lyrics text.

[0211] The electronic device inputs the third lyrics feature into the decoding network in the first song generation model, and the decoding network decodes the third lyrics feature to obtain a target song, which matches the lyrics text. That is, the lyrics of the target song are the lyrics text.

[0212] Exemplarily, the decoding network can be a decoding network in a VAE model architecture.

[0213] In the disclosed embodiment, the second lyric feature obtained after cross processing is transformed according to the transformation parameters generated by the first lyric feature, which can effectively extract the potential features of the lyrics, so that the transformed third lyric feature can more accurately represent the emotion, rhythm and other information of the lyric text, providing high-quality input for subsequent decoding, which is conducive to improving the matching degree between the decoded target song and the lyric text.

[0214] Figure 8 is a flowchart of another song generation method according to an exemplary embodiment. Figure 8 As shown, the song generation process can be divided into two parts: front-end lyric text analysis and back-end song synthesis. The front-end part provides a lyric analysis function for the input lyric text, and the back-end part synthesizes the song based on the analysis result of the lyric text. Among them, the song generation method of the embodiment of the present disclosure can be applied to any scenario where songs need to be generated, such as song creation, film and television soundtracks, game background music and other fields.

[0215] The method provided by the embodiment of the present disclosure introduces randomly generated noise features when intelligently generating a matching target song through lyrics text. The lyrics text provides semantic information for the generation process of the target song, and the random noise features provide the potential ability to generate songs with different rhythms. By transforming the noise features according to the transformation parameters generated by the lyrics features, the rhythm of the song can be flexibly adjusted according to the lyrics text without relying on the knowledge of the musical score, thereby improving the convenience of generating songs. In addition, the present disclosure combines the two text-guiding mechanisms of feature cross-processing and dynamic adjustment of transformation parameters, which can more comprehensively capture and utilize the lyrics features, which is conducive to ensuring the quality of the generated songs.

[0216] Furthermore, executing the process of generating a target song based on the lyrics text through the first song generation model is helpful to improving the convenience and efficiency of the song generation process.

[0217] It should be noted that the disclosed embodiment only takes the example of an electronic device executing the song generation method of steps 601 to 606 through a first song generation model. In another embodiment, the electronic device may also directly execute the song generation method of steps 601 to 606 without relying on the first song generation model.

[0218] In the above Figure 6 Based on the example, Figure 4 As shown, the number of adaptive transformation networks in the first song generation model is k, and the k adaptive transformation networks are connected in sequence. Then, the detailed process of generating the target song is as follows Fig. 9 Embodiment of the invention. Fig. 9 is a flowchart of another song generation method according to an exemplary embodiment, the method is executed by an electronic device, see Fig. 9 , the method comprises the following steps:

[0219] In step 901, the electronic device extracts features from the lyrics text through a feature extraction network in a first song generation model to obtain a first lyrics feature.

[0220] The process of extracting the first lyrics feature in step 901 is the same as the process of extracting the first lyrics feature in step 601 above, and will not be repeated here.

[0221] In step 902, the electronic device maps the first lyric feature into a first first transformation parameter and a first second transformation parameter through a first adaptive transformation network.

[0222] like Figure 5 As shown, the first adaptive transformation network includes a multi-layer perceptron, the electronic device inputs the first lyrics feature and the time step feature into the multi-layer perceptron in the first adaptive transformation network, and the multi-layer perceptron in the first adaptive transformation network outputs the first first transformation parameter and the first second transformation parameter.

[0223] The process of generating the first first transformation parameter and the first second transformation parameter in step 902 is the same as the process of generating the first transformation parameter and the second transformation parameter in step 602, and will not be repeated here.

[0224] In step 903, the electronic device performs feature transformation on the first noise feature based on the first first transformation parameter through the first adaptive transformation network to obtain the first second noise feature.

[0225] like Figure 5As shown, the first adaptive transformation network includes a normalization layer and a linear transformation layer, and the electronic device inputs the first noise feature into the normalization layer to obtain the normalized first noise feature. The electronic device inputs the normalized first noise feature and the first first transformation parameter output by the multilayer perceptron in the first adaptive transformation network into the linear transformation layer, and the linear transformation layer outputs the first second noise feature.

[0226] The process of generating the first second noise feature in step 903 is the same as the process of generating the second noise feature in step 603, and will not be repeated here.

[0227] In step 904, the electronic device cross-processes the first lyrics feature and the first second noise feature through the first adaptive transformation network to obtain the first second lyrics feature by fusing them.

[0228] like Figure 5 As shown, the first adaptive transformation network includes an attention layer, and the electronic device inputs the first lyrics feature and the first second noise feature output by the linear transformation layer in the first adaptive transformation network into the attention layer, and the attention layer outputs the first second lyrics feature.

[0229] The process of generating the first second lyrics feature in step 904 is the same as the process of generating the second lyrics feature in step 604 above, and will not be repeated here.

[0230] In step 905, the electronic device performs feature transformation on the first second lyrics feature based on the first second transformation parameter through the first adaptive transformation network to obtain the first third lyrics feature.

[0231] like Figure 5 As shown, the first adaptive transformation network includes a scale transformation layer, and the electronic device inputs the first second lyrics feature output by the attention layer in the first adaptive transformation network and the first second transformation parameter output by the multilayer perceptron in the first adaptive transformation network into the scale transformation layer, and the scale transformation layer outputs the first third lyrics feature.

[0232] Among them, the first third lyrics feature is used to input into the second adaptive transformation network, such as Figure 5 As shown, the first third lyrics feature is input into the normalization layer in the second adaptive transformation network.

[0233] The process of generating the first third lyrics feature in step 905 is the same as the process of generating the third lyrics feature in step 605, and will not be repeated here.

[0234] In step 906, the electronic device generates an mth first transformation parameter and an mth second transformation parameter based on the first lyrics feature through the mth adaptive transformation network, where m is an integer greater than 1 and not greater than k.

[0235] Take m equal to 2 as an example, Figure 5 As shown, the second adaptive transformation network includes a multi-layer perceptron, the electronic device inputs the first lyrics feature and the time step feature into the multi-layer perceptron in the second adaptive transformation network, and the multi-layer perceptron in the second adaptive transformation network outputs the second first transformation parameter and the second second transformation parameter.

[0236] Take m equal to k as an example, Figure 5 As shown, the kth adaptive transformation network includes a multi-layer perceptron, the electronic device inputs the first lyrics feature and the time step feature into the multi-layer perceptron in the kth adaptive transformation network, and the multi-layer perceptron in the kth adaptive transformation network outputs the kth first transformation parameter and the kth second transformation parameter.

[0237] In step 907, the electronic device performs feature transformation on the m-1th third lyrics feature based on the m-1th first transformation parameter through the m-th adaptive transformation network to obtain the m-th second noise feature, wherein the m-1th third lyrics feature is the third lyrics feature output by the m-1th adaptive transformation network.

[0238] Take m equal to 2 as an example, Figure 5 As shown, the second adaptive transformation network includes a normalization layer and a linear transformation layer. The electronic device inputs the first third lyrics feature into the normalization layer to obtain the normalized first third lyrics feature, and inputs the normalized first third lyrics feature and the second first transformation parameter output by the multilayer perceptron in the second adaptive transformation network into the linear transformation layer. The linear transformation layer performs feature transformation on the normalized first third lyrics feature based on the second first transformation parameter, and outputs the second second noise feature.

[0239] Take m equal to k as an example, Figure 5 As shown, the k-th adaptive transformation network includes a normalization layer and a linear transformation layer. The electronic device inputs the k-1th third lyrics feature into the normalization layer to obtain the normalized k-1th third lyrics feature, and inputs the normalized k-1th third lyrics feature and the k-th first transformation parameter output by the multilayer perceptron in the k-th adaptive transformation network into the linear transformation layer. The linear transformation layer performs feature transformation on the normalized k-1th third lyrics feature based on the k-th first transformation parameter, and outputs the k-th second noise feature.

[0240] In step 908, the electronic device cross-processes the first lyrics feature and the mth second noise feature through the mth adaptive transformation network to obtain the mth second lyrics feature by fusing them.

[0241] Take m equal to 2 as an example, Figure 5 As shown, the second adaptive transformation network includes an attention layer, and the electronic device inputs the first lyrics feature and the second second noise feature output by the linear transformation layer in the second adaptive transformation network into the attention layer, and the attention layer outputs the second second lyrics feature.

[0242] Take m equal to k as an example, Figure 5 As shown, the kth adaptive transformation network includes an attention layer, and the electronic device inputs the first lyrics feature and the kth second noise feature output by the linear transformation layer in the kth adaptive transformation network into the attention layer, and the attention layer outputs the kth second lyrics feature.

[0243] In step 909, the electronic device performs feature transformation on the mth second lyrics feature through the mth adaptive transformation network based on the mth second transformation parameter to obtain the mth third lyrics feature.

[0244] Take m equal to 2 as an example, Figure 5 As shown, the second adaptive transformation network includes a scale transformation layer. The electronic device inputs the second second lyrics feature output by the attention layer in the second adaptive transformation network and the second second transformation parameter output by the multilayer perceptron in the second adaptive transformation network into the scale transformation layer. The scale transformation layer performs feature transformation on the second second lyrics feature based on the second second transformation parameter and outputs the second third lyrics feature.

[0245] Take m equal to k as an example, Figure 5 As shown, the k-th adaptive transformation network includes a scale transformation layer. The electronic device inputs the k-th second lyrics feature output by the attention layer in the k-th adaptive transformation network and the k-th second transformation parameter output by the multilayer perceptron in the k-th adaptive transformation network into the scale transformation layer. The scale transformation layer performs feature transformation on the k-th second lyrics feature based on the k-th second transformation parameter and outputs the k-th third lyrics feature.

[0246] In step 910, the electronic device decodes the kth third lyrics feature through a decoding network to obtain a target song.

[0247] The electronic device obtains the kth third lyrics feature, which is the third lyrics feature output by the kth adaptive network, that is, the third lyrics feature output by the last adaptive network. The electronic device inputs the kth third lyrics feature into the decoding network to obtain the target song output by the decoding network, and the target song matches the lyrics text.

[0248] The method provided by the embodiment of the present disclosure introduces randomly generated noise features when intelligently generating a matching target song through lyrics text. The lyrics text provides semantic information for the generation process of the target song, and the random noise features provide the potential ability to generate songs with different rhythms. By transforming the noise features according to the transformation parameters generated by the lyrics features, the rhythm of the song can be flexibly adjusted according to the lyrics text without relying on the knowledge of the musical score, thereby improving the convenience of generating songs. In addition, the present disclosure combines the two text-guiding mechanisms of feature cross-processing and dynamic adjustment of transformation parameters, which can more comprehensively capture and utilize the lyrics features, which is conducive to ensuring the quality of the generated songs.

[0249] Furthermore, through the layer-by-layer processing of k adaptive transformation networks, the first lyrics feature and the noise feature are transformed at multiple levels, the feature expression is gradually refined, and irrelevant information in the noise feature is gradually eliminated, while the features useful for generating the target song are retained and enhanced.

[0250] Moreover, the first lyric feature is gradually cross-processed with noise features at different levels through a multi-layer adaptive transformation network, and the feature fusion effect is optimized layer by layer, which helps to retain the semantic and rhythmic features of the lyrics. The cross-processing operation of each layer can capture the deeper relationship between the lyrics features and the noise features, enriching the expressive power of the second lyric feature.

[0251] Moreover, through a multi-layer adaptive transformation network, the second lyrics feature obtained after the cross processing is transformed according to the second transformation parameters generated by each layer to obtain the transformed third lyrics feature, and the feature transformation effect is optimized layer by layer, which can effectively extract the potential features of the lyrics, so that the transformed third lyrics feature can more accurately represent the emotion, rhythm and other information of the lyrics text. The last third lyrics feature output by the last layer of the adaptive transformation network is used as a high-quality input of the decoding network, which is conducive to improving the matching degree between the decoded target song and the lyrics text.

[0252] Moreover, since the number of adaptive transformation networks can be flexibly adjusted, the complexity of the generation process can be controlled, thereby significantly improving the performance of the model. The multi-level network structure supports flexible adjustment for different application scenarios, adapting to more song styles and specific task requirements, and broadening the application scope of the model.

[0253] The training process of the first song generation model in the above embodiment can be seen in the following Fig.10 Embodiment of the invention. Fig.10 is a flowchart of a method for training a song generation model according to an exemplary embodiment, the method being executed by an electronic device, see Fig.10 , the method comprises the following steps:

[0254] In step 1001, the electronic device obtains a sample lyric text and a sample song that matches the sample lyric text.

[0255] The matching of the sample song and the sample lyrics text means that the lyrics in the sample song are the sample lyrics text.

[0256] In step 1002, the electronic device extracts features of the sample lyrics text by using the second song generation model to obtain first sample lyrics features.

[0257] In step 1003, the electronic device maps the first sample lyrics feature into a first sample transformation parameter through a second song generation model.

[0258] The first sample transformation parameter is used to transform the first sample noise feature randomly generated in step 1004. In some embodiments, the electronic device generates the first sample transformation parameter and the second sample transformation parameter based on the first sample lyrics feature through a second song generation model.

[0259] In step 1004, the electronic device uses the second song generation model to perform feature transformation on the first sample noise feature based on the first sample transformation parameter to obtain a second sample noise feature, where the first sample noise feature is a randomly generated noise feature.

[0260] The process of generating the first sample lyrics feature and the second sample noise feature in the above steps 1002 to 1005 is the same as the process of generating the first lyrics feature and the second noise feature in the above steps 601 to 604, and will not be repeated here.

[0261] In step 1005, the electronic device encodes the sample song through the second song generation model to obtain target sample lyrics features.

[0262] In the disclosed embodiment, the electronic device encodes the sample song through the second song generation model to obtain the target sample lyrics feature. The target sample lyrics feature is the real feature of the sample song.

[0263] In step 1006, the electronic device trains the second song generation model based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature to obtain a first song generation model, and the first song generation model is used to generate a song that matches any lyrics text.

[0264] The first sample lyrics feature and the second sample noise feature are used to generate a song that matches the sample lyrics text, and the target sample lyrics feature is the real feature of the song that matches the sample lyrics text. In other words, the target sample lyrics feature can be used to accurately generate a song that matches the sample lyrics text. Therefore, based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature, the second song generation model is trained, so that the trained first song generation model can generate more accurate first sample lyrics features and second sample noise features, so as to use more accurate first sample lyrics features and second sample noise features to generate a song that better matches the sample lyrics text, thereby ensuring the accuracy of the trained first song generation model.

[0265] In the disclosed embodiment, when a matching target song is intelligently generated through lyrics text, randomly generated noise features are introduced. The lyrics text provides semantic information for the generation process of the target song, and the random noise features provide the potential ability to generate songs with different rhythms. By transforming the noise features according to the transformation parameters generated according to the lyrics features, the rhythm of the song can be flexibly adjusted according to the lyrics text without relying on knowledge of musical scores, thereby improving the generation capability of the song generation model.

[0266] In the above Fig.10 Based on the example, Fig.11 As shown, step 1006 includes the following steps 1016-1036.

[0267] In step 1016, the electronic device uses the second song generation model to cross-process the first sample lyrics feature and the second sample noise feature to obtain the second sample lyrics feature.

[0268] Among them, step 1016 is the same as the above step 604 and will not be repeated here.

[0269] In step 1026, the electronic device determines a loss parameter based on the second sample lyrics feature and the target sample lyrics feature.

[0270] The target sample lyrics feature is the real feature of the sample song, and the second sample lyrics feature is obtained by the second song generation model. The smaller the difference between the second sample lyrics feature generated by the song generation model and the target sample lyrics feature, the smaller the difference between the song decoded based on the second sample lyrics feature and the sample song, that is, the more accurate the song generation model is. Therefore, based on the second sample lyrics feature and the target sample lyrics feature, the second song generation model is trained, and the training goal is to reduce the difference between the second sample lyrics feature and the target sample lyrics feature, thereby improving the accuracy of the model. In the disclosed embodiment, the electronic device determines the loss parameter for training the model based on the second sample lyrics feature and the target sample lyrics feature. Exemplarily, the electronic device determines the loss parameter based on the difference between the second sample lyrics feature and the target sample lyrics feature, and the loss parameter is positively correlated with the difference, so the training goal is to reduce the loss parameter.

[0271] In some embodiments, as described in step 1003 above, the second song generation model also generates second sample transformation parameters, then step 1026 includes: the electronic device uses the second song generation model to perform feature transformation on the second sample lyrics features based on the second sample transformation parameters to obtain third sample lyrics features; based on the third sample lyrics features and the target sample lyrics features, determines the loss parameters.

[0272] Among them, the process of generating the third sample lyrics feature is the same as the process of generating the third lyrics feature in the above step 605, and will not be repeated here. The third sample lyrics feature is used to decode and obtain a song that matches the sample lyrics text. Therefore, the electronic device trains the second song generation model based on the third sample lyrics feature and the target sample lyrics feature. The training goal is to reduce the difference between the third sample lyrics feature and the target sample lyrics feature, thereby improving the accuracy of the model. In the disclosed embodiment, the electronic device determines the loss parameter for training the model based on the third sample lyrics feature and the target sample lyrics feature. Exemplarily, the electronic device determines the loss parameter based on the difference between the third sample lyrics feature and the target sample lyrics feature. The loss parameter is positively correlated with the difference, so the training goal is to reduce the loss parameter.

[0273] In the disclosed embodiment, the second sample lyrics feature obtained after cross processing is transformed according to the sample transformation parameters generated by the first sample lyrics feature, which can effectively extract the potential features of the sample lyrics, so that the transformed third sample lyrics feature can more accurately represent the emotion, rhythm and other information of the sample lyrics text, which is beneficial to improving the accuracy of the third sample lyrics feature and accelerating the model training speed.

[0274] In step 1036, the electronic device trains the second song generation model based on the loss parameter to obtain the first song generation model.

[0275] After determining the loss parameter, the electronic device trains the second song generation model based on the loss parameter. The training goal is to reduce the loss parameter, thereby improving the accuracy of the model and obtaining a trained first song generation model.

[0276] In some embodiments, the second song generation model includes a feature extraction network, an adaptive transformation network and an encoding network, and the step of generating the first sample lyrics feature is performed by the feature extraction network; the step of generating the first sample transformation parameter, the second sample noise feature and the second sample lyrics feature is performed by the adaptive transformation network; and the step of generating the target sample lyrics feature is performed by the encoding network.

[0277] This step 1007 includes: based on the loss parameter, adjusting the model parameters of the second song generation model to obtain the third song generation model; removing the encoding network in the third song generation model, adding the trained decoding network to the third song generation model, and obtaining the first song generation model.

[0278] Exemplarily, the encoding network in the first song generation model is a trained encoding network, so when adjusting the model parameters of the second song generation model, there is no need to adjust the model parameters of the encoding network. For example, the trained encoding network and the trained decoding network belong to the VAE model architecture, and the encoding network and the decoding network in the VAE model architecture are synchronously trained to obtain the trained encoding network and the trained decoding network.

[0279] Fig.12 is a schematic diagram of a structure of a second song generation model according to an exemplary embodiment. Fig.12 As shown, the second song generation model includes a feature extraction network, k adaptive transformation networks and an encoding network. The second song generation model is trained to obtain a third song generation model. The encoding network in the third song generation model is removed, and the trained decoding network is added to obtain Figure 4 The first song generation model is shown.

[0280] In the disclosed embodiment, by determining the loss parameter based on the difference between the lyrics features of the second sample and the lyrics features of the target sample, the gap between the model output and the target can be quantified. This approach ensures that the model can gradually approach the feature distribution of the target sample during the generation process, which helps to improve the feature learning ability of the model.

[0281] In addition, the encoding network is used to learn the characteristic distribution of songs in the model training stage, and the decoding network is used to generate songs in the model usage stage. The functions of the encoding network and the decoding network are separated, making the model more flexible and easy to expand. The network structure can be adjusted according to different needs to support different types of generation tasks, providing great flexibility for expanding new generation strategies or processing different data.

[0282] Moreover, the present disclosure combines the two text guidance mechanisms of feature cross processing and dynamic adjustment of transformation parameters, which can more comprehensively capture and utilize the lyrics features, and is conducive to ensuring the accuracy of the song generation model. Moreover, through the diversified text guidance mechanism, the song generation model can more comprehensively utilize the feature information, thereby accelerating the convergence process, greatly reducing the number of training rounds required for the song generation model to reach the optimal solution, effectively saving time and resources, and improving the efficiency of training.

[0283] Fig.13 is a block diagram of a song generation device according to an exemplary embodiment. Fig.13 , the device includes a feature extraction unit 1301 and a feature processing unit 1302.

[0284] The feature extraction unit 1301 is configured to perform feature extraction on the lyrics text to obtain a first lyrics feature;

[0285] The feature processing unit 1302 is configured to perform mapping of the first lyrics feature into a first transformation parameter;

[0286] The feature processing unit 1302 is further configured to perform feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature, where the first noise feature is a randomly generated noise feature;

[0287] The feature processing unit 1302 is further configured to generate a target song that matches the lyrics text based on the first lyrics feature and the second noise feature.

[0288] The device provided by the embodiment of the present disclosure introduces randomly generated noise features when intelligently generating a matching target song through lyrics text. The lyrics text provides semantic information for the generation process of the target song, and the random noise features provide the potential ability to generate songs with different rhythms. By transforming the noise features according to the transformation parameters generated according to the lyrics features, the rhythm of the song can be flexibly adjusted according to the lyrics text without relying on the knowledge of the musical score, thereby improving the convenience of generating songs. By dynamically adjusting the transformation parameters, the lyrics features can be more comprehensively captured and utilized, which is conducive to ensuring the quality of the generated songs.

[0289] In one possible implementation, see Fig.13, the feature processing unit 1302 is configured to perform:

[0290] Cross-processing the first lyrics feature and the second noise feature to obtain a second lyrics feature;

[0291] Based on the second lyrics feature, a target song matching the lyrics text is generated.

[0292] In one possible implementation, see Fig.13 , a feature processing unit 1302, configured to perform mapping of the first lyrics feature into a first transformation parameter and a second transformation parameter;

[0293] The feature processing unit 1302 is further configured to perform feature transformation on the second lyrics feature based on the second transformation parameter to obtain a third lyrics feature;

[0294] The song generation unit 1303 is configured to decode the third lyrics feature to obtain a target song that matches the lyrics text.

[0295] In one possible implementation, see Fig.13 The feature processing unit 1302 is further configured to perform:

[0296] Determine a current time step feature, where the time step feature represents a position of a current audio frame to be generated in a target song, where the target song includes multiple audio frames;

[0297] Fusing the first lyrics feature with the time step feature to obtain a first fused feature;

[0298] The first fused feature is mapped to a first transformation parameter, or the first fused feature is mapped to a first transformation parameter and a second transformation parameter.

[0299] In one possible implementation, see Fig.13 , the feature extraction unit 1301 is configured to perform:

[0300] Converting the lyrics text into a phoneme sequence, where the phoneme sequence includes the phonemes of multiple words in the lyrics text;

[0301] Feature extraction is performed on the phoneme sequence to obtain the first lyrics feature.

[0302] In one possible implementation, see Fig.13 , the feature processing unit 1302 is configured to perform:

[0303] Acquire song attribute features of a reference song, where the song attribute features include at least one of a song style feature or a song duration feature;

[0304] The first lyrics feature and the song attribute feature are fused to obtain a second fused feature;

[0305] Cross-processing the second fusion feature and the second noise feature to obtain a second lyrics feature;

[0306] Among them, the song attributes of the target song are consistent with the song attributes of the reference song.

[0307] In one possible implementation, see Fig.13 , the feature processing unit 1302 is configured to perform:

[0308] Using the first mapping matrix, perform feature mapping on the second fused feature to obtain a query feature;

[0309] Using the second mapping matrix, feature mapping is performed on the second noise feature to obtain a key feature and a value feature;

[0310] The query feature, key feature and value feature are fused to obtain the second lyrics feature.

[0311] In a possible implementation, the number of adaptive transformation networks is k, and the k adaptive transformation networks are connected in sequence; the feature processing unit 1302 is configured to perform:

[0312] Mapping the first lyrics features into k first transformation parameters respectively through k adaptive transformation networks;

[0313] The feature processing unit 1302 is further configured to execute:

[0314] By using a first adaptive transformation network, based on a first first transformation parameter, the first noise feature is transformed to obtain a first second noise feature;

[0315] Through the mth adaptive transformation network, based on the mth first transformation parameter, the m-1th third lyrics feature is transformed to obtain the mth second noise feature; wherein the m-1th third lyrics feature is the third lyrics feature output by the m-1th adaptive transformation network, and m is an integer greater than 1 and not greater than k.

[0316] In one possible implementation, see Fig.13 , the feature processing unit 1302 is configured to perform:

[0317] Through the first adaptive transformation network, the first lyrics feature and the first second noise feature are cross-processed to obtain the first second lyrics feature by fusion;

[0318] Through the mth adaptive transformation network, the first lyrics feature and the mth second noise feature are cross-processed and fused to obtain the mth second lyrics feature.

[0319] In one possible implementation, see Fig.13 , the k adaptive transformation networks also generate k second transformation parameters; the feature processing unit 1302 is configured to perform feature transformation on the first second lyrics feature based on the first second transformation parameter through the first adaptive transformation network to obtain the first third lyrics feature;

[0320] The feature processing unit 1302 is further configured to perform feature transformation on the mth second lyrics feature based on the mth second transformation parameter through the mth adaptive transformation network to obtain the mth third lyrics feature;

[0321] The feature processing unit 1302 is further configured to decode the kth third lyrics feature through a decoding network to obtain a target song.

[0322] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0323] Fig.14 is a block diagram of a training device for a song generation model according to an exemplary embodiment. Fig.14 The device includes an acquisition unit 1401, a feature extraction unit 1402, a feature processing unit 1403 and a model training unit 1404.

[0324] The acquisition unit 1401 is configured to acquire the sample lyrics text and the sample song matching the sample lyrics text;

[0325] The feature extraction unit 1402 is configured to perform feature extraction on the sample lyrics text through the second song generation model to obtain first sample lyrics features;

[0326] The feature processing unit 1403 is configured to execute mapping of the first sample lyrics feature into a first sample transformation parameter through a second song generation model;

[0327] The feature processing unit 1403 is further configured to perform feature transformation on the first sample noise feature based on the first sample transformation parameter through the second song generation model to obtain a second sample noise feature, where the first sample noise feature is a randomly generated noise feature;

[0328] The feature processing unit 1403 is further configured to perform encoding of the sample song through the second song generation model to obtain target sample lyrics features;

[0329] The model training unit 1404 is configured to train the second song generation model based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature to obtain the first song generation model, and the first song generation model is used to generate a song that matches any lyrics text.

[0330] In one possible implementation, see Fig.14 , the model training unit 1404 is configured to perform:

[0331] Through the second song generation model, the first sample lyrics feature and the second sample noise feature are cross-processed to obtain the second sample lyrics feature;

[0332] Determining a loss parameter based on the second sample lyrics feature and the target sample lyrics feature;

[0333] Based on the loss parameter, the second song generation model is trained to obtain the first song generation model.

[0334] In one possible implementation, see Fig.14 The second song generation model includes a feature extraction network, an adaptive transformation network and an encoding network, wherein the step of generating the first sample lyrics feature is performed by the feature extraction network; the step of generating the first sample transformation parameter, the second sample noise feature and the second sample lyrics feature is performed by the adaptive transformation network; the step of generating the target sample lyrics feature is performed by the encoding network;

[0335] The model training unit 1404 is configured to perform:

[0336] Based on the loss parameter, adjusting the model parameters of the second song generation model to obtain a third song generation model;

[0337] The encoding network in the third song generation model is removed, and the trained decoding network is added to the third song generation model to obtain the first song generation model.

[0338] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0339] The embodiment of the present disclosure provides an electronic device, which includes: a processor, and a memory for storing instructions executable by the processor. The processor is configured to execute the instructions to implement the song generation method or the song generation model training method as described above.

[0340] In some embodiments, the electronic device includes a terminal. Fig.151 is a block diagram of a terminal 1500 according to an exemplary embodiment. The terminal 1500 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 1500 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0341] The terminal 1500 includes a processor 1501 and a memory 1502 .

[0342] The processor 1501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. The memory 1502 may include one or more computer-readable storage media, which may be non-transitory. The memory 1502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1502 is used to store at least one program code, which is used to be executed by the processor 1501 to implement the song generation method or the song generation model training method provided in the method embodiment of the present disclosure.

[0343] In some embodiments, the terminal 1500 may further optionally include: a peripheral device interface 1503 and at least one peripheral device. The processor 1501, the memory 1502 and the peripheral device interface 1503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1503 via a bus, a signal line or a circuit board. The peripheral device includes: at least one of a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507 and a power supply 1508.

[0344] The peripheral device interface 1503 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1501 and the memory 1502. In some embodiments, the processor 1501, the memory 1502, and the peripheral device interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1501, the memory 1502, and the peripheral device interface 1503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0345] The radio frequency circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1504 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1504 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 1504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1504 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0346] The display screen 1505 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1505 is a touch display screen, the display screen 1505 also has the ability to collect touch signals on the surface or above the surface of the display screen 1505. The touch signal can be input to the processor 1501 as a control signal for processing. At this time, the display screen 1505 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1505 can be one, arranged on the front panel of the electronic device 1500; in other embodiments, the display screen 1505 can be at least two, respectively arranged on different surfaces of the electronic device 1500 or in a folding design; in other embodiments, the display screen 1505 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device 1500. Even, the display screen 1505 can also be arranged as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1505 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0347] The camera assembly 1506 is used to capture images or videos. Optionally, the camera assembly 1506 includes a front camera and a rear camera. The front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1506 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0348] The audio circuit 1507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1501 for processing, or input them into the radio frequency circuit 1504 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1500. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1507 may also include a headphone jack.

[0349] The power supply 1508 is used to power various components in the electronic device 1500. The power supply 1508 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1508 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0350] Those skilled in the art will understand that Fig.15 The structure shown in the figure does not constitute a limitation on the terminal 1500, and the terminal 1500 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0351] In some embodiments, the electronic device includes a server. Fig.16 This is a schematic diagram of the structure of a server according to an exemplary embodiment. The server 1600 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1601 and one or more memories 1602, wherein the memory 1602 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1601 to implement the methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0352] In an exemplary embodiment, a computer-readable storage medium is also provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the steps in the above-mentioned song generation method or song generation model training method. For example, the computer-readable storage medium can be a ROM (Read Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0353] In an exemplary embodiment, a computer program product is also provided, including a computer program, which implements the steps in the above-mentioned song generation method or song generation model training method when executed by a processor of an electronic device.

[0354] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0355] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A song generation method, characterized in that: The method comprises: Perform feature extraction on the lyrics text to obtain a first lyrics feature; Mapping the first lyrics feature into a first transformation parameter; Based on the first transformation parameter, a first noise feature is transformed to obtain a second noise feature, where the first noise feature is a randomly generated noise feature; Based on the first lyrics feature and the second noise feature, a target song matching the lyrics text is generated.

2. The song generation method according to claim 1, characterized in that: The step of generating a target song matching the lyrics text based on the first lyrics feature and the second noise feature comprises: Cross-processing the first lyrics feature and the second noise feature to obtain a second lyrics feature; Based on the second lyrics feature, a target song matching the lyrics text is generated.

3. The song generation method according to claim 2, characterized in that: Mapping the first lyrics feature into a first transformation parameter includes: Mapping the first lyrics feature into the first transformation parameter and the second transformation parameter; The step of generating a target song matching the lyrics text based on the second lyrics feature includes: Based on the second transformation parameter, performing feature transformation on the second lyrics feature to obtain a third lyrics feature; The third lyrics feature is decoded to obtain a target song that matches the lyrics text.

4. The song generation method according to claim 1 or 3, characterized in that: The method further comprises: Determine a current time step feature, wherein the time step feature represents a position of a current audio frame to be generated in the target song, wherein the target song includes a plurality of audio frames; Fusing the first lyrics feature with the time step feature to obtain a first fused feature; The first fused feature is mapped to the first transformation parameter, or the first fused feature is mapped to the first transformation parameter and the second transformation parameter.

5. The song generation method according to claim 1, characterized in that: The step of extracting features from the lyrics text to obtain a first lyrics feature includes: Converting the lyrics text into a phoneme sequence, wherein the phoneme sequence includes the phonemes of a plurality of words in the lyrics text; Perform feature extraction on the phoneme sequence to obtain the first lyrics feature.

6. The song generation method according to claim 2, characterized in that: The cross-processing of the first lyrics feature and the second noise feature to obtain the second lyrics feature by fusing the first lyrics feature and the second noise feature comprises: Acquire song attribute features of a reference song, wherein the song attribute features include at least one of a song style feature or a song duration feature; Fusing the first lyrics feature and the song attribute feature to obtain a second fused feature; Cross-processing the second fusion feature and the second noise feature to obtain the second lyrics feature; The song attributes of the target song are consistent with the song attributes of the reference song.

7. The song generation method according to claim 6, characterized in that: The cross-processing of the second fusion feature and the second noise feature to obtain the second lyrics feature includes: Using the first mapping matrix, performing feature mapping on the second fused features to obtain query features; Using a second mapping matrix, performing feature mapping on the second noise feature to obtain a key feature and a value feature; The query feature, the key feature and the value feature are fused to obtain the second lyrics feature.

8. The song generation method according to claim 2, characterized in that: The step of generating the first transformation parameter, the second noise feature and the second lyrics feature is performed by an adaptive transformation network, the number of the adaptive transformation networks is k, and the k adaptive transformation networks are connected in sequence; the step of mapping the first lyrics feature to the first transformation parameter includes: Mapping the first lyrics features into k first transformation parameters respectively through k said adaptive transformation networks; The step of performing feature transformation on the first noise feature based on the first transformation parameter to obtain the second noise feature includes: Performing feature transformation on the first noise feature based on the first first transformation parameter through the first adaptive transformation network to obtain the first second noise feature; Through the mth adaptive transformation network, based on the mth first transformation parameter, feature transformation is performed on the m-1th third lyrics feature to obtain the mth second noise feature; wherein the m-1th third lyrics feature is the third lyrics feature output by the m-1th adaptive transformation network, and m is an integer greater than 1 and not greater than k.

9. The song generation method according to claim 8, characterized in that: The cross-processing of the first lyrics feature and the second noise feature to obtain the second lyrics feature by fusing the first lyrics feature and the second noise feature comprises: Through the first adaptive transformation network, the first lyrics feature and the first second noise feature are cross-processed to obtain the first second lyrics feature by fusing; The first lyrics feature and the mth second noise feature are cross-processed through the mth adaptive transformation network to obtain the mth second lyrics feature.

10. The song generation method according to claim 9, characterized in that: The k adaptive transformation networks further generate k second transformation parameters; and based on the second lyrics feature, generating a target song matching the lyrics text comprises: By using the first of the adaptive transformation networks, based on the first of the second transformation parameters, the first of the second lyrics features is subjected to feature transformation to obtain the first of the third lyrics features; By using the mth said adaptive transformation network, based on the mth said second transformation parameter, performing feature transformation on the mth said second lyrics feature to obtain the mth said third lyrics feature; The kth third lyrics feature is decoded through a decoding network to obtain the target song.

11. A method for training a song generation model, characterized in that: The method comprises: Acquire sample lyrics text and a sample song that matches the sample lyrics text; Using the second song generation model, feature extraction is performed on the sample lyrics text to obtain first sample lyrics features; Mapping the first sample lyrics feature into a first sample transformation parameter through the second song generation model; By using the second song generation model, based on the first sample transformation parameter, a feature transformation is performed on the first sample noise feature to obtain a second sample noise feature, wherein the first sample noise feature is a randomly generated noise feature; The sample song is encoded by the second song generation model to obtain target sample lyrics features; Based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature, the second song generation model is trained to obtain a first song generation model, and the first song generation model is used to generate a song that matches any lyrics text.

12. The method for training a song generation model according to claim 11, characterized in that: The method of training the model parameters of the second song generation model based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature to obtain the first song generation model includes: The first sample lyrics feature and the second sample noise feature are cross-processed by the second song generation model to obtain a second sample lyrics feature; Determining a loss parameter based on the second sample lyrics feature and the target sample lyrics feature; Based on the loss parameter, the second song generation model is trained to obtain the first song generation model.

13. The method for training a song generation model according to claim 12, characterized in that: The second song generation model comprises a feature extraction network, an adaptive transformation network and an encoding network, wherein the step of generating the first sample lyrics feature is performed by the feature extraction network; the step of generating the first sample transformation parameter, the second sample noise feature and the second sample lyrics feature is performed by the adaptive transformation network; and the step of generating the target sample lyrics feature is performed by the encoding network; The step of training the second song generation model based on the loss parameter to obtain the first song generation model comprises: Based on the loss parameter, adjusting the model parameters of the second song generation model to obtain a third song generation model; The encoding network in the third song generation model is removed, and the trained decoding network is added to the third song generation model to obtain the first song generation model.

14. A song generating device, characterized in that: The device comprises: A feature extraction unit is configured to perform feature extraction on the lyrics text to obtain a first lyrics feature; A feature processing unit, configured to perform mapping of the first lyrics feature into a first transformation parameter; The feature processing unit is further configured to perform feature transformation on the first noise feature based on the first transformation parameter to obtain a second noise feature, wherein the first noise feature is a randomly generated noise feature; The feature processing unit is further configured to generate a target song matching the lyrics text based on the first lyrics feature and the second noise feature.

15. A training device for a song generation model, characterized in that: The device comprises: An acquisition unit is configured to acquire a sample lyrics text and a sample song matching the sample lyrics text; A feature extraction unit is configured to perform feature extraction on the sample lyrics text through a second song generation model to obtain first sample lyrics features; a feature processing unit configured to execute mapping of the first sample lyrics feature into a first sample transformation parameter through the second song generation model; The feature processing unit is further configured to perform feature transformation on the first sample noise feature based on the first sample transformation parameter through the second song generation model to obtain a second sample noise feature, wherein the first sample noise feature is a randomly generated noise feature; The feature processing unit is further configured to perform encoding of the sample song through the second song generation model to obtain target sample lyrics features; The model training unit is configured to train the second song generation model based on the first sample lyrics feature, the second sample noise feature and the target sample lyrics feature to obtain a first song generation model, wherein the first song generation model is used to generate a song that matches any lyrics text.

16. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the song generation method as described in any one of claims 1 to 10, or to implement the training method of the song generation model as described in any one of claims 11 to 13.

17. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the song generation method as described in any one of claims 1 to 10, or to execute the song generation model training method as described in any one of claims 11 to 13.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the song generation method described in any one of claims 1 to 10, or implements the training method of the song generation model described in any one of claims 11 to 13.

Images