Song generation method, song generation model training method and corresponding device

Abstract

The embodiment of the invention discloses a song generation method, a song generation model training method and a corresponding device. The song generation method comprises the steps that input data are acquired, and the input data comprise lyrics and global description information; generating description information of the plurality of song paragraphs based on the lyrics and the global description information; respectively encoding the lyrics, the global description information and the description information of the plurality of song paragraphs to obtain the feature representation of the lyrics, the feature representation of the global description information and the feature representation of the description information of the plurality of song paragraphs; obtaining a description feature representation based on the feature representation of the global description information and the feature representation of the description information of the multiple song paragraphs; and inputting the description feature representation and the feature representation of the lyrics as guide conditions into a song generation model, and generating a song by the song generation model based on the guide conditions, the song generation model being implemented based on a diffusion model. According to the invention, fine control of songs can be realized, and the song generation effect is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a song generation method, a training method for a song generation model, and a corresponding device. Background Technology

[0002] Songs, as a vehicle for deep emotional expression, are loved by most users, and a considerable number of users have a passion for creating them. Traditional songwriting requires professional knowledge of music theory and audio signal processing, making it a high-barrier-to-entry process. The long creation cycle from lyrics to music limits the creative opportunities for the general public. In recent years, song generation technology based on deep learning algorithms has achieved significant breakthroughs, enabling the generation of complete works including vocals and accompaniment in a short time using models. This technology is profoundly reshaping the traditional songwriting paradigm, greatly lowering the entry barrier for song creation, and providing professional music practitioners with efficient tools for innovation.

[0003] Although some song generation technologies are now available that can automatically generate songs after users input lyrics and global description text, they can only achieve global, coarse-grained song control, and the generation effect still needs to be improved. Summary of the Invention

[0004] In view of this, this application provides a song generation method, a song generation model training method, and a corresponding device to achieve fine-grained control of songs and improve the song generation effect.

[0005] This application provides the following solution: Firstly, a song generation method is provided, comprising: acquiring input data, including lyrics and global descriptive information; generating descriptive information for multiple song segments based on the lyrics and global descriptive information; encoding the lyrics, global descriptive information, and descriptive information for multiple song segments respectively to obtain feature representations of the lyrics, global descriptive information, and descriptive information for multiple song segments; obtaining descriptive feature representations based on the feature representations of the global descriptive information and the descriptive information for multiple song segments; and inputting the descriptive feature representations and the feature representations of the lyrics as guiding conditions into a song generation model, wherein the song generation model generates a song based on the guiding conditions, and the song generation model is implemented based on a diffusion model.

[0006] Optionally, based on lyrics and global description information, multiple song segment description information are generated, including: generating a first prompt instruction based on lyrics and global description information, the first prompt instruction being used to prompt the large language model to generate song segment description information, inputting the first prompt instruction into the large language model, and obtaining the description information of multiple song segments generated by the large language model based on the first prompt instruction; or, generating a second prompt instruction based on lyrics, global description information, and user-specified segment description categories, the second prompt instruction being used to prompt the large language model to generate song segment description information of the specified segment description category, inputting the second prompt instruction into the large language model, and obtaining the description information of multiple song segments generated by the large language model based on the second prompt instruction.

[0007] Optionally, based on the feature representation of global descriptive information and the feature representation of descriptive information of multiple song segments, a descriptive feature representation is obtained, including: aligning the feature representation of global descriptive information and the feature representation of descriptive information of multiple song segments with the song duration to obtain a global temporal feature representation and multiple segment temporal feature representations respectively; and fusing the global temporal feature representation and the multiple segment temporal feature representations to obtain the descriptive feature representation.

[0008] Optionally, the feature representations of the global description information and the feature representations of the description information of multiple song segments are aligned with the song duration to obtain the global temporal feature representation and the temporal feature representations of multiple segments, respectively. This includes: copying the feature representation of the global description information onto the song duration to obtain the global temporal feature representation; and copying the feature representations of the description information of multiple song segments into their respective segment duration windows to obtain the multiple segment temporal feature representations.

[0009] Optionally, the global temporal feature representation and multiple paragraph temporal feature representations are fused to obtain a descriptive feature representation, including: concatenating the global temporal feature representation and multiple paragraph temporal feature representations; and performing a linear projection transformation on the concatenated feature representation to obtain the descriptive feature representation.

[0010] Optionally, before encoding the lyrics, global description information, and description information of multiple song segments respectively, the method further includes: returning the description information of multiple song segments to the user, obtaining the description information of multiple song segments edited by the user; and encoding the description information of multiple song segments, including: encoding the description information of multiple song segments edited by the user.

[0011] Optionally, before encoding the lyrics, the process further includes: predicting the duration based on the lyrics, global description information, and description information of multiple song segments; generating timestamp information for the lyrics; returning the timestamp information for the lyrics to the user; and obtaining the timestamp information edited by the user. Encoding the lyrics includes: encoding the lyrics and their timestamp information to obtain a feature representation of the lyrics.

[0012] Secondly, a training method for a song generation model is provided, comprising: acquiring first training data including multiple training samples, wherein the training samples include song samples and their corresponding lyrics, global description information, and description information of multiple song segments; training a song generation model using the first training data, wherein the song generation model is implemented based on a diffusion model; wherein the training includes: encoding the lyrics, global description information, and description information of multiple song segments respectively to obtain feature representations of the lyrics, feature representations of the global description information, and feature representations of the description information of multiple song segments; obtaining a description feature representation based on the feature representations of the global description information and the feature representations of the description information of multiple song segments; inputting the description feature representation and the feature representation of the lyrics as guiding conditions into the song generation model, wherein the song generation model generates a song based on the guiding conditions; and updating the parameters of the song generation model using a preset loss function.

[0013] Optionally, obtaining the first training data, which includes multiple training samples, includes: obtaining multiple song samples, and for each song sample, performing the following steps: separating human voices from the song samples and performing speech recognition on the human voices to obtain the lyrics corresponding to the song samples; using a music understanding model to obtain the understood content of the song samples, and obtaining the global description information corresponding to the song samples based on the understood content; using a segmentation model to segment the song samples, and inputting the segmented multiple song segments into the music understanding model to obtain the description information of the multiple song segments.

[0014] Optionally, multiple song samples are obtained, including: obtaining multiple candidate songs; performing at least one of the following filtering processes on the multiple candidate songs to obtain multiple song samples: detecting the time ratio of music and vocals in the candidate songs, and filtering out candidate songs whose time ratio is lower than a preset ratio threshold; filtering out candidate songs whose audio metadata does not meet a preset first requirement based on the audio metadata of the candidate songs; and using an audio quality assessment model to assess the quality of the candidate songs, and filtering out candidate songs whose quality does not meet a preset first quality requirement.

[0015] Optionally, the process of separating human voices from song samples and performing speech recognition on the human voices to obtain the lyrics corresponding to the song samples includes: separating human voices from song samples and performing speech recognition on the human voices; if there is no lyrics file for the song samples, then the result of speech recognition is used as the lyrics corresponding to the lyrics samples; if there is a lyrics file for the lyrics samples, then after deleting the song meta-information in the lyrics file, the similarity between the lyrics file and the speech recognition result is calculated; if the similarity is greater than or equal to a preset similarity threshold, then the lyrics corresponding to the song samples are obtained based on the lyrics file; if the similarity is less than the similarity threshold, then the song samples are filtered out.

[0016] Optionally, training a song generation model using the first training data includes: pre-training the song generation model using the first training data; selecting candidate songs whose audio metadata meets the second requirement and / or whose quality meets the second quality requirement from the training data to obtain second training data, wherein the second requirement and the second quality requirement are higher than the first requirement and the first quality requirement used when obtaining the first training data; fine-tuning the pre-trained song generation model using the second training data; and performing reinforcement learning on the fine-tuned song generation model to obtain a song generation model used to generate songs.

[0017] Optionally, reinforcement learning is performed on the song generation model obtained through fine-tuning training, including: using the fine-tuned song generation model to generate multiple songs for the same lyrics; constructing multiple sample pairs using the multiple songs, each sample pair including positive and negative samples, where positive samples are selected from the multiple songs whose quality is greater than or equal to a preset first quality threshold, and negative samples are selected from the multiple songs whose quality is less than or equal to a preset second quality threshold, where the first quality threshold is greater than the second quality threshold, and the quality difference between positive and negative samples in the same sample pair is greater than a preset difference threshold; and using the multiple sample pairs to perform reinforcement learning on the song generation model to maximize the difference between the probability of the song generation model generating positive samples and the probability of generating negative samples.

[0018] Optionally, the loss function is a conditional flow matching loss function to minimize the difference between the vector field predicted by the song generation model and the target vector field; wherein the predicted vector field is the path from the noise latent vector to the latent vector of the song generated by the song generation model, and the target vector field is the path from the noise latent vector to the latent vector of the song sample.

[0019] Thirdly, a song generation device is provided, comprising: a data acquisition unit configured to acquire input data, including lyrics and global description information; a segment planning unit configured to generate description information for multiple song segments based on the lyrics and global description information; an encoding processing unit configured to encode the lyrics, global description information, and description information for multiple song segments respectively to obtain feature representations of the lyrics, feature representations of the global description information, and feature representations of the description information for multiple song segments; a description processing unit configured to obtain description feature representations based on the feature representations of the global description information and the feature representations of the description information for multiple song segments; and a song generation unit configured to input the description feature representations and the feature representations of the lyrics as guiding conditions into a song generation model, and generate a song based on the guiding conditions, wherein the song generation model is implemented based on a diffusion model.

[0020] Fourthly, a training device for a song generation model is provided, comprising: a sample acquisition unit configured to acquire first training data including multiple training samples, the training samples including song samples and their corresponding lyrics, global description information, and description information of multiple song segments; and a model training unit configured to train a song generation model using the first training data, the song generation model being implemented based on a diffusion model; wherein the training includes: encoding the lyrics, global description information, and description information of multiple song segments respectively to obtain feature representations of the lyrics, feature representations of the global description information, and feature representations of the description information of multiple song segments; obtaining a description feature representation based on the feature representations of the global description information and the feature representations of the description information of multiple song segments; inputting the description feature representation and the feature representation of the lyrics as guiding conditions into the song generation model, and having the song generation model generate a song based on the guiding conditions; and updating the parameters of the song generation model using a preset loss function.

[0021] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, performs the steps of the method described in the first or second aspect above.

[0022] A sixth aspect provides an electronic device comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, perform the steps of the method described in the first or second aspect above.

[0023] In a seventh aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method as described in the first or second aspect above.

[0024] According to the specific embodiments provided in this application, the following technical effects are disclosed: 1) This application can generate description information for multiple song segments based on the input lyrics and global description information. The description information of the song segments, together with the global description information and lyrics, is encoded and input as a guiding condition into the song generation model to generate a song. This not only enables global control of the song, but also enhances the ability to control the local and fine-grained aspects of the song, improves the song's sense of hierarchy and expressiveness, and optimizes the song generation effect.

[0025] 2) This application can automatically generate description information for multiple song segments based on lyrics and global description information using a large language model, or generate description information for multiple song segments with user-specified segment description categories. It makes full use of the understanding ability of the large language model, transforming the global description information and lyrics of the song into "segment-level" structural descriptions. No manual input is required from the user, resulting in a higher level of intelligence and a better user experience.

[0026] 3) This application aligns the feature representations of global description information and song segment description information with the song duration and then fuses them to obtain description features that integrate global and segment-level conditions, thereby providing global and segment-level guidance for subsequent song generation models.

[0027] 4) This application predicts the duration based on lyrics, global description information, and description information of multiple song segments to obtain the timestamp information of the lyrics, and then encodes it. This introduces the alignment paradigm of lyrics in time into the lyrics encoding, which is more conducive to controlling the rhythm and cadence of the song while reducing the error rate of lyrics.

[0028] 5) This application can return the description information of multiple generated song segments to the user for editing, allowing the user to incorporate their unique creative inspiration and refined creative needs into the description information of the song segments, and then apply it to song generation. This gives the user a greater degree of refined control over the song and a better interactive experience.

[0029] 6) When training the song generation model, this application automatically obtains the lyrics corresponding to the song samples by separating human voices from song samples and performing speech recognition. It also automatically obtains the global description information and description information of multiple song segments corresponding to the song samples by using a music understanding model and a segment segmentation model. This achieves automatic annotation of training data, reduces manual costs, and improves training efficiency.

[0030] 7) This application improves the performance and effectiveness of the song generation model by filtering candidate songs based on the time ratio of music and vocals, audio metadata, and quality, and then selecting superior song samples for training.

[0031] 8) This application uses voice separation and speech recognition, and further combines lyrics files to automatically and accurately determine the lyrics of song samples, thereby providing a training basis for the high accuracy of the song generation model.

[0032] 9) This application adopts a three-stage model training method of "pre-training-fine-tuning-reinforcement learning". First, the general capabilities of the song generation model are accumulated through pre-training, allowing the song generation model to master the underlying rules, complete the accumulation of massive knowledge and improve the model's generalization ability. Then, the song generation model is fine-tuned through high-quality samples to improve the professional capabilities of the song generation model. Finally, reinforcement learning enables the song generation model to generate songs that better meet human aesthetic needs.

[0033] 10) This application employs a conditional flow matching loss function to minimize the difference between the vector field predicted by the song generation model and the target vector field. This can efficiently model the complex distribution of songs, improve the fine-grained control of songs based on conditions, and effectively avoid collapse and gradient vanishing, resulting in stronger stability.

[0034] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a system architecture diagram applicable to the embodiments of this application.

[0037] Figure 2 A flowchart of a song generation method provided in an embodiment of this application.

[0038] Figure 3 This is a schematic diagram of the structure of the song generation system provided in an embodiment of this application.

[0039] Figure 4 A flowchart illustrating the training method for the song generation model provided in this application embodiment.

[0040] Figure 5 This is a schematic diagram of the structure of the song sample processing pipeline provided in an embodiment of this application.

[0041] Figure 6 A schematic block diagram of a song generation apparatus provided in an embodiment of this application.

[0042] Figure 7 A schematic block diagram of a training apparatus for a song generation model provided in an embodiment of this application.

[0043] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0045] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0046] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0047] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0048] While existing song generation technologies can automatically generate songs after users input lyrics and a global description, the global description often fails to fully capture the expressive shifts within the song, resulting in generated songs lacking emotional depth, limited instrument combinations, and nuanced layering. Furthermore, these technologies struggle to express the creator's nuanced creative intentions; for example, users often desire to enhance the emotion in certain parts of the song or add accompaniment in others, but such localized and refined control cannot be achieved.

[0049] This application provides a novel approach. To facilitate understanding of this application, the system architecture on which it is based will first be described. Figure 1 The system architecture diagram applicable to the embodiments of this application is as follows: Figure 1As shown, the system architecture may include: user equipment, a song generation device located on the server side, and a training device for the song generation model.

[0050] The user equipment and the server can communicate with each other. The user equipment and the server can be connected directly or indirectly through wired or wireless communication methods, which is not limited herein.

[0051] User devices can include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and personal computers (PCs). Smart mobile devices can include mobile phones, tablets, laptops, personal digital assistants (PDAs), and connected cars. Smart home devices can include smart TVs, smart refrigerators, and so on. Wearable devices can include smartwatches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (devices that support both virtual and augmented reality).

[0052] A server can be a standalone server, a server cluster, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a hosting product within the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak service scalability.

[0053] Before performing the song generation task, the training device for the song generation model can be trained using the method provided in the embodiments of this application to obtain the song generation model.

[0054] Users can input data through their user devices, which then send the input data over the network to a song generation device on the server. The song generation device uses a trained song generation model to generate a song and returns it to the user device over the network. The user device then provides the received song to the user.

[0055] Apart from Figure 1 In addition to the architecture shown, a computer terminal device with strong computing power can also use the method provided in the embodiments of this application to train the song generation model and / or generate songs.

[0056] It should be understood that Figure 1 The number of user devices, training devices for song generation models, song generation devices, and song generation models shown in the diagram is merely illustrative. Depending on implementation needs, any number of user devices, training devices for song generation models, song generation devices, and song generation models can be included.

[0057] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0058] Figure 2 This is a flowchart of a song generation method provided in an embodiment of this application. This method can be... Figure 1 The song generation device in the system shown is executed. For example... Figure 2 As shown, the method may include the following steps: Step 201: Obtain input data, which includes lyrics and global description information.

[0059] Step 202: Generate description information for multiple song segments based on lyrics and global description information.

[0060] Step 203: Encode the lyrics, global description information, and description information of multiple song segments respectively to obtain feature representations of the lyrics, global description information, and description information of multiple song segments.

[0061] Step 204: Based on the feature representation of global description information and the feature representation of description information of multiple song segments, obtain the description feature representation.

[0062] Step 205: Input the descriptive feature representation and the lyric feature representation as guiding conditions into the song generation model, and generate the song based on the guiding conditions. The song generation model is implemented based on the diffusion model.

[0063] As can be seen from the above process, this application can first generate description information for multiple song segments based on the input lyrics and global description information. The description information of the song segments, together with the global description information and lyrics, is encoded and input as a guiding condition into the song generation model to generate a song. This not only enables global control of the song, but also enhances the ability to control the local and fine-grained aspects of the song, improves the song's sense of hierarchy and expressiveness, and optimizes the song generation effect.

[0064] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments.

[0065] First, the above step 201, namely "acquiring input data, which includes lyrics and global description information", will be described in detail with reference to the embodiments.

[0066] The global description information includes at least one of the following: song style, singer's gender, timbre and song emotion, and music technical parameters.

[0067] Song styles include pop (emphasizing smooth melodies and brisk rhythms), rock (with electric guitar and drums as the core instruments, and a style that leans towards excitement), folk (with acoustic guitar and harmonica as the main instruments, and a simple and narrative style), and electronic music (relying on synthesizers and electronic sound effects, with strong and varied rhythms).

[0068] Timbre, such as brightness (e.g., clear timbre, mellow timbre, etc.) and texture characteristics (e.g., hoarse timbre, pure timbre, ethereal timbre, etc.).

[0069] Song emotion can define the overall emotional direction of the generated song. For example, positive emotions include cheerful (upbeat rhythm, mostly guitar and drums, with an upward melody), healing (slow rhythm, mostly piano and violin, with a soft melody), and exciting (strong rhythm, mostly electric guitar and bass, with a soaring melody); negative emotions include sadness (slow rhythm, mostly piano and cello, with a downward melody), and loneliness (simple instrumentation, mostly single instrument accompaniment, with a gentle melody); neutral or complex emotions include narrative (emotions are not extremely biased, focusing on storytelling, with the melody rising and falling with the lyrics) and atmosphere (emotions are subtle, relying on the atmosphere created by instruments).

[0070] Music technical parameters can determine the audio quality and technical specifications of the generated song, including at least one of the following: instrument type (such as piano, guitar, violin, etc.), tempo (which can define the rhythm speed and intensity of the generated song), scale (which can define the pitch range and modal logic of the generated song, such as natural major, natural minor, harmonic minor, etc.), meter (which can define the rhythmic pattern and strong-weak cycle of the generated song), and duration.

[0071] Optionally, music technical parameters may also include sampling rate (which determines the frequency response accuracy of the audio), number of channels (which determines the spatial sense of the audio), bit depth (which determines the dynamic range of the audio), compression rate (which determines the size of the audio file, storage efficiency, and sound quality), and so on.

[0072] The following describes step 202, namely "generating description information for multiple song segments based on lyrics and global description information," in detail with reference to an embodiment.

[0073] This step leverages the text understanding and reasoning capabilities of a large language model to plan multiple song segments based on lyrics and global descriptive information, thereby generating descriptive information for each song segment. The descriptive information for each song segment includes at least one of the following: emotional content and musical technical parameters.

[0074] One possible approach is to generate a first prompt instruction based on lyrics and global description information. This first prompt instruction is used to prompt the large language model to generate description information for song segments. The first prompt instruction is then input into the large language model, and description information for multiple song segments generated by the large language model based on the first prompt instruction is obtained.

[0075] Lyrics serve as the basis for dividing a song into segments, while global descriptive information includes at least one of the following: song style, singer's gender, timbre, song emotion, and musical technical parameters. Thus, the lyrics and global descriptive information can be used to generate a first prompt instruction, which instructs the large language model to generate descriptive information for multiple song segments.

[0076] The large language model can generate descriptive information for multiple song segments based on preset segment types. For example, it can preset five segment types: intro, verse, chorus, bridge, and outro, and generate segment descriptions for each of these five segments separately. Alternatively, the large language model can also generate segment descriptions based on its understanding of the lyrics and global descriptive information, selecting segments with confidence levels that meet the confidence requirements (i.e., important segments) based on the confidence level of each segment. For example, based on the confidence level of each segment, it can select to generate segment descriptions for the intro, verse, and outro separately.

[0077] This application utilizes a large language model to effectively expand lyrics and global descriptive information into descriptive information for multiple song segments. This provides fine-grained control information at the segment level for subsequent song generation, thereby enhancing the layering and expressiveness of the generated songs. Furthermore, this application allows the large language model to automatically generate descriptive information for multiple song segments via a first prompt command, eliminating the need for users to manually define segment structures and significantly lowering the barrier to entry for aspiring creators.

[0078] As another feasible approach, users can also define their own paragraph description categories, such as specifying emotional changes or instrument types. In this method, a second prompt instruction can be generated based on the lyrics, global description information, and the user-defined paragraph description categories. This second prompt instruction prompts the large language model to generate description information for song paragraphs of the specified paragraph description categories. The second prompt instruction is then input into the large language model, and description information for multiple song paragraphs generated by the large language model based on the second prompt instruction is obtained.

[0079] For users with music expertise, the generated songs have more personalized requirements. Therefore, users can customize the paragraph description categories, which can then generate second prompts based on lyrics, global description information, and user-specified paragraph description categories. This allows the large language model to generate descriptions of multiple song paragraphs that better meet user needs based on the second prompts, thus improving user interactivity.

[0080] In addition to the methods mentioned above, users can also specify information such as the number of paragraphs to be generated or the paragraph type in the prompt instructions.

[0081] Alternatively, a large language model can also be called a paragraph planner.

[0082] For example, assuming the user inputs lyrics and global description information, this step can output the description information for each of the five sections: beginning, verse, chorus, bridge, and ending.

[0083] The following describes in detail step 203, namely, "encoding the lyrics, global description information, and description information of multiple song segments respectively to obtain feature representations of the lyrics, global description information, and description information of multiple song segments," with reference to the embodiments.

[0084] like Figure 3 As shown, a lyrics encoder can be used to encode lyrics to obtain a high-dimensional feature representation of lyrics; a global conditional encoder can be used to encode global descriptive information to obtain a high-dimensional feature representation of global descriptive information; and a paragraph conditional encoder can be used to encode descriptive information of multiple song paragraphs to obtain a high-dimensional feature representation of descriptive information of multiple song paragraphs.

[0085] Among them, the global conditional encoder, paragraph conditional encoder and lyrics encoder can adopt a text encoder to better encode the relevant descriptive information of the singer's timbre, such as age and gender, as well as the relevant descriptive information of global attributes such as song style and song emotion from the long text description.

[0086] Optionally, if other encoders can also effectively encode fine-grained control conditions such as song style, instrument type, singer attributes, and song sensuality, then other encoders can also be used to encode lyrics, global description information, and description information of multiple song segments.

[0087] In some embodiments, before encoding the lyrics, global description information, and description information of multiple song segments respectively, the method further includes: returning the description information of multiple song segments to the user and obtaining the description information of multiple song segments edited by the user.

[0088] After generating descriptions of multiple song segments, the large language model can further enhance interactivity with users and meet their personalized needs by returning the descriptions of multiple song segments to the user before encoding the lyrics, global descriptions, and descriptions of multiple song segments separately. Users can edit (e.g., delete, modify, add, etc.) the descriptions of multiple song segments to incorporate their personalized creative inspiration and refined creative needs into the descriptions of the song segments.

[0089] In this case, the paragraph condition encoder encodes the descriptive information of multiple song paragraphs edited by the user.

[0090] After a user edits the description information of multiple song segments, a segment conditional encoder can be used to encode the edited description information of multiple song segments to obtain a high-dimensional feature representation of the description information of multiple song segments. This makes the subsequently generated songs more in line with the user's personalized needs and provides a better user experience.

[0091] In some other embodiments, duration prediction can be performed based on the lyrics, global description information, and description information of multiple song segments before encoding the lyrics, such as... Figure 3 As shown, a duration predictor is used to predict the duration based on lyrics, global description information, and description information from multiple paragraphs, generating timestamp information for the lyrics. This timestamp information can also be returned to the user for editing, allowing the user to obtain the edited timestamp information.

[0092] The timestamp information of the lyrics can be used to achieve precise synchronization between the lyrics text and the audio. By marking the corresponding audio start timestamp (optionally, the audio end timestamp can also be marked) for each stanza (or each sentence) of lyrics, it can be ensured that the corresponding lyrics can be accurately displayed when the audio is played to the corresponding moment.

[0093] In addition, users can also edit the timestamp information of the lyrics generated by the duration predictor, which can improve the efficiency of song creation, enhance the creative freedom of users, and meet their personalized creative needs.

[0094] The encoding process performed on the lyrics based on this is actually encoding the lyrics and their timestamp information to obtain a feature representation of the lyrics.

[0095] The lyrics include timestamp information, which refers to lyrics in a file format (LRC format, etc.) that includes sentence-level timestamps.

[0096] After users edit the timestamp information of the lyrics, a lyrics encoder can be used to encode the lyrics and their timestamp information (or the timestamp information edited by the user) to obtain a high-dimensional feature representation of the lyrics. During the encoding process, the temporal alignment paradigm of the lyrics can be introduced into the lyrics encoding, which is more conducive to controlling the rhythm and cadence of the song, while reducing the error rate of the lyrics, and also makes the subsequently generated songs more in line with the user's personalized needs, resulting in a better user experience.

[0097] The following describes step 204, namely "obtaining a descriptive feature representation based on the feature representation of global descriptive information and the feature representation of descriptive information of multiple song segments," in detail with reference to the embodiments.

[0098] Since the feature representations of global description information and the feature representations of description information of multiple song segments are inconsistent with the song duration in the time dimension, in this step, we can first align the feature representations of global description information and the feature representations of description information of multiple song segments with the song duration to obtain global temporal feature representation and multiple segment temporal feature representation respectively; then we fuse the global temporal feature representation and the multiple segment temporal feature representation to obtain the description feature representation.

[0099] By aligning the feature representations of global descriptive information and the feature representations of descriptive information from multiple song segments with the song duration and then fusing them, a descriptive feature that integrates global and segment-level conditions can be obtained. This provides global and segment-level guidance for the subsequent song generation model, enabling not only global control over the generated song but also enhanced local and refined control capabilities, improving the song's hierarchy and expressiveness, and optimizing the overall song generation effect.

[0100] Specifically, when aligning the feature representation of global descriptive information with the song duration, the feature representation of global descriptive information can be copied onto the song duration to obtain a global temporal feature representation. However, when aligning the feature representations of descriptive information for multiple song segments with the song duration, the feature representations of descriptive information for each song segment can be copied into their respective segment duration windows to obtain multiple segment temporal feature representations.

[0101] For the global descriptive information, after encoding it through a global conditional encoder, a feature representation of the global descriptive information can be obtained. This feature representation is then replicated across the song's duration (i.e., broadcast to every time frame within the song's duration), resulting in a global temporal feature representation. Since the global descriptive information defines the overall tone of the song and needs to remain effective across all time frames, replicating its feature representation across the song's duration ensures that the song generation model is aware of global attribute constraints when generating audio for each time frame. This prevents local time frame audio from deviating from the overall style, thus guaranteeing the overall style of the generated song.

[0102] For the description information of multiple song segments, each segment's description information is encoded by a segment conditional encoder to obtain the feature information of each segment's description information. The feature information of each segment's description information is then copied (or broadcast) within its corresponding segment duration window, thus transmitting the feature information of each segment's description information to every time frame within the corresponding segment duration window, resulting in multiple segment temporal feature representations. Since the description information of a song segment can define the local dynamic attributes of that segment, it only needs to be effective within the segment duration window of that segment. This allows for the copying of the feature representations of multiple song segments' description information within their respective segment duration windows, resulting in multiple segment temporal feature representations. This enables fine-grained control at the segment level, enhancing the layering and expressiveness of the generated song.

[0103] Furthermore, by copying the feature representation of global descriptive information onto the song duration to obtain global temporal feature representation, and by copying the feature representation of descriptive information of multiple song segments into their respective segment duration windows to obtain multiple segment temporal feature representations, this lightweight operation can also reduce the computational burden of the song generation model and ensure the efficiency of the song generation model in generating songs.

[0104] As one possible approach, when fusing global temporal feature representations and multiple paragraph temporal feature representations, the global temporal feature representations and multiple paragraph temporal feature representations can be concatenated; the concatenated feature representations are then transformed by linear projection to obtain descriptive feature representations.

[0105] For example, a projection layer consisting of three linear projection matrices transforms the stitched feature representation to obtain a descriptive feature representation, which can be denoted as... .in, E text Characterization describes feature representation;T The sequence length of the latent vectors representing the subsequent song generation model can be determined by the total duration of the song. d text The text feature dimensions used to represent global descriptive information and descriptive information of song segments.

[0106] Optionally, when using a lyrics encoder to encode the lyrics and their timestamp information, if the timestamp duration is appropriate, the final output of the lyrics encoder will also be of length [length missing]. T The sequence can be denoted as... ,in, E lyrics The characteristics of lyrics are represented. d lyrics This represents the dimension of the text features used in the lyrics.

[0107] By effectively fusing global temporal feature representations and multiple segment temporal feature representations, both global and local control information can be preserved. This complete preservation of multi-granular control information avoids information loss and optimizes the song generation effect of the song generation model.

[0108] By performing a linear projection transformation on the concatenated feature representations, the high-dimensional concatenated feature representations can be mapped to the same dimensional space, thereby obtaining a dimensionally unified descriptive feature representation. This ensures that the input data of the song generation model is dimensionally unified, avoiding the problem that the song generation model cannot generate songs or the generated songs are of poor quality due to incompatible input data dimensions. It can also reduce the computational burden of the song generation model.

[0109] The following describes step 205, namely, "inputting the descriptive feature representation and the feature representation of the lyrics as guiding conditions into the song generation model, and generating a song based on the guiding conditions, wherein the song generation model is based on a diffusion model," in detail with reference to the embodiments.

[0110] like Figure 3 As shown, the descriptive feature representation and the lyric feature representation can be used as guiding conditions. The latent vector of the noisy audio is input into the song generation model. The song generation model performs denoising processing on the latent vector of the noisy audio at multiple time steps based on the guiding conditions. Finally, the dependent variable is decoded by the decoder to obtain the song.

[0111] Figure 4 This is a flowchart illustrating the training method for the song generation model provided in this application embodiment. This method can be... Figure 1 The training device for the song generation model in the system shown is executed. For example... Figure 4 As shown, the method may include the following steps: Step 401: Obtain first training data including multiple training samples. The training samples include song samples and their corresponding lyrics, global description information, and description information of multiple song segments.

[0112] Step 402: Train a song generation model using the first training data. The song generation model is based on a diffusion model. The training includes: encoding the lyrics, global description information, and description information of multiple song segments to obtain feature representations of the lyrics, global description information, and description information of multiple song segments; obtaining description feature representations based on the feature representations of the global description information and description information of multiple song segments; inputting the description feature representations and the feature representations of the lyrics as guiding conditions into the song generation model, which then generates a song based on these guiding conditions; and updating the parameters of the song generation model using a preset loss function.

[0113] As can be seen from the above process, in the training process of the song generation model, this application uses training samples containing fine-grained descriptive information of the song to train the song generation model. This enables the song training model to learn to capture the emotional fluctuations of different song segments, so that the finally trained song generation model not only has the ability to control the song globally, but also has a stronger ability to control the song locally and in a refined manner. This can enhance the sense of layering and expressiveness of the generated song and optimize the song generation effect.

[0114] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments.

[0115] First, the above step 401, namely "obtaining first training data including multiple training samples, wherein the training samples include song samples and their corresponding lyrics, global description information and description information of multiple song segments", will be described in detail with reference to the embodiments.

[0116] In this step, when acquiring training samples, song samples can be manually selected and labeled with lyrics, global description information, and description information for multiple song segments. However, this method is relatively inefficient. Therefore, as a preferred alternative, a more efficient method can be adopted, such as... Figure 5 The process shown obtains training samples, such as Figure 5 As shown, the following steps may be included: Step 501: Obtain multiple song samples.

[0117] In this embodiment, multiple candidate songs can be obtained first. These candidate songs can be obtained from publicly available data platforms on the Internet or from the backflow data of internal applications. After obtaining the candidate songs, at least one of the following filters can be applied to obtain multiple song samples: The time ratio of music to vocals in candidate songs is detected, and candidate songs with a time ratio lower than a preset threshold are filtered out. This method can filter out songs consisting solely of vocals, or songs that are nearly solely vocals.

[0118] Based on the audio metadata of candidate songs, candidate songs whose audio metadata does not meet the preset first requirement are filtered out. The audio metadata includes audio duration, sampling rate, number of channels, etc.

[0119] An audio quality assessment model is used to evaluate the quality of candidate songs, filtering out those that do not meet a preset first quality requirement, thereby improving the quality of songs generated by the trained song generation model. In this embodiment, some existing audio quality assessment models can be used, such as Audiobox Aesthetics and SongEval.

[0120] In this embodiment, the filtering criteria vary depending on the training stage. During the pre-training stage of the song generation model, the requirements for each metric are lower, aiming to include as many song samples as possible. However, during the fine-tuning stage, the quality requirements for the song samples are higher. For example, in the pre-training stage, the sampling rate threshold is set to 32 kHz, while in the fine-tuning stage, the threshold is increased to 44 kHz.

[0121] Then, for each song sample, perform the following steps S12~S14 respectively: Step 502: Separate the human voice from the song sample and perform speech recognition on the human voice to obtain the lyrics corresponding to the song sample.

[0122] In the lyrics preprocessing process, lyrics transcription (i.e., obtaining lyrics through speech recognition of human voice) is necessary for song samples that do not contain lyrics information. After performing voice-accompaniment separation on the song samples, automatic speech recognition can be performed on the human voice track (optionally, different models can be used for lyrics transcription for song samples in different languages).

[0123] If the song sample does not include an LRC format lyrics file, the speech recognition result can be directly used as the lyrics corresponding to the lyrics sample.

[0124] If the song sample includes an LRC format lyrics file, a large language model can be used to initially clean these LRC lyrics files, removing song metadata. Then, the similarity between the lyrics file and the speech recognition result is calculated, for example, by calculating the edit distance between them. A larger edit distance indicates a smaller similarity between the lyrics file and the speech recognition result; a smaller edit distance indicates a larger similarity. Therefore, when the similarity between the lyrics file and the speech recognition result is large, the lyrics corresponding to the song sample can be obtained based on the LRC format lyrics file, and song samples with excessively large edit distances are filtered out, as excessively large distances indicate that the LRC format lyrics file is inaccurate. In other words, if the similarity is greater than or equal to a preset similarity threshold, the lyrics corresponding to the song sample are obtained based on the lyrics file; if the similarity is less than the similarity threshold, the song sample is filtered out.

[0125] Step 503: Use the music understanding model to obtain the understanding content of the song sample, and obtain the global description information corresponding to the song sample based on the understanding content.

[0126] The music understanding model can employ a model such as Audio Flamingo 3, which can automatically label attributes such as style, singer characteristics, instrument type, pitch, and emotion of song samples, thereby obtaining global descriptive information of the song samples.

[0127] In addition, structured segmentation information of the song samples is required, such as "start, verse, chorus, bridging, and ending". The structured annotation and timestamp location of the songs can also be achieved through a music understanding model. Specifically, step 504 is executed: the song samples are segmented using a segmentation model, and the resulting multiple song segments are input into the music understanding model to obtain descriptive information for each segment.

[0128] The segmentation module is responsible for dividing the song samples into segments, such as "start, verse, chorus, bridging, and ending," as well as the start and end timestamps of each segment. The resulting song segments are then input into the music understanding model to obtain fine-grained descriptive information for each segment.

[0129] The following describes step 402, namely "training the song generation model using the first training data", in detail with reference to the embodiments.

[0130] The song generation model involved in this application embodiment can be implemented based on a diffusion model. During the training process, please refer to... Figure 3The lyrics, global descriptive information, and descriptive information from multiple song segments are encoded separately to obtain feature representations for the lyrics, global descriptive information, and descriptive information from multiple song segments. Based on the feature representations of the global descriptive information and the descriptive information from multiple song segments, a descriptive feature representation is obtained. The descriptive feature representation and the feature representation of the lyrics are then used as guiding conditions input into the song generation model, which generates the song based on these guiding conditions. For a detailed explanation of the above processing steps, please refer to [the documentation / reference needed for...]. Figure 2 and Figure 3 The details of the illustrated embodiment will not be repeated here. During the training process, after the song generation model generates a song, the parameters of the song generation model can be updated using a preset loss function.

[0131] As one possible approach, the loss function can be a conditional flow matching loss function to minimize the difference between the vector field predicted by the song generation model when generating songs and the target vector field; where the predicted vector field is the path from the noise latent vector to the latent vector of the song generated by the song generation model, and the target vector field is the path from the noise latent vector to the latent vector of the song sample.

[0132] The conditional flow matching loss function described above enables the song generation model to learn a conditional vector field. This approach minimizes the difference between the vector field predicted by the song generation model and the target vector field, making its vector field direction approximate the linear flow direction between the target data. This enables efficient modeling of complex song distributions, improves the model's fine-grained control over songs based on conditions, and effectively avoids crashes and gradient vanishing, resulting in greater stability. The conditional flow matching loss function can take the following form: in, Characterizes the conditional flow matching loss function; Represents the variable , Follows distribution , Follows distribution The mathematical expectation. Characterizes the distribution of noise audio; Characterize the distribution of song samples; Characterize the time step; Characterization of guiding conditions. The target vector field is the path from the noise latent vector to the latent vector of the song sample; in other words, the path from the distribution of the noise audio to the distribution of the song sample. For time steps The distribution of latent vectors.

[0133] If the vector field predicted by the song generation model The closer the direction is to the target vector field The direction, that is, the predicted vector field. The closer the direction is to the direction from noise to clear audio, the smaller the loss value. If the vector field predicted by the song generation model... The further the direction deviates from the target vector field The greater the direction of the loss, the greater the loss value.

[0134] During the training process, the song generation model needs to ensure global consistency and conditional adaptability in the denoising direction. Global consistency means that regardless of… Regardless of the noise level, the direction of the vector field predicted by the song generation model must always point to the clear audio latent variables. This prevents the song generation model from going astray during the denoising process (e.g., from moving from noise to more chaotic directions); conditional fitness represents the vector field predicted by the song generation model based on global consistency. Input must match the conditions Therefore, by optimizing the song generation model to ensure global consistency and conditional adaptability in the denoising direction, it can be ensured that the song generation model can not only gradually restore clear audio from noise during inference, but also generate songs that are more in line with human aesthetic needs in strict accordance with user preferences.

[0135] The Euler ODESolver can be used to reduce resource overhead and improve efficiency when solving the loss function, especially suitable for the need for rapid iteration during training.

[0136] Furthermore, classifier-free guidance (CFG) can be used to balance the quality (fit with guidance conditions) and diversity (avoiding monotonous results) of the songs generated by the model. This can be achieved in the following forms: in, This represents the field flow predicted by the model under unguided conditions. This refers to the predicted field flow under guided conditions, while This involves using negative guiding conditions to direct the corresponding field flow. Under negative guiding conditions, the guiding conditions corresponding to the lyric feature representation are set to empty, while both global guiding conditions and local paragraph-level guiding conditions are set to negative prompt words, thus enabling more precise, fine-grained control. and These are weighting coefficients, and after experimentation, and It can be set to 3 and 1 respectively.

[0137] In the embodiments of this application, a one-stage training method or a multi-stage training method can be adopted.

[0138] As a preferred implementation, a three-stage training method can be adopted. This mainly includes: pre-training the song generation model using first training data; selecting candidate songs whose audio metadata meets the second requirement and / or whose quality meets the second quality requirement from the training data to obtain second training data, where the second requirement and second quality requirement are higher than the first requirement and first quality requirement used when obtaining the first training data; fine-tuning the pre-trained song generation model using the second training data; and performing reinforcement learning on the fine-tuned song generation model to obtain a song generation model used to generate songs.

[0139] In the pre-training phase, candidate songs whose audio metadata meets the first requirement and / or whose quality meets the first quality requirement can be used as song samples to constitute the first training data. For example, candidate songs with a sampling rate higher than a first preset sampling rate threshold (e.g., 32kHz), candidate songs with a duration within a preset range (e.g., 30 seconds to 6 minutes), and candidate songs with an automatic quality evaluation score higher than a first preset score threshold (e.g., ranking in the top 95%). By enabling the song generation model to learn the feature representation of global descriptive information, the feature representation of lyrics, and the basic mapping relationship between latent variables, the song generation model grasps the general rules of song generation (e.g., the association between musical style and instrument combination, the matching of lyric rhyme and rhythm, the basic rendering logic of audio waveforms, etc.), thus laying the foundation for subsequent fine-grained control, reducing the difficulty of subsequent training processes, improving the training efficiency of the song generation model, and ensuring the generalization ability of the song generation model, avoiding the problem of the song generation model generating too many similar songs.

[0140] During the fine-tuning training phase, higher-quality song samples can be selected. This means candidate songs whose audio metadata meets the second requirement and / or whose quality meets the second quality requirement can be used as the first training data. Examples include candidate songs with a sampling rate higher than a second preset sampling rate threshold (e.g., 44kHz), dual-channel candidate songs, and candidate songs with an automatic quality evaluation score higher than a second preset score threshold (e.g., ranking in the top 50%). Based on the song generation model trained in the pre-training phase, the fine-tuning training phase further enhances the model's fine-grained control and high-quality song generation capabilities. This involves training the song generation model with higher-quality second training data, making its response to fine-grained guiding conditions more accurate, improving the subtlety of audio rendering, and ensuring the quality of the generated songs.

[0141] In the reinforcement learning phase, the final generation effect of the song generation model can be optimized based on human preferences. This involves scoring the quality of songs generated by the fine-tuned model and filtering sample pairs by setting a quality score threshold. In each sample pair, the difference between the quality score of the song corresponding to the positive sample and the quality score of the song corresponding to the negative sample must be greater than a preset threshold. The obtained sample pairs are then used to perform reinforcement learning on the song generation model. This involves inputting the lyrics, global description information, and description information of multiple song segments from the sample pair into the song generation model for reinforcement learning. The goal is to maximize the difference between the probability of generating positive samples and the probability of generating negative samples, allowing the model to strengthen the features of positive samples and suppress the features of negative samples. This ultimately enables the song generation model to generate songs that better meet human aesthetic needs.

[0142] One feasible approach is to use a fine-tuned song generation model to generate multiple songs for the same lyrics. Multiple sample pairs are then constructed using these songs, each pair consisting of positive and negative samples. Positive samples are selected from songs whose quality is greater than or equal to a preset first quality threshold, while negative samples are selected from songs whose quality is less than or equal to a preset second quality threshold. The first quality threshold is greater than the second quality threshold, and the quality difference between the positive and negative samples in the same sample pair is greater than a preset difference threshold. Reinforcement learning is then performed on the song generation model using these multiple sample pairs to maximize the difference between the probability of generating a positive sample and the probability of generating a negative sample.

[0143] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0144] According to another embodiment, a song generation apparatus is provided. Figure 6 This is a schematic block diagram of a song generation apparatus provided in an embodiment of this application. The apparatus is disposed in... Figure 1 The server side in the illustrated architecture. For example... Figure 6 As shown, the song generation device 600 includes: a data acquisition unit 601, a paragraph planning unit 602, an encoding processing unit 603, a description processing unit 604, and a song generation unit 605. The main functions of each component are as follows: The data acquisition unit 601 is configured to acquire input data, including lyrics and global description information.

[0145] The paragraph planning unit 602 is configured to generate description information for multiple song paragraphs based on lyrics and global description information.

[0146] The encoding processing unit 603 is configured to encode the lyrics, global description information, and description information of multiple song segments respectively, to obtain feature representations of the lyrics, feature representations of the global description information, and feature representations of the description information of multiple song segments.

[0147] The description processing unit 604 is configured to obtain a description feature representation based on the feature representation of global description information and the feature representation of description information of multiple song segments.

[0148] The song generation unit 605 is configured to input the descriptive feature representation and the lyric feature representation as guiding conditions into the song generation model, and the song generation model generates a song based on the guiding conditions, wherein the song generation model is implemented based on a diffusion model.

[0149] As one optional implementation, when the paragraph planning unit 602 generates description information for multiple song paragraphs based on lyrics and global description information, it is specifically configured to: generate a first prompt instruction based on lyrics and global description information, the first prompt instruction being used to prompt the large language model to generate description information for song paragraphs; input the first prompt instruction into the large language model; and obtain description information for multiple song paragraphs generated by the large language model based on the first prompt instruction; or, generate a second prompt instruction based on lyrics, global description information, and user-specified paragraph description categories, the second prompt instruction being used to prompt the large language model to generate description information for song paragraphs of the specified paragraph description categories; input the second prompt instruction into the large language model; and obtain description information for multiple song paragraphs generated by the large language model based on the second prompt instruction.

[0150] As one optional implementation, when the description processing unit 604 obtains the description feature representation based on the feature representation of global description information and the feature representation of description information of multiple song segments, it is specifically configured to: align the feature representation of global description information and the feature representation of description information of multiple song segments with the song duration to obtain global temporal feature representation and multiple segment temporal feature representation respectively; and fuse the global temporal feature representation and the multiple segment temporal feature representation to obtain the description feature representation.

[0151] As one optional implementation, when the description processing unit 604 aligns the feature representation of the global description information and the feature representation of the description information of multiple song segments with the song duration to obtain the global temporal feature representation and the temporal feature representation of multiple segments respectively, it is specifically configured to: copy the feature representation of the global description information onto the song duration to obtain the global temporal feature representation; and copy the feature representation of the description information of multiple song segments into their respective segment duration windows to obtain the multiple segment temporal feature representations.

[0152] As one of the optional implementations, when the description processing unit 604 fuses the global temporal feature representation and multiple paragraph temporal feature representations to obtain the description feature representation, it is specifically configured to: concatenate the global temporal feature representation and multiple paragraph temporal feature representations; and perform linear projection transformation on the concatenated feature representation to obtain the description feature representation.

[0153] Furthermore, the data acquisition unit 601 is also configured to: return the description information of multiple song segments to the user before encoding the lyrics, global description information and description information of multiple song segments respectively, and obtain the description information of multiple song segments edited by the user; the encoding processing unit 603 is specifically configured to: encode the description information of multiple song segments edited by the user when encoding the description information of multiple song segments.

[0154] Furthermore, the data acquisition unit 601 is also configured to: before encoding the lyrics, perform duration prediction based on the lyrics, global description information, and description information of multiple song segments, generate timestamp information of the lyrics, return the timestamp information of the lyrics to the user, and obtain the timestamp information after the user edits; when encoding the lyrics, the encoding processing unit 603 is specifically configured to: encode the lyrics and their timestamp information to obtain the feature representation of the lyrics.

[0155] According to another embodiment, a training apparatus for a song generation model is provided. Figure 7 This is a schematic block diagram of a training device for a song generation model provided in an embodiment of this application. The device is disposed in... Figure 1 The server side in the illustrated architecture. For example... Figure 7 As shown, the training device 700 for the song generation model includes a sample acquisition unit 701 and a model training unit 702. The main functions of each component are as follows: The sample acquisition unit 701 is configured to acquire first training data including multiple training samples, the training samples including song samples and their corresponding lyrics, global description information and description information of multiple song segments.

[0156] The model training unit 702 is configured to train a song generation model using the first training data. The song generation model is based on a diffusion model. The training includes: encoding the lyrics, global description information, and description information of multiple song segments to obtain feature representations of the lyrics, global description information, and description information of multiple song segments; obtaining description feature representations based on the feature representations of the global description information and description information of multiple song segments; inputting the description feature representations and the feature representations of the lyrics as guiding conditions into the song generation model, which then generates a song based on the guiding conditions; and updating the parameters of the song generation model using a preset loss function.

[0157] As one optional implementation, when the sample acquisition unit 701 acquires first training data including multiple training samples, it is specifically configured to: acquire multiple song samples, and for each song sample, perform the following: separate the human voice from the song sample, and perform speech recognition on the human voice to obtain the lyrics corresponding to the song sample; use a music understanding model to obtain the understood content of the song sample, and obtain the global description information corresponding to the song sample based on the understood content; use a segmentation model to segment the song sample, and input the multiple segmented song samples into the music understanding model to obtain the description information of the multiple song segments.

[0158] As one optional implementation, when acquiring multiple song samples, the sample acquisition unit 701 is specifically configured to: acquire multiple candidate songs; perform at least one of the following filtering processes on the multiple candidate songs to obtain multiple song samples: detect the time ratio of music and vocals in the candidate songs, and filter out candidate songs whose time ratio is lower than a preset ratio threshold; based on the audio metadata of the candidate songs, filter out candidate songs whose audio metadata does not meet a preset first requirement; and use an audio quality assessment model to assess the quality of the candidate songs, and filter out candidate songs whose quality does not meet a preset first quality requirement.

[0159] As one optional implementation, when the sample acquisition unit 701 separates the human voice from the song sample and performs speech recognition on the human voice to obtain the lyrics corresponding to the song sample, it is specifically configured to: separate the human voice from the song sample and perform speech recognition on the human voice; if the song sample does not have a lyrics file, then the result of speech recognition is used as the lyrics corresponding to the lyrics sample; if the lyrics sample has a lyrics file, then after deleting the song meta-information in the lyrics file, the similarity between the lyrics file and the speech recognition result is calculated; if the similarity is greater than or equal to a preset similarity threshold, then the lyrics corresponding to the song sample are obtained based on the lyrics file; if the similarity is less than the similarity threshold, then the song sample is filtered out.

[0160] As one optional implementation, when training the song generation model using the first training data, the model training unit 702 is specifically configured to: pre-train the song generation model using the first training data; select candidate songs whose audio metadata meets the second requirement and / or whose quality meets the second quality requirement from the training data to obtain second training data, wherein the second requirement and the second quality requirement are higher than the first requirement and the first quality requirement used when obtaining the first training data; fine-tune the pre-trained song generation model using the second training data; and perform reinforcement learning on the fine-tuned song generation model to obtain a song generation model used to generate songs.

[0161] As one optional implementation, when the model training unit 702 performs reinforcement learning on the song generation model obtained through fine-tuning training, it is specifically configured to: generate multiple songs for the same lyrics using the song generation model obtained through fine-tuning training; construct multiple sample pairs using the multiple songs, the sample pairs including positive samples and negative samples, the positive samples are selected from the multiple songs whose quality is greater than or equal to a preset first quality threshold, the negative samples are selected from the multiple songs whose quality is less than or equal to a preset second quality threshold, the first quality threshold is greater than the second quality threshold, and the quality difference between the positive samples and negative samples in the same sample pair is greater than a preset difference threshold; and perform reinforcement learning on the song generation model using multiple sample pairs to maximize the difference between the probability of the song generation model generating positive samples and the probability of generating negative samples.

[0162] Furthermore, the loss function is the conditional flow matching loss function, which minimizes the difference between the vector field predicted by the song generation model and the target vector field; where the predicted vector field is the path from the noise latent vector to the latent vector of the song generated by the song generation model, and the target vector field is the path from the noise latent vector to the latent vector of the song sample.

[0163] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0164] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0165] And an electronic device, comprising: One or more processors; and A memory associated with one or more processors, the memory being used to store program instructions that, when read and executed by one or more processors, perform the steps of any of the methods in the foregoing method embodiments.

[0166] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods in the foregoing method embodiments.

[0167] in, Figure 8 The schematic block diagram of the electronic device provided in the embodiments of this application may specifically include a processor 810, a video display adapter 811, a disk drive 812, an input / output interface 813, a network interface 814, and a memory 820. The processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820 can be communicatively connected via a communication bus 830.

[0168] The processor 810 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0169] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 can store the operating system 821 for controlling the operation of the electronic device 800, and the basic input / output system (BIOS) 822 for controlling the low-level operations of the electronic device 800. Additionally, it can store a web browser 823, a data storage management system 824, a song generation device 600, and a song generation model training device 700, etc. The aforementioned song generation device 600 and song generation model training device 700 can be application programs that specifically implement the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 820 and executed by the processor 810.

[0170] The input / output interface 813 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0171] Network interface 814 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0172] Bus 830 includes a pathway for transmitting information between various components of the device, such as processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820.

[0173] It should be noted that although the above-described device only shows the processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, memory 820, bus 830, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0174] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0175] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A song generation method, characterized in that, The method includes: Acquire input data, which includes lyrics and global description information; Based on the lyrics and global description information, description information for multiple song segments is generated; The lyrics, the global description information, and the description information of the multiple song segments are encoded respectively to obtain the feature representations of the lyrics, the global description information, and the description information of the multiple song segments; Based on the feature representation of the global description information and the feature representation of the description information of the multiple song segments, a description feature representation is obtained; The descriptive feature representation and the lyric feature representation are used as guiding conditions to input into the song generation model, and the song generation model generates a song based on the guiding conditions, wherein the song generation model is implemented based on a diffusion model.

2. The method according to claim 1, characterized in that, The step of generating description information for multiple song segments based on the lyrics and global description information includes: A first prompt instruction is generated based on the lyrics and global description information. This first prompt instruction is used to prompt the large language model to generate description information for song segments. The first prompt instruction is input into the large language model, and description information for multiple song segments generated by the large language model based on the first prompt instruction is obtained; or, Based on the lyrics, global description information, and user-specified paragraph description categories, a second prompt instruction is generated. The second prompt instruction is used to prompt the large language model to generate description information of song paragraphs of the specified paragraph description categories. The second prompt instruction is input into the large language model, and description information of multiple song paragraphs generated by the large language model based on the second prompt instruction is obtained.

3. The method according to claim 1, characterized in that, The feature representation based on the global description information and the feature representation of the description information of the multiple song segments is used to obtain a description feature representation, including: The feature representations of the global description information and the feature representations of the description information of the multiple song segments are aligned with the song duration to obtain the global temporal feature representation and the temporal feature representations of the multiple segments, respectively. The global temporal feature representation and the multiple segment temporal feature representations are fused together to obtain the descriptive feature representation.

4. The method according to claim 3, characterized in that, The step of aligning the feature representations of the global description information and the feature representations of the description information of the multiple song segments with the song duration to obtain the global temporal feature representation and the temporal feature representations of the multiple segments includes: The feature representation of the global description information is copied onto the song duration to obtain a global temporal feature representation; The feature representations of the description information of the multiple song segments are copied into their respective segment duration windows to obtain multiple segment temporal feature representations.

5. The method according to claim 3, characterized in that, The process of fusing the global temporal feature representation and the multiple segment temporal feature representations to obtain a descriptive feature representation includes: The global temporal feature representation and the multiple segment temporal feature representations are concatenated; The concatenated feature representation is transformed by linear projection to obtain the descriptive feature representation.

6. The method according to any one of claims 1 to 5, characterized in that, Before encoding the lyrics, the global description information, and the description information of the multiple song segments respectively, the method further includes: returning the description information of the multiple song segments to the user and obtaining the description information of the multiple song segments edited by the user. Encoding the description information of the multiple song segments includes: encoding the description information of the multiple song segments after user editing.

7. The method according to claim 6, characterized in that, Before encoding the lyrics, the process further includes: predicting the duration based on the lyrics, the global description information, and the description information of the multiple song segments; generating timestamp information for the lyrics; returning the timestamp information for the lyrics to the user; and obtaining the timestamp information edited by the user. The encoding process for the lyrics includes: encoding the lyrics and their timestamp information to obtain a feature representation of the lyrics.

8. A training method for a song generation model, characterized in that, The method includes: Acquire first training data including multiple training samples, wherein the training samples include song samples and their corresponding lyrics, global description information and description information of multiple song segments; A song generation model is trained using the first training data. The song generation model is based on a diffusion model. The training includes: encoding the lyrics, the global description information, and the description information of the multiple song segments to obtain feature representations of the lyrics, the global description information, and the description information of the multiple song segments; obtaining a description feature representation based on the feature representations of the global description information and the description information of the multiple song segments; inputting the description feature representation and the feature representation of the lyrics as guiding conditions into the song generation model, and having the song generation model generate a song based on the guiding conditions; and updating the parameters of the song generation model using a preset loss function.

9. The method according to claim 8, characterized in that, The acquisition of first training data, which includes multiple training samples, includes: Obtain multiple song samples, and perform the following steps for each song sample: The human voice is extracted from the song sample, and speech recognition is performed on the human voice to obtain the lyrics corresponding to the song sample; The song sample is understood using a music understanding model, and global description information corresponding to the song sample is obtained based on the understood content. The song sample is divided into segments using a segmentation model. The resulting multiple song segments are then input into the music understanding model to obtain descriptive information for each song segment.

10. The method according to claim 9, characterized in that, The acquisition of multiple song samples includes: Get multiple candidate songs; Perform at least one of the following filtering processes on the multiple candidate songs to obtain the multiple song samples: Detect the time ratio of music and vocals in the candidate songs, and filter out candidate songs whose time ratio is lower than a preset ratio threshold. Based on the audio metadata of the candidate songs, filter out candidate songs whose audio metadata does not meet the preset first requirement. The candidate songs are evaluated using an audio quality assessment model, and candidate songs whose quality does not meet the preset first quality requirement are filtered out.

11. The method according to claim 9, characterized in that, The step of separating human voices from the song samples and performing speech recognition on the human voices to obtain the lyrics corresponding to the song samples includes: The human voice is extracted from the song sample, and speech recognition is performed on the human voice; If the song sample does not have a lyrics file, the result of the speech recognition will be used as the lyrics corresponding to the lyrics sample. If the lyrics sample exists in a lyrics file, the song metadata in the lyrics file is deleted, and the similarity between the lyrics file and the speech recognition result is calculated. If the similarity is greater than or equal to a preset similarity threshold, the lyrics corresponding to the song sample are obtained based on the lyrics file. If the similarity is less than the similarity threshold, the song sample is filtered out.

12. The method according to any one of claims 8 to 11, characterized in that, The step of training the song generation model using the first training data includes: The song generation model is pre-trained using the first training data; Candidate songs whose audio metadata meets the second requirement and / or whose quality meets the second quality requirement are selected from the training data to obtain second training data. The second requirement and the second quality requirement are higher than the first requirement and the first quality requirement used when the first training data was obtained. The pre-trained song generation model is fine-tuned using the second training data. The song generation model obtained through fine-tuning training is subjected to reinforcement learning to obtain a song generation model used to generate songs.

13. The method according to claim 12, characterized in that, The reinforcement learning of the song generation model obtained through fine-tuning training includes: The song generation model obtained through fine-tuning training can generate multiple songs with the same lyrics. Multiple sample pairs are constructed using the multiple songs. Each sample pair includes positive samples and negative samples. The positive samples are selected from the multiple songs whose quality is greater than or equal to a preset first quality threshold. The negative samples are selected from the multiple songs whose quality is less than or equal to a preset second quality threshold. The first quality threshold is greater than the second quality threshold. The quality difference between the positive samples and the negative samples in the same sample pair is greater than a preset difference threshold. The song generation model is reinforced using the multiple sample pairs to maximize the difference between the probability of generating the positive sample and the probability of generating the negative sample.

14. The method according to any one of claims 8 to 11, characterized in that, The loss function is the conditional flow matching loss function, which minimizes the difference between the vector field predicted by the song generation model and the target vector field. The predicted vector field is the path from the noise latent vector to the latent vector of the song generated by the song generation model, and the target vector field is the path from the noise latent vector to the latent vector of the song sample.

15. A song generation device, characterized in that, The device includes: The data acquisition unit is configured to acquire input data, which includes lyrics and global description information. The paragraph planning unit is configured to generate description information for multiple song paragraphs based on the lyrics and global description information; The encoding processing unit is configured to encode the lyrics, the global description information, and the description information of the multiple song segments respectively to obtain feature representations of the lyrics, feature representations of the global description information, and feature representations of the description information of the multiple song segments. The description processing unit is configured to obtain a description feature representation based on the feature representation of the global description information and the feature representation of the description information of the multiple song segments; The song generation unit is configured to input the descriptive feature representation and the lyric feature representation as guiding conditions into the song generation model, and the song generation model generates a song based on the guiding conditions, wherein the song generation model is implemented based on a diffusion model.

16. A training device for a song generation model, characterized in that, The device includes: The sample acquisition unit is configured to acquire first training data including multiple training samples, wherein the training samples include song samples and their corresponding lyrics, global description information and description information of multiple song segments; The model training unit is configured to train a song generation model using the first training data. The song generation model is based on a diffusion model. The training includes: encoding the lyrics, the global description information, and the description information of the multiple song segments to obtain feature representations of the lyrics, the global description information, and the description information of the multiple song segments; obtaining a description feature representation based on the feature representations of the global description information and the description information of the multiple song segments; inputting the description feature representation and the feature representation of the lyrics as guiding conditions into the song generation model, and having the song generation model generate a song based on the guiding conditions; and updating the parameters of the song generation model using a preset loss function.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 14.

18. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 14.

19. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 14.

Images