Song synthesis model training method and device, song synthesis method and device

Abstract

The application relates to a training method of an artificial intelligence song synthesis model, comprising the following steps: obtaining an initial sample set, wherein the initial sample set comprises initial samples of multiple sound sources, and the initial samples comprise recorded audio, source lyric duration information of the recorded audio and source score information of the recorded audio; based on sample augmentation of audio transformation of the recorded audio in the initial samples, an augmented sample set is obtained, wherein the augmented samples of the augmented sample set comprise augmented audio obtained through audio transformation, augmented lyric duration information of the augmented audio and augmented score information of the augmented audio; model pre-training is performed according to the initial sample set and the augmented sample set, and an initial song synthesis model is obtained; audio of a target sound source is obtained, and timbre features are extracted based on the audio of the target sound source; the initial song synthesis model is trained based on the timbre features, and a song synthesis model is obtained. The method can realize timbre customization of a synthesized song, thereby improving the naturalness of song synthesis.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium, and computer program product for training a song synthesis model, as well as a method, apparatus, computer equipment, storage medium, and computer program product for song synthesis. Background Technology

[0002] With the development of computer technology, song synthesis technology has emerged. This technology can synthesize multiple pieces of music into a complete audio file, or synthesize corresponding singing audio based on lyrics and sheet music.

[0003] Traditional vocal synthesis involves using a trained synthesis model to synthesize corresponding vocals based on lyrics and sheet music. Currently, the training data for the synthesis model is mainly constructed by manually recording songs. However, vocal synthesis requires a high degree of data diversity, and manually recorded songs can be unstable in areas with limited distribution of high or low notes, resulting in poor naturalness in the synthesized songs. Summary of the Invention

[0004] Based on this, it is necessary to provide a training method, apparatus, computer device, computer-readable storage medium, and computer program product for a song synthesis model that can improve the naturalness of song synthesis, as well as a song synthesis method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the naturalness of song synthesis.

[0005] This application provides a method for training a song synthesis model, the method comprising:

[0006] Obtain an initial sample set, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio.

[0007] Based on the audio transformation of the recorded audio in the initial sample, sample augmentation is performed to obtain an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after the audio transformation, the augmented lyrics duration information of the augmented audio, and the augmented musical score information of the augmented audio.

[0008] The model is pre-trained based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis.

[0009] Acquire the audio of the target sound source, and extract timbre features based on the audio of the target sound source;

[0010] The initial model for song synthesis is trained based on the timbre features to obtain the song synthesis model.

[0011] This application also provides a training device for a song synthesis model, the device comprising:

[0012] The acquisition module is used to acquire an initial sample set, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio.

[0013] An augmentation module is used to augment samples based on the audio transformation of the recorded audio in the initial sample to obtain an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after the audio transformation, the augmented lyrics duration information of the augmented audio, and the augmented musical score information of the augmented audio.

[0014] The pre-training module is used to pre-train the model based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis.

[0015] An extraction module is used to acquire the audio of a target sound source and extract timbre features based on the audio of the target sound source;

[0016] The training module is used to train the initial model for song synthesis based on the timbre features to obtain the song synthesis model.

[0017] In one embodiment, the augmentation module is further configured to: transform the recorded audio of the plurality of initial samples according to an audio transformation method to obtain augmented audio corresponding to each of the recorded audio; determine augmented lyric duration information corresponding to each of the augmented audio based on the source lyric duration information of the plurality of initial samples; adjust the source score information of the plurality of initial samples through a score transformation method that matches the audio transformation method to obtain augmented score information corresponding to each of the augmented audio; and form an augmented sample set based on each of the augmented audio, the augmented lyric duration information of each of the augmented audio, and the augmented score information of each of the augmented audio.

[0018] In one embodiment, the augmentation module is further configured to perform pitch adjustment processing on the recorded audio of the plurality of initial samples respectively to obtain augmented audio corresponding to each of the recorded audio; use the source lyrics duration information of each recorded audio of the plurality of initial samples as the augmented lyrics duration information corresponding to the corresponding augmented audio; and perform scale adjustment processing on the notes of the source musical score information of the plurality of recorded audio according to the pitch adjustment processing of the recorded audio of the plurality of initial samples to obtain augmented musical score information corresponding to each of the augmented audio.

[0019] In one embodiment, the augmentation module is further configured to: divide the recorded audio of the multiple initial samples to obtain audio segments corresponding to each recorded audio; for each recorded audio, concatenate the audio segments of the corresponding recorded audio in adjacent order to obtain multiple augmented audio of the corresponding recorded audio; divide the source lyric duration information of the multiple initial samples according to the division processing of the recorded audio of the multiple initial samples to obtain lyric duration information segments corresponding to each audio segment; concatenate the lyric duration information segments of each audio segment according to the concatenation processing of each audio segment to obtain augmented lyric duration information corresponding to each augmented audio; divide the source musical score information of the multiple initial samples according to the division processing of the recorded audio of the multiple initial samples to obtain musical score information segments corresponding to each audio segment; concatenate the musical score information segments of each audio segment according to the concatenation processing of each audio segment to obtain augmented musical score information corresponding to each augmented audio.

[0020] In one embodiment, the pre-training module is further configured to: obtain sample audio, sample lyric duration information corresponding to the sample audio, and sample musical score information corresponding to the sample audio from the set consisting of the initial sample set and the augmented sample set; perform feature encoding based on the sample lyric duration information and sample musical score information of the sample audio to obtain sample encoding features; extend the duration features of the sample encoding features according to the sample lyric duration features of the sample lyric duration information to obtain sample duration extended encoding features; extract sample timbre features of the sample audio; concatenate the sample timbre features and the sample duration extended encoding features and perform acoustic feature extraction to obtain predicted spectral features; synthesize a predicted song based on the predicted spectral features; construct a target loss function based on the synthesis loss between the predicted song and the sample audio; and perform model pre-training based on the target loss function to obtain an initial model for song synthesis.

[0021] In one embodiment, the pre-training module is further configured to perform duration prediction processing based on the sample lyric features of the sample lyric duration information, the sample musical score features of the sample musical score information, and the sample timbre features to obtain the predicted phoneme duration corresponding to each phoneme in the sample lyric duration information; determine the phoneme duration loss between the predicted phoneme duration of each phoneme and the sample phoneme duration of each phoneme in the sample lyric duration information; determine the synthesis loss between the predicted song and the sample audio; and construct a target loss function based on the phoneme duration loss and the synthesis loss.

[0022] In one embodiment, the pre-training module is further configured to: determine the predicted syllable duration corresponding to each syllable in the sample lyrics duration information based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information; determine the syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information; and construct a target loss function based on the phoneme duration loss, the syllable duration loss, and the synthesis loss.

[0023] In one embodiment, the pre-training module is further configured to perform gradient inversion processing on the encoded features of the samples, and classify the samples based on the features obtained by the gradient inversion processing to obtain the classification result of the sample audio; determine the adversarial loss between the classification result and the classification label of the sample audio; determine the synthesis loss between the predicted song and the sample audio; and construct a target loss function based on the adversarial loss and the synthesis loss.

[0024] In one embodiment, the pre-training module is further configured to extract sample spectral features of the sample audio and determine the spectral loss between the predicted spectral features and the sample spectral features; determine the synthesis loss between the predicted song and the sample audio; and construct a target loss function based on the spectral loss and the synthesis loss.

[0025] In one embodiment, the training module is used to adjust the parameters of the initial duration model, the initial acoustic model, and the initial vocoder based on the timbre characteristics of the target sound source, so as to obtain a song synthesis model that matches the timbre of the target sound source.

[0026] This application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0027] An initial sample set is obtained, comprising initial samples from multiple sound sources, including recorded audio, source lyrics duration information, and source musical score information. An augmented sample set is obtained by performing audio transformations on the recorded audio in the initial sample set, comprising augmented audio obtained through the audio transformations, augmented lyrics duration information, and augmented musical score information. A model pre-training is performed based on the initial sample set and the augmented sample set to obtain an initial song synthesis model. Audio from a target sound source is obtained, and timbre features are extracted from the audio of the target sound source. The initial song synthesis model is trained based on the timbre features to obtain a song synthesis model.

[0028] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0029] An initial sample set is obtained, comprising initial samples from multiple sound sources, including recorded audio, source lyrics duration information, and source musical score information. An augmented sample set is obtained by performing audio transformations on the recorded audio in the initial sample set, comprising augmented audio obtained through the audio transformations, augmented lyrics duration information, and augmented musical score information. A model pre-training is performed based on the initial sample set and the augmented sample set to obtain an initial song synthesis model. Audio from a target sound source is obtained, and timbre features are extracted from the audio of the target sound source. The initial song synthesis model is trained based on the timbre features to obtain a song synthesis model.

[0030] This application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0031] An initial sample set is obtained, comprising initial samples from multiple sound sources, including recorded audio, source lyrics duration information, and source musical score information. An augmented sample set is obtained by performing audio transformations on the recorded audio in the initial sample set, comprising augmented audio obtained through the audio transformations, augmented lyrics duration information, and augmented musical score information. A model pre-training is performed based on the initial sample set and the augmented sample set to obtain an initial song synthesis model. Audio from a target sound source is obtained, and timbre features are extracted from the audio of the target sound source. The initial song synthesis model is trained based on the timbre features to obtain a song synthesis model.

[0032] The training method, apparatus, computer equipment, storage medium, and computer program product of the aforementioned song synthesis model acquire an initial sample set consisting of initial samples from multiple sound sources. The recorded audio, the duration of the source lyrics of the recorded audio, and the source musical score information of the recorded audio included in the initial samples serve as the basic data for sample augmentation. Based on the audio transformation of the recorded audio in the initial samples, an augmented sample set is obtained. This augmented sample set includes augmented audio obtained through audio transformation, the duration of the augmented lyrics of the augmented audio, and the augmented musical score information of the augmented audio. This allows for the acquisition of a large amount of training corpus through sample augmentation, expanding the quantity and richness of the training corpus. Furthermore, by obtaining more training corpus on top of the basic data through sample augmentation, the requirement for a large amount of basic data can be reduced, thus significantly reducing the cost of manual recording and annotation of the basic data. Pre-training the model based on the large amount of training corpus contained in the initial sample set and the augmented sample set improves the robustness of the pre-trained initial song synthesis model. The system extracts timbre features from the target sound source and trains an initial song synthesis model based on these features. It can fine-tune the pre-trained initial song synthesis model based on individual audio data, accurately obtaining a song synthesis model that matches the target timbre of the target sound source. This effectively achieves customized timbre effects in song synthesis, thereby improving the naturalness of the synthesized song.

[0033] This application provides a song synthesis method, the method comprising:

[0034] Obtain target lyrics and target sheet music information, and perform feature encoding based on the target lyrics and target sheet music information to obtain encoded features;

[0035] Obtain the lyrics duration features of the target lyrics, and extend the duration features of the encoding features according to the lyrics duration features to obtain the duration extended encoding features;

[0036] The timbre features of the target timbre are determined, and the timbre features and the duration-extended coding features are concatenated and then acoustic features are extracted to obtain the target spectral features.

[0037] A target song is synthesized based on the target spectral features, and the target song is matched with the target lyrics, the target musical score information, and the target timbre.

[0038] This application also provides a song synthesis apparatus, the apparatus comprising:

[0039] The encoding module is used to acquire target lyrics and target musical score information, and to perform feature encoding based on the target lyrics and the target musical score information to obtain encoded features;

[0040] An extension module is used to obtain the lyrics duration features of the target lyrics, and extend the duration features of the encoding features according to the lyrics duration features to obtain duration extended encoding features;

[0041] The determination module is used to determine the timbre features of the target timbre, and then perform acoustic feature extraction after concatenating the timbre features and the duration-extended coding features to obtain the target spectral features;

[0042] A synthesis module is used to synthesize a target song based on the target spectral features, wherein the target song is matched with the target lyrics, the target musical score information and the target timbre.

[0043] In one embodiment, the apparatus further includes a duration prediction module, which is configured to perform duration prediction processing based on the lyric features of the target lyrics, the musical score features of the target musical score information, and the timbre features of the target timbre to obtain the phoneme duration corresponding to each phoneme in the target lyrics; and extend the duration features of the encoding features according to the phoneme duration of each phoneme to obtain the duration-extended encoding features.

[0044] This application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0045] The process involves: acquiring target lyrics and target sheet music information; performing feature encoding based on the target lyrics and target sheet music information to obtain encoded features; acquiring the duration features of the target lyrics and expanding the duration features of the encoded features according to the duration features to obtain duration-extended encoded features; determining the timbre features of the target timbre; concatenating the timbre features and the duration-extended encoded features and then extracting acoustic features to obtain target spectral features; and synthesizing a target song based on the target spectral features, wherein the target song matches the target lyrics, the target sheet music information, and the target timbre.

[0046] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0047] The process involves: acquiring target lyrics and target sheet music information; performing feature encoding based on the target lyrics and target sheet music information to obtain encoded features; acquiring the duration features of the target lyrics and expanding the duration features of the encoded features according to the duration features to obtain duration-extended encoded features; determining the timbre features of the target timbre; concatenating the timbre features and the duration-extended encoded features and then extracting acoustic features to obtain target spectral features; and synthesizing a target song based on the target spectral features, wherein the target song matches the target lyrics, the target sheet music information, and the target timbre.

[0048] This application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0049] The process involves: acquiring target lyrics and target sheet music information; performing feature encoding based on the target lyrics and target sheet music information to obtain encoded features; acquiring the duration features of the target lyrics and expanding the duration features of the encoded features according to the duration features to obtain duration-extended encoded features; determining the timbre features of the target timbre; concatenating the timbre features and the duration-extended encoded features and then extracting acoustic features to obtain target spectral features; and synthesizing a target song based on the target spectral features, wherein the target song matches the target lyrics, the target sheet music information, and the target timbre.

[0050] The aforementioned song synthesis method, apparatus, computer equipment, storage medium, and computer program products perform feature encoding on the target lyrics and target musical score information as needed to obtain encoded features containing both lyric and musical score features. The duration features of the target lyrics are obtained, and the duration features of the encoded features are expanded according to these lyric duration features. Using the actual duration of the target lyrics as the basis for expansion makes the resulting extended duration encoded features more accurate. Furthermore, for synthesis scenarios requiring strict accompaniment matching, using the actual duration as the basis for expansion, rather than using predicted duration, allows the duration to accurately align with the timing of the accompaniment, resulting in a more harmonious and natural sound between the lyrics and melody of the synthesized song, making it more pleasing to the ear. The timbre features of the target timbre are determined, and after concatenating the timbre features and the extended duration encoded features, acoustic feature extraction is performed to obtain target spectral features containing the target timbre. Based on these target spectral features, a target song matching the target lyrics, target musical score information, and target timbre is synthesized, thereby effectively achieving a customized effect for the target timbre. Furthermore, given the existing lyrics duration information, the system combines lyrics, sheet music, and timbre to synthesize a target song with a specific timbre, enabling the song synthesis to have a timbre customization function, thereby improving the naturalness of the synthesized song. Attached Figure Description

[0051] Figure 1This is an application environment diagram of the training method for the song synthesis model in one embodiment;

[0052] Figure 2 This is a flowchart illustrating the training method of a song synthesis model in one embodiment;

[0053] Figure 3 This is a schematic diagram illustrating the process of raising and lowering the pitch of notes in the source musical score information in one embodiment.

[0054] Figure 4 This is a schematic diagram of the sample augmentation process in one embodiment;

[0055] Figure 5 This is a schematic diagram of the process of obtaining an initial model for song synthesis by pre-training the model based on an initial sample set and an augmented sample set in one embodiment.

[0056] Figure 6 This is a diagram illustrating the overall architecture for model pre-training in one embodiment.

[0057] Figure 7 This is a flowchart illustrating a song synthesis method in one embodiment;

[0058] Figure 8 This is an overall architecture diagram of a song synthesis model in one embodiment;

[0059] Figure 9 This is a structural block diagram of a training device for a song synthesis model in one embodiment;

[0060] Figure 10 This is a structural block diagram of a song synthesis device in one embodiment;

[0061] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, and data mining. For example, it can be applied to the field of Artificial Intelligence (AI), where AI is the theory, method, technology, and application system for simulating, extending, and expanding human intelligence using digital computers or machines controlled by digital computers, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. The solutions provided in the embodiments of this application relate to a training method for an AI-based song synthesis model, which is specifically described through the following embodiments.

[0064] The training method for the song synthesis model provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other servers. Both terminal 102 and server 104 can independently execute the training method of the song synthesis model provided in this embodiment. Terminal 102 and server 104 can also work together to execute the training method of the song synthesis model provided in this embodiment. When terminal 102 and server 104 work together to execute the training method of the song synthesis model provided in this embodiment, terminal 102 obtains an initial sample set, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio. Terminal 102 sends an initial sample set to server 104. Server 104 performs sample augmentation based on the audio transformation of the recorded audio in the initial sample set, obtaining an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after audio transformation, the duration information of the augmented lyrics of the augmented audio, and the augmented musical score information of the augmented audio. Server 104 performs model pre-training based on the initial sample set and the augmented sample set to obtain an initial model for song synthesis. Terminal 102 acquires the audio of the target sound source and sends it to server 104. Server 104 extracts timbre features based on the audio of the target sound source. Server 104 trains the initial model for song synthesis based on the timbre features to obtain a song synthesis model. This song synthesis model can be deployed on server 104 or terminal 102.

[0065] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, portable wearable devices, etc. The terminal 102 can run applications or have client applications installed. These applications can be communication applications, email applications, video applications, music applications, and image processing applications, etc. Music applications can be song synthesis applications and music playback applications, but are not limited to these. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 and the server 104 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0066] In one embodiment, the song synthesis method can also be applied to, for example... Figure 1 In the application environment shown, both terminal 102 and server 104 can independently execute the song synthesis method provided in this embodiment. Terminal 102 and server 104 can also work together to execute the song synthesis method provided in this embodiment.

[0067] It should be noted that the quantities of "multiple" mentioned in the embodiments of this application all refer to the quantity of "at least two".

[0068] The training of the song synthesis model and the song synthesis method in this embodiment can be applied to any device with speech synthesis capabilities, including but not limited to smart speakers, speakers with screens, smartwatches, smartphones, smart homes, smart cars and other smart devices, as well as smart robots, virtual anchors, virtual teaching assistants, AI customer service, and text-to-speech (TTS) cloud services. Furthermore, the method proposed in this embodiment can reduce the cost of AI data recording and enhance the skills of AI devices, while providing a wide range of entertainment applications.

[0069] In one embodiment, such as Figure 2 As shown, a training method for a song synthesis model is provided, which can be applied to... Figure 1 Computer equipment (computer equipment can be) Figure 1 Taking a terminal or server as an example, the following steps are included:

[0070] Step S202: Obtain an initial sample set, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio.

[0071] Here, "recorded audio" refers to the recorded singing audio. "Source lyrics duration information" refers to the duration annotation information of the lyrics corresponding to the recorded audio, which may include at least one of the following: lyrics, the duration of each phoneme in the lyrics, or the duration of each syllable. A phoneme is the smallest unit of speech defined based on the natural attributes of speech; phonemes are divided into two main categories: vowels and consonants. A syllable is a unit of speech composed of one or more phonemes; it is the smallest unit of speech in a language formed by the combination of a single vowel phoneme and a consonant phoneme. A single vowel phoneme can also constitute a syllable.

[0072] Each syllable can correspond to one word in the lyrics, and each syllable can correspond to multiple phonemes. The syllable duration of each syllable is the sum of the phoneme durations of the phonemes corresponding to each syllable.

[0073] Musical score is a way of recording music using symbols. Source score information refers to the melody information corresponding to the recorded audio, including notes, note values, rhythm, legato, and sustain. Notes are symbols on the score that represent duration or pitch. Slurs indicate performance information; they connect several notes of different pitches, indicating that these notes should be played smoothly and legato. Note value is the duration of a note's performance. Each syllable can correspond to a word in lyrics, and the duration of each syllable is the sum of the note values ​​of the notes corresponding to that syllable. Pitch refers to the different levels of sound, that is, the height of a note; pitch is one of the fundamental characteristics of sound.

[0074] Specifically, the computer device can acquire recorded audio from multiple sound sources locally or from other devices or networks, and obtain the source lyrics duration information and source musical score information for each recorded audio. The recorded audio, its source lyrics duration information, and its source musical score information constitute a single initial sample. Based on the initial samples from multiple sound sources, an initial sample set is generated, which includes initial samples from multiple sound sources.

[0075] In this embodiment, the computer device can acquire the source lyrics information corresponding to each recorded audio, and perform duration annotation on each source lyrics information to obtain the duration annotation information of each phoneme and each syllable in the source lyrics. The duration annotation information of each syllable is used as the duration annotation information of each word in the source lyrics. The duration annotation information of each phoneme and each syllable in the source lyrics is used as the source lyrics duration information corresponding to that source lyrics information, that is, the source lyrics duration information of the recorded audio.

[0076] Step S204: Based on the audio transformation of the recorded audio in the initial sample, sample augmentation is performed to obtain an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after audio transformation, the augmented lyrics duration information of the augmented audio, and the augmented musical score information of the augmented audio.

[0077] Sample augmentation refers to increasing the richness of the initial samples, including expanding the number and type of initial samples, but not limited to these. Audio transformation refers to the way audio attribute parameters are changed; it involves generating new audio by adjusting the audio attribute parameters. Audio attribute parameters include, but are not limited to, audio duration, pitch, and tempo.

[0078] Augmented audio refers to audio obtained by performing audio transformations on recorded audio. Augmented audio differs from the corresponding recorded audio in at least one attribute parameter, such as pitch or duration.

[0079] Augmented lyric duration information refers to the duration annotation information of the lyrics corresponding to the augmented audio, which may include at least one of the following: lyrics, the duration of each phoneme in the lyrics, or the duration of each syllable. Augmented musical score information refers to the melody information corresponding to the augmented audio, including notes, note values, meter, legato, sustain, etc.

[0080] Specifically, for each initial sample in the initial sample set, the computer device performs audio transformation processing on the recorded audio of each initial sample to adjust the attribute parameters of the recorded audio, obtaining augmented audio corresponding to each recorded audio. Based on the audio transformation processing of the recorded audio, the computer device processes the source lyrics duration information of the recorded audio accordingly, obtaining augmented lyrics duration information corresponding to the augmented audio of the recorded audio. Based on the audio transformation processing of the recorded audio, the computer device performs corresponding score transformation processing on the source musical score information of the recorded audio, obtaining augmented musical score information corresponding to the augmented audio of the recorded audio.

[0081] The computer device uses augmented audio, augmented lyrics duration information, and augmented musical score information to form a single augmented sample. These augmented samples constitute an augmented sample set.

[0082] In this embodiment, the computer device can select multiple initial samples from the initial sample set, and perform sample augmentation based on the audio transformation of the recorded audio in the selected initial samples to obtain an augmented sample set. The augmented sample set includes the augmented samples corresponding to each selected initial sample.

[0083] Step S206: Perform model pre-training based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis.

[0084] Specifically, the initial model for song synthesis refers to the model obtained through pre-training. The computer equipment can use the initial samples in the initial sample set and the augmented samples in the augmented sample set as training samples for model pre-training. The constructed model is pre-trained using each initial sample and each augmented sample, and the model parameters are adjusted during training until the pre-training stopping condition is met, thus obtaining the initial model for song synthesis.

[0085] In this embodiment, the computer device acquires sample audio from a set consisting of an initial sample set and an augmented sample set, and acquires the duration information of the sample lyrics and the corresponding musical score information. The computer device pre-trains the constructed model based on each sample audio, the duration information of the sample lyrics, and the musical score information, adjusting the model parameters during training until the pre-training stopping condition is met, thus obtaining the initial model for song synthesis.

[0086] When the acquired sample audio is recorded audio, the duration of the source lyrics corresponding to the recorded audio is used as the duration of the sample lyrics corresponding to the sample audio, and the duration of the source sheet music corresponding to the recorded audio is used as the duration of the sample sheet music corresponding to the sample audio. When the acquired sample audio is augmented audio, the duration of the augmented lyrics corresponding to the augmented audio is used as the duration of the sample lyrics corresponding to the sample audio, and the duration of the augmented sheet music corresponding to the augmented audio is used as the duration of the sample sheet music corresponding to the sample audio.

[0087] Step S208: Obtain the audio of the target sound source and extract timbre features based on the audio of the target sound source.

[0088] Timbre, as one of the attributes of sound, is primarily determined by its overtones. Timbre characteristics characterize the features and essence of sound, also known as "sound quality." A target sound source refers to the sound source of a target object, and the audio of the target sound source refers to the audio recorded from the sound of the target object.

[0089] Specifically, the computer device can acquire the audio corresponding to the target sound source and extract features from the audio of the target sound source to obtain the corresponding timbre features.

[0090] In this embodiment, the audio of the target sound source refers to the singing audio recorded by the voice of the target object.

[0091] In this embodiment, the computer device can acquire the singing audio corresponding to the target sound source, the source lyrics duration information corresponding to the singing audio, and the source musical score information corresponding to the singing audio.

[0092] Step S210: Train the initial model for song synthesis based on timbre features to obtain the song synthesis model.

[0093] Specifically, the computer equipment trains the initial model of song synthesis based on timbre features, and adjusts the parameters of the initial model during training until the pre-training stopping condition is met, thus obtaining a song synthesis model that matches the target timbre of the target sound source.

[0094] When multiple target sound sources exist, features are extracted from the audio of each target sound source to obtain the timbre features corresponding to each target sound source. Based on the timbre features of each target sound source, the initial song synthesis model is trained separately to obtain song synthesis models that match the target timbre of each target sound source.

[0095] The song synthesis model is used to synthesize songs based on target lyrics and target musical score information, resulting in a target song sung in the target timbre. This target timbre can be selected by the user from multiple candidate timbres.

[0096] In this embodiment, the computer device can acquire multiple singing audios corresponding to the target sound source, as well as the lyrics duration information and musical score information corresponding to each singing audio. The initial song synthesis model is trained using the multiple singing audios, lyrics duration information, and musical score information corresponding to the target sound source to obtain a song synthesis model whose timbre matches the target sound source.

[0097] When multiple target sound sources exist, the initial song synthesis model is trained using the singing audio, lyrics duration information, and musical score information of each target sound source to obtain a song synthesis model that matches the timbre of each target sound source. Different target sound sources represent different target objects.

[0098] For example, acquiring singing data for target object A includes multiple singing audio recordings of target object A, along with the lyrics duration and sheet music information for each recording. The initial song synthesis model is trained using the singing data of target object A to obtain a song synthesis model that matches the timbre of target object A. When multiple target objects A, B, and C exist, the initial song synthesis model is trained using the singing data of each target object, resulting in song synthesis models matching the timbre of target object A, B, and C, respectively. Alternatively, the song synthesis models matching the timbre of targets A, B, and C can be used as sub-models, and these sub-models can be integrated into a single target song synthesis model. When a user uses the target song synthesis model to synthesize a song, they can input target lyrics and sheet music information, and select the target timbre from multiple candidate timbres provided by the model. The target lyrics and sheet music information are then synthesized using the sub-model matching the target timbre to obtain the target song sung with the target timbre.

[0099] In this embodiment, an initial sample set consisting of initial samples from multiple sound sources is obtained. The recorded audio, the duration of the source lyrics, and the source musical score information included in the initial samples are used as the base data for sample augmentation. Based on the audio transformation of the recorded audio in the initial samples, an augmented sample set is obtained. This augmented sample set includes augmented audio obtained after audio transformation, the duration of the augmented lyrics, and the augmented musical score information. This allows for the acquisition of a large amount of training data, expanding the quantity and richness of the training corpus. Furthermore, sample augmentation can obtain more training data on top of the base data, reducing the requirement for a large amount of base data and significantly reducing the cost of manual recording and annotation of the base data. Pre-training the model using the large amount of training data contained in the initial and augmented sample sets improves the robustness of the pre-trained song synthesis initial model. The system extracts timbre features from the target sound source and trains an initial song synthesis model based on these features. It can fine-tune the pre-trained initial song synthesis model based on individual audio data, accurately obtaining a song synthesis model that matches the target timbre of the target sound source. This effectively achieves customized timbre effects in song synthesis, thereby improving the naturalness of the synthesized song.

[0100] In one embodiment, sample augmentation is performed based on audio transformations of recorded audio in the initial samples to obtain an augmented sample set, including:

[0101] The recorded audio of multiple initial samples is transformed according to the audio transformation method to obtain augmented audio corresponding to each recorded audio. Based on the source lyric duration information of multiple initial samples, the augmented lyric duration information corresponding to each augmented audio is determined. The source score information of multiple initial samples is adjusted by a score transformation method that matches the audio transformation method to obtain augmented score information corresponding to each augmented audio. The corresponding augmented samples are formed based on each augmented audio, the augmented lyric duration information of each augmented audio, and the augmented score information of each augmented audio.

[0102] The audio transformation methods include at least one of pitch adjustment processing of the recorded audio or splicing processing of audio segments of the recorded audio. Pitch adjustment processing includes at least one of volume boosting or volume lowering processing. Volume boosting refers to raising the pitch of the recorded audio, while volume lowering refers to lowering the pitch of the recorded audio.

[0103] Musical score transformation methods include at least one of the following: adjusting the scale of the source musical score information or splicing musical score information fragments from the source musical score information. A scale is a group of notes arranged in ascending or descending order of pitch, based on a certain mode.

[0104] Specifically, the computer device adjusts the attribute parameters of the recorded audio from multiple initial samples according to an audio transformation method, obtaining augmented audio corresponding to each recorded audio. The computer device determines a duration transformation method that matches the audio transformation method, and processes the duration information of the source lyrics from the multiple initial samples using the duration transformation method, obtaining augmented lyrics duration information corresponding to each source lyrics duration information. The augmented lyrics duration information corresponds to the source lyrics duration information, the source lyrics duration information corresponds to the recorded audio, and the recorded audio corresponds to the augmented audio, thus obtaining the augmented lyrics duration information corresponding to the augmented audio. The duration transformation method includes at least one of renaming the source lyrics duration information or splicing segments of the source lyrics duration information.

[0105] The computer equipment determines a score transformation method that matches the audio transformation method. This method is then used to adjust the source score information of multiple initial samples, resulting in augmented score information corresponding to each source score. Since the augmented score information corresponds to the source score information, the source score information corresponds to the recorded audio, and the recorded audio corresponds to the augmented audio, the augmented score information corresponding to the augmented audio can be obtained.

[0106] The computer device uses the augmented audio, the duration of the augmented lyrics, and the augmented musical score to form a single augmented sample, thereby obtaining an augmented sample corresponding to each of the multiple initial samples. Each initial sample can correspond to multiple augmented samples.

[0107] In this embodiment, multiple initial samples can be augmented based on different audio transformation methods of the recorded audio. For example, pitch adjustment processing can be performed on the recorded audio of at least one initial sample, audio segment splicing processing can be performed on the recorded audio of at least one initial sample, and both pitch adjustment processing and audio segment splicing processing can be performed on the recorded audio of at least one initial sample. Alternatively, all the recorded audio of the initial samples can be augmented using the same audio transformation method. The processing of the source lyrics duration information and source musical score information of the initial samples is similar. It can be understood that the transformation method of the recorded audio of the initial sample matches the duration transformation method of the source lyrics duration information and the musical score transformation method of the source musical score information of the initial sample.

[0108] In this embodiment, the recorded audio of multiple initial samples is transformed according to an audio transformation method to obtain augmented audio corresponding to each recorded audio, enabling the processing of existing audio to obtain a greater number of audio samples. Based on the source lyric duration information of multiple initial samples, the augmented lyric duration information corresponding to each augmented audio is determined, ensuring that each augmented audio corresponds to the correct lyric duration annotation information, guaranteeing the accuracy of the lyrics and lyric duration in the transformed audio. By using a music score transformation method that matches the audio transformation method, the source music score information of multiple initial samples is adjusted, automatically modifying the existing music score information according to the music score transformation method that matches the audio transformation, realizing music score augmentation, and ensuring that each augmented audio corresponds to the correct music score information. Based on each augmented audio, each augmented lyric duration information, and each augmented music score information, corresponding augmented samples are formed, effectively increasing the number of training samples and improving the diversity of training samples. Furthermore, augmenting existing samples reduces the cost of manual sample collection and improves the efficiency of training sample collection.

[0109] In one embodiment, transforming the recorded audio of multiple initial samples according to an audio transformation method to obtain augmented audio corresponding to each recorded audio includes: performing pitch adjustment processing on the recorded audio of multiple initial samples to obtain augmented audio corresponding to each recorded audio.

[0110] Based on the source lyrics duration information of multiple initial samples, determine the augmented lyrics duration information corresponding to each augmented audio, including: using the source lyrics duration information of each recorded audio of multiple initial samples as the augmented lyrics duration information corresponding to the corresponding augmented audio.

[0111] By using a musical score transformation method that matches the audio transformation method, the source musical score information of multiple initial samples is adjusted to obtain augmented musical score information corresponding to each augmented audio. This includes: adjusting the pitch of the recorded audio of multiple initial samples according to the pitch adjustment process, and adjusting the scale of the notes in the source musical score information of multiple recorded audio to obtain augmented musical score information corresponding to each augmented audio.

[0112] Specifically, pitch adjustment processing includes at least one of pitch boosting or pitch reduction. A computer device can perform at least one of pitch boosting or pitch reduction processing on multiple initial sample recorded audio files to obtain adjusted audio corresponding to each recorded audio file. The computer device can then use this adjusted audio as augmented audio for the recorded audio.

[0113] For example, for multiple initial samples, the recorded audio of each initial sample can be processed by boosting, or by reducing the audio of each initial sample, or by both boosting and reducing the audio of each initial sample. Alternatively, the recorded audio of some initial samples can be boosted, and the recorded audio of others can be reduced.

[0114] In other embodiments, the computer device may splice together the audio segments of the adjusted audio to obtain augmented audio.

[0115] Pitch adjustment processing does not change the duration of the lyrics in the recorded audio. The duration of the original lyrics in the recorded audio can be directly used as the duration of the augmented lyrics in the corresponding augmented audio. This means that the duration of the original lyrics in each recorded audio from multiple initial samples can be renamed, thus using the duration of the original lyrics in the recorded audio as the duration of the augmented lyrics in the corresponding augmented audio.

[0116] Scale adjustment processing includes scale raising or scale lowering. Scale raising is matched with pitch sharpening, and scale lowering is matched with pitch flattening. For recorded audio undergoing pitch sharpening, the notes in the source score are raised to obtain augmented score information corresponding to the augmented audio. For recorded audio undergoing pitch flattening, the notes in the source score are lowered to obtain augmented score information corresponding to the augmented audio.

[0117] An augmented sample consists of the augmented audio obtained by raising the pitch of the recorded audio, the duration information of the augmented lyrics corresponding to the augmented audio, and the augmented musical score information obtained by raising the pitch of the source musical score information of the recorded audio. Similarly, an augmented sample consists of the augmented audio obtained by lowering the pitch of the recorded audio, the duration information of the augmented lyrics corresponding to the augmented audio, and the augmented musical score information obtained by lowering the pitch of the source musical score information of the recorded audio.

[0118] like Figure 3 The diagram shown illustrates the process of raising and lowering the pitch of notes in the source musical score information of the recorded audio in one embodiment.

[0119] The recorded audio is uniformly transposed up by a semitone using an audio processing tool. For example, the original note of the lyric "lái" in the source sheet music information in the figure is E♭, and after transposing up by a semitone, the note of "lái" becomes E. That is the sheet music information after transposing up by a semitone. One semitone is equal to 100 cents, and the transposing up of the recorded audio and the source sheet music information can be achieved by using the command "sox orig_audio_file pitch_shifted_up_audio_file pitch 100" of the audio processing tool SOX. The transposing up of the source sheet music information means raising the scale of the notes in the source sheet music information. As an audio processing software, SOX supports the mutual conversion of different audio format files and the adjustment of audio styles, such as pitch adjustment, speed adjustment, etc.

[0120] Similarly, the recorded audio is uniformly transposed down by a semitone using an audio processing tool. For example, the original note of the lyric "lái" in the source sheet music information in the figure is E♭, and after transposing down by a semitone, the note of "lái" becomes D. The transposing down of the recorded audio and the source sheet music information can be achieved by using the command "sox orig_audio_file pitch_shifted_down_audio_file pitch -100" of the audio processing tool SOX. The transposing down of the source sheet music information means lowering the scale of the notes in the source sheet music information.

[0121] Transposing up 1 key means raising the pitch of each frame of the audio by half a note. Transposing down 1 key means lowering the pitch of each frame of the audio by half a note.

[0122] In this embodiment, the recorded audio of multiple initial samples is respectively subjected to pitch adjustment processing to obtain augmented audio corresponding to each recorded audio, so that the obtained augmented audio covers a wider range of pitches, thereby being able to automatically expand the number of audio. The pitch adjustment processing only adjusts the pitch and has no influence on the lyrics and the duration of the lyrics. Then, the source lyric duration information of each recorded audio of the multiple initial samples is directly used as the augmented lyric duration information corresponding to the corresponding augmented audio, which can ensure the accuracy of the lyrics and the duration of the lyrics of the augmented audio. According to the pitch adjustment processing of the recorded audio of the multiple initial samples, the scale of the notes in the source sheet music information of the multiple recorded audio is adjusted, so that the sheet music adjustment of the recorded audio corresponds to the pitch adjustment of the recorded audio, thereby ensuring that each augmented audio corresponds to the correct sheet music information, effectively guaranteeing the mapping among the augmented audio, the augmented lyric duration information, and the augmented sheet music information, as well as the accuracy of the mapping relationship among the three, and further enhancing the effectiveness and accuracy of data augmentation.

[0123] In one embodiment, pitch adjustment processing is performed on the recorded audio of multiple initial samples to obtain augmented audio corresponding to each recorded audio. This includes: performing pitch adjustment processing on the recorded audio of multiple initial samples to obtain adjusted audio corresponding to each recorded audio; dividing each adjusted audio to obtain audio segments corresponding to each adjusted audio; and for each recorded audio, splicing the audio segments corresponding to the adjusted audio of the corresponding recorded audio in adjacent order to obtain multiple augmented audio of the corresponding recorded audio.

[0124] The source lyrics duration information of each recorded audio from multiple initial samples is used as the augmented lyrics duration information corresponding to the corresponding augmented audio. This includes: dividing the source lyrics duration information of the corresponding recorded audio into segments according to the segmentation process of each adjusted audio, to obtain the lyrics duration information segments corresponding to each audio segment of each adjusted audio; and splicing the lyrics duration information segments of each audio segment into segments according to the splicing process of each audio segment, to obtain the augmented lyrics duration information corresponding to each augmented audio.

[0125] Based on the pitch adjustment processing of the recorded audio from multiple initial samples, the notes of the source musical score information of the multiple recorded audio are subjected to scale adjustment processing to obtain augmented musical score information corresponding to each augmented audio. This includes: adjusting the notes of the source musical score information of the multiple recorded audio from multiple initial samples according to the pitch adjustment processing to obtain adjusted musical score information corresponding to each adjusted audio; dividing the adjusted musical score information of the corresponding adjusted audio according to the division processing to obtain musical score information segments corresponding to each audio segment of the corresponding adjusted audio; and splicing the musical score information segments of each audio segment according to the splicing processing to obtain augmented musical score information corresponding to each augmented audio.

[0126] In this embodiment, the pitch adjustment processing includes either pitch boosting or pitch reduction processing, and the adjusted audio includes at least one of pitch boosting or pitch reduction audio. Pitch boosting audio is the audio obtained by boosting the recorded audio, and pitch reduction audio is the audio obtained by reducing the pitch of the recorded audio. If the same recorded audio is subjected to both pitch boosting and pitch reduction processing, then the augmented audio corresponding to that recorded audio includes both the pitch boosting audio obtained by pitch boosting and the pitch reduction audio obtained by pitch reduction processing.

[0127] In this embodiment, after pitch adjustment processing of the recorded audio, the adjusted audio is then divided and segmented to obtain more augmented audio. Following the same division and segmentation of the adjusted audio, the source lyrics duration information of the recorded audio is divided and segmented in the same way, accurately obtaining the augmented lyrics duration information corresponding to each augmented audio. Based on the matching relationship between pitch adjustment processing and scale adjustment processing, the source musical score information of the recorded audio is adjusted accordingly. Then, following the same division and segmentation of the adjusted audio, the adjusted musical score information is divided and segmented, accurately obtaining the augmented musical score information corresponding to each augmented audio, thus effectively ensuring the accuracy of the mapping relationship between audio, lyrics duration information, and musical score information.

[0128] In one embodiment, such as Figure 4 As shown, the recorded audio of multiple initial samples is transformed according to the audio transformation method to obtain the augmented audio corresponding to each recorded audio, including steps S402-S404:

[0129] Step S402: Divide the recorded audio of multiple initial samples into segments to obtain the audio segments corresponding to each recorded audio.

[0130] Specifically, for each of the multiple initial samples, the pause points in the recorded audio of the initial sample are determined, and the recorded audio is divided and processed according to the pause points to obtain multiple audio segments corresponding to each recorded audio.

[0131] In this embodiment, each recorded audio file can be randomly divided to obtain a corresponding audio segment for each recorded audio file. Furthermore, the recorded audio can be randomly divided into a preset number of audio segments.

[0132] In other embodiments, the recorded audio can be divided into multiple audio segments according to a preset duration. For example, the recorded audio can be divided into multiple 2-5 second audio segments.

[0133] Step S404: For each recorded audio, the audio segments of the corresponding recorded audio are spliced ​​together in adjacent order to obtain multiple augmented audio segments of the corresponding recorded audio.

[0134] Specifically, for multiple audio segments recorded, the computer device concatenates at least two audio segments in an adjacent order to obtain multiple concatenated audio segments, which are then used as augmented audio segments. Following the same process, multiple augmented audio segments corresponding to each recorded audio segment can be obtained.

[0135] For example, if the recorded audio is divided into four audio segments, ABCD, with AB adjacent, BC adjacent, and CD adjacent, then AB can be spliced ​​together, BC can be spliced ​​together, CD can be spliced ​​together, ABC can be spliced ​​together, and BCD can be spliced ​​together to obtain multiple augmented audio segments.

[0136] In this embodiment, the computer device can splice multiple audio segments of recorded audio in an adjacent order to obtain augmented audio that meets different duration ranges. For example, the different duration ranges are 0-5 seconds, 5-8 seconds, and 8-13 seconds. The recorded audio is segmented according to the pause points to obtain audio segments of approximately 2-5 seconds each. Multiple segmented audio segments are then combined adjacently in sequence to form augmented audio of 0-5 seconds, 5-8 seconds, and 8-13 seconds, respectively. For example, if two adjacent audio segments after segmentation are 2 seconds and 4 seconds, they are combined to obtain a 6-second augmented audio, which meets the duration range of 5-8 seconds.

[0137] Based on the source lyric duration information of multiple initial samples, determine the augmented lyric duration information corresponding to each augmented audio, including steps S406-S408:

[0138] Step S406: According to the segmentation processing of the recorded audio of multiple initial samples, the source lyrics duration information of multiple initial samples is divided to obtain the lyrics duration information segment corresponding to each audio segment.

[0139] Specifically, the computer device divides the recorded audio from multiple initial samples into segments, and performs the same segmentation process on the source lyrics duration information of each recorded audio segment to obtain the lyrics duration information segment corresponding to each audio segment.

[0140] Step S408: According to the splicing process of each audio segment, the lyrics duration information segments of each audio segment are spliced ​​together to obtain the augmented lyrics duration information corresponding to each augmented audio segment.

[0141] Specifically, the computer device can determine the lyric duration information segments corresponding to each of the multiple audio segments constituting the augmented audio, and splice the multiple lyric duration information segments in an adjacent order according to the splicing process of the multiple audio segments to obtain the augmented lyric duration information corresponding to each augmented audio.

[0142] By using a musical score transformation method that matches the audio transformation method, the source musical score information of multiple initial samples is adjusted to obtain augmented musical score information corresponding to each augmented audio, including steps S410-S412:

[0143] Step S410: According to the division processing of the recorded audio of multiple initial samples, the source score information of multiple initial samples is divided to obtain the score information segment corresponding to each audio segment.

[0144] Specifically, the computer equipment divides the recorded audio from multiple initial samples into segments, and performs the same segmentation process on the source score information of each recorded audio segment to obtain the score information segment corresponding to each audio segment.

[0145] Step S412: According to the splicing process of each audio segment, the musical score information segments of each audio segment are spliced ​​together to obtain the augmented musical score information corresponding to each augmented audio segment.

[0146] Specifically, the computer device can determine the musical score information segments corresponding to each of the multiple audio segments that constitute the augmented audio, and splice the multiple audio segments in an adjacent order according to the splicing process to obtain the augmented musical score information corresponding to each augmented audio.

[0147] In this embodiment, the recorded audio of multiple initial samples is divided into segments to obtain audio segments corresponding to each recorded audio. For each recorded audio, the audio segments are spliced ​​together in adjacent order to obtain multiple augmented audio segments. By dividing the recorded audio and splicing the audio segments, more augmented audio segments can be obtained, effectively achieving audio augmentation. Following the division of the recorded audio of multiple initial samples, the source lyrics duration information of the multiple initial samples is divided, ensuring a one-to-one correspondence between each audio segment and each lyrics duration information segment. Following the splicing process of each audio segment, the lyrics duration information segments of each audio segment are spliced ​​together, ensuring a one-to-one correspondence between the augmented audio and the augmented lyrics duration information. Following the division of the recorded audio of multiple initial samples, the source musical score information of the multiple initial samples is divided, ensuring an accurate mapping relationship between each audio segment and each musical score information segment. By splicing together the audio segments, the musical score information segments of each audio segment are concatenated, ensuring an accurate mapping between the augmented audio and the augmented musical score information. This effectively augments the samples, resulting in more training samples. Furthermore, the augmented samples obtained through segmentation and splicing acquire more contextual information, thereby enhancing the robustness of the song synthesis model when receiving input information.

[0148] In one embodiment, such as Figure 5 As shown, the initial model for song synthesis is obtained by pre-training the model based on the initial sample set and the augmented sample set, including:

[0149] Step S502: Obtain sample audio, sample lyric duration information corresponding to the sample audio, and sample musical score information corresponding to the sample audio from the set consisting of the initial sample set and the augmented sample set.

[0150] Specifically, the computer device acquires sample audio from the set consisting of the initial sample set and the augmented sample set, and acquires the sample lyrics duration information corresponding to the sample audio, as well as the sample musical score information corresponding to the sample audio.

[0151] When the acquired sample audio is recorded audio, the duration information of the source lyrics corresponding to the recorded audio is used as the duration information of the sample lyrics corresponding to the sample audio, and the duration information of the source musical score corresponding to the recorded audio is used as the duration information of the sample musical score corresponding to the sample audio.

[0152] When the acquired sample audio is augmented audio, the duration information of the augmented lyrics corresponding to the augmented audio is used as the duration information of the sample lyrics corresponding to the sample audio, and the duration information of the augmented musical score corresponding to the augmented audio is used as the duration information of the sample musical score corresponding to the sample audio.

[0153] Step S504: Perform feature encoding based on the sample lyrics duration information and sample musical score information of the sample audio to obtain sample encoded features.

[0154] Specifically, computer equipment can extract features from the sample lyrics duration information and sample musical score information of the sample audio, respectively, to obtain sample lyrics features and sample lyrics duration features corresponding to the sample lyrics duration information, as well as sample musical score features corresponding to the sample musical score information. Sample lyrics features may include phonemes of the lyrics, and may also include syllables. Sample lyrics duration features may include the phoneme duration corresponding to each phoneme, and may also include the syllable duration corresponding to each syllable. Musical score features include notes, and may also include beats.

[0155] The computer equipment concatenates the sample lyrics features and sample musical score features, and performs feature encoding on the concatenated features to obtain the corresponding sample encoded features.

[0156] Step S506: Based on the sample lyrics duration features of the sample lyrics duration information, extend the duration features of the sample coding features to obtain the extended coding features of the sample duration.

[0157] Specifically, the computer equipment extends the duration features of the sample encoding features according to the duration features of the sample lyrics to obtain the corresponding extended duration encoding features of the samples.

[0158] In this embodiment, the duration feature of the sample lyrics includes the duration of phonemes. The computer device then expands the duration feature of each phoneme in the sample lyrics duration information to obtain the corresponding extended duration encoding feature of the sample encoding feature.

[0159] In other embodiments, the duration feature of the sample lyrics includes syllable duration. The computer device then expands the duration feature of each syllable in the sample lyrics duration information according to the syllable duration of each syllable to obtain the corresponding extended duration encoding feature of the sample.

[0160] Step S508: Extract the sample timbre features of the sample audio, concatenate the sample timbre features and the sample duration extended coding features, and then extract the acoustic features to obtain the predicted spectral features.

[0161] Specifically, the computer equipment extracts features from the sample audio to obtain the corresponding sample timbre features. After concatenating the sample timbre features and the sample duration extended coding features, the computer equipment extracts acoustic features from the concatenated features to extract spectral features and obtain the predicted spectral features.

[0162] Step S510: Synthesize a predicted song based on the predicted spectral features, and construct a target loss function based on the synthesis loss between the predicted song and the sample audio.

[0163] Here, the synthesis loss refers to the overall loss between the predicted song and the sample audio.

[0164] Specifically, the computer device can synthesize a predicted song based on predicted spectral features, using sample audio as the corresponding ground truth label for the predicted song. The computer device calculates the difference between the predicted song and the corresponding sample audio; this difference is the synthesis loss. A target loss function is constructed based on the synthesis loss between the predicted song and the sample audio.

[0165] Step S512: Perform model pre-training based on the target loss function to obtain the initial model for song synthesis.

[0166] Specifically, the computer equipment pre-trains the model based on the target loss function, adjusts the model parameters based on the target loss during pre-training, and continues training until the pre-training stopping condition is met, resulting in a pre-trained initial model for song synthesis. This pre-trained initial model is used to synthesize corresponding songs based on lyrics and sheet music information. Meeting the pre-training stopping condition can be achieved by reaching a preset number of training iterations, a preset number of iterations, or the loss value being less than or equal to a loss threshold. The loss value refers to the target loss calculated by the target loss function.

[0167] For example, if the target loss calculated by the target loss function is greater than the loss threshold, the model parameters are adjusted and training continues until the target loss during pre-training is less than or equal to the loss threshold, at which point pre-training stops, resulting in a pre-trained initial model for song synthesis. Alternatively, the model parameters are adjusted based on the target loss and training continues until the preset number of training iterations or the preset number of iterations during pre-training is reached, at which point a pre-trained initial model for song synthesis is obtained.

[0168] In this embodiment, sample audio, corresponding sample lyrics duration information, and corresponding sample musical score information are obtained from a set consisting of an initial sample set and an augmented sample set. This allows the initial and augmented samples to be used as training samples for model training, enabling the model to be trained using richer training samples and improving its robustness. Feature encoding is performed based on the sample lyrics duration information and sample musical score information of the sample audio to obtain sample encoded features. The duration features of the sample encoded features are then expanded according to the sample lyrics duration features to obtain sample duration extended encoded features. Sample timbre features are extracted from the sample audio. The sample timbre features and sample duration extended encoded features are concatenated and then used for acoustic feature extraction to obtain predicted spectral features. A predicted song is synthesized based on the predicted spectral features, thus obtaining the song predicted by the model. A target loss function is constructed based on the synthesis loss between the predicted song and the real sample audio. Model pre-training is performed based on the target loss function to accurately obtain the initial model for song synthesis. Furthermore, training with augmented samples can greatly expand the training corpus for singing synthesis, thereby improving the model's stability, expressiveness, and practicality when faced with different combinations of lyrics and melodies.

[0169] In one embodiment, the method further includes: performing duration prediction processing based on the sample lyrics duration features of the sample lyrics duration information, the sample musical score features of the sample musical score information, and the sample timbre features to obtain the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information; and determining the phoneme duration loss between the predicted phoneme duration of each phoneme and the sample phoneme duration of each phoneme in the sample lyrics duration information.

[0170] The target loss function is constructed based on the synthesis loss between the predicted song and the sample audio, including: determining the synthesis loss between the predicted song and the sample audio; and constructing the target loss function based on the phoneme duration loss and the synthesis loss.

[0171] The phoneme duration loss refers to the difference between the model's predicted duration for each phoneme and the actual duration of each phoneme. The sample phoneme duration is the actual duration of the phoneme.

[0172] Specifically, the computer device can extract features from the sample audio, specifically the duration information of the sample lyrics and the sample musical score, to obtain sample lyric features corresponding to the duration information of the sample lyrics and sample musical score features corresponding to the sample musical score. The sample lyric features may include phonemes and syllables. The sample lyric duration features may include the phoneme duration corresponding to each phoneme and the syllable duration corresponding to each syllable. The musical score features include notes and may include beats. The computer device also extracts features from the sample audio to obtain the corresponding sample timbre features.

[0173] The computer device performs duration prediction processing based on the features of the sample lyrics, the sample musical score, and the sample timbre, obtaining the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information. Further, the computer device performs duration prediction processing based on the phonemes, notes, beats, and sample timbre features of the sample audio, obtaining the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information.

[0174] The computer device determines the sample phoneme duration corresponding to each phoneme in the sample lyrics duration information, calculates the difference between the predicted phoneme duration and the corresponding sample phoneme duration for each phoneme, and thus obtains the phoneme duration loss for each phoneme. The computer device calculates the synthesis loss between the predicted song and the sample audio, and constructs a target loss function based on the phoneme duration loss and the synthesis loss. Further, the computer device can sum the phoneme duration loss and the synthesis loss to obtain the target loss function. Alternatively, the computer device can obtain the weights corresponding to the phoneme duration loss and the synthesis loss, and perform a weighted summation of the phoneme duration loss, the synthesis loss, and their respective weights to obtain the target loss function.

[0175] In this embodiment, the phoneme duration loss characterizes the difference between the model's predicted duration and the actual duration of each phoneme. Duration prediction is performed based on the sample lyrics features of the sample lyrics duration information, the sample musical score features of the sample musical score information, and the sample timbre features. This yields the predicted phoneme duration for each phoneme in the sample lyrics duration information. Therefore, the model's loss in predicting the duration of each phoneme can be determined based on the difference between the predicted duration and the actual duration of each phoneme in the sample lyrics duration information. A target loss function is constructed by combining the phoneme duration loss and the synthesis loss. This takes into account the influence of local factors in phoneme duration prediction and global factors in song synthesis on the model. Training the model using both local and global losses further improves the model's accuracy in phoneme duration prediction, thereby enhancing the accuracy of the initial song synthesis model.

[0176] In one embodiment, the method further includes: determining the predicted syllable duration corresponding to each syllable in the sample lyrics duration information based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information; and determining the syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information.

[0177] Based on phoneme duration loss and synthesis loss, a target loss function is constructed, including: constructing a target loss function based on phoneme duration loss, syllable duration loss and synthesis loss.

[0178] Among them, the syllable duration loss characterizes the difference between the predicted duration of each syllable and the actual duration of each syllable by the model.

[0179] Specifically, for each syllable in the sample lyrics duration information, the computer device determines the phonemes that constitute each syllable, sums the predicted phoneme durations corresponding to the phonemes constituting the syllable, and sums the sample phoneme durations corresponding to the phonemes constituting the syllable to obtain the sample syllable duration. Following the same processing method, the computer device can obtain the predicted syllable duration and the sample syllable duration for each syllable.

[0180] The computer device can calculate the difference between the predicted syllable duration and the corresponding sample syllable duration for each syllable, thus obtaining the syllable duration loss for each syllable. The computer device calculates the synthesis loss between the predicted song and the sample audio, and constructs a target loss function based on the duration losses of each phoneme, each syllable, and the synthesis loss. Further, the computer device can sum the duration losses of each phoneme, each syllable, and the synthesis loss to obtain the target loss function. Alternatively, the computer device can obtain the weights corresponding to the phoneme duration loss, syllable duration loss, and synthesis loss, and perform a weighted summation of these weights to obtain the target loss function.

[0181] In this embodiment, based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information, the predicted syllable duration corresponding to each syllable in the sample lyrics duration information is determined. The syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information is determined. A target loss function is constructed by combining phoneme duration loss, syllable duration loss, and synthesis loss. This can take into account the influence of multiple local factors such as phoneme duration prediction and syllable duration prediction, as well as global factors of song synthesis on the model. Thus, by combining multiple local losses and global losses to train the model, the accuracy of the model in phoneme duration prediction and syllable duration prediction can be further improved, thereby improving the accuracy of the initial model for song synthesis.

[0182] In one embodiment, the method further includes: performing gradient inversion on the sample encoded features, and classifying based on the features obtained from the gradient inversion to obtain a classification result for the sample audio; and determining the adversarial loss between the classification result and the classification label of the sample audio.

[0183] The objective loss function is constructed based on the synthesis loss between the predicted song and the sample audio, including: determining the synthesis loss between the predicted song and the sample audio; and constructing the objective loss function based on the adversarial loss and the synthesis loss.

[0184] This refers to the difference between the sample encoding features extracted by the adversarial loss representation model and the sample encoding features expected to be extracted. The classification label of the sample audio refers to the true classification of the sample audio, that is, the true category of the audio sung by which object the sample audio belongs.

[0185] Specifically, the computer device performs gradient inversion processing on the sample encoding features to obtain gradient inverted features. Based on these gradient inverted features, the computer device classifies the sample audio, obtaining a classification result. This classification result characterizes the classification of the sound source of the sample audio, and is used to determine which object the sound source belongs to, i.e., to determine which object sang the sample audio.

[0186] The computer device calculates the difference between the classification result and the classification label of the sample audio, thus obtaining the adversarial loss. The computer device calculates the synthesis loss between the predicted song and the sample audio, and constructs a target loss function based on the adversarial and synthesis losses. Further, the computer device can sum the adversarial and synthesis losses to obtain the target loss function. Alternatively, the computer device can obtain the weights corresponding to the adversarial and synthesis losses, and perform a weighted sum of the adversarial loss, synthesis loss, and their respective weights to obtain the target loss function.

[0187] In this embodiment, the computer device inputs the sample encoding features into an adversarial object classifier, performs gradient inversion processing on the sample encoding features through the object classifier, and performs classification based on the features obtained from the gradient inversion processing, outputting the classification result.

[0188] In this embodiment, adversarial loss reflects the difference between the sample coding features extracted by the model and the expected sample coding features. By performing gradient inversion on the sample coding features and classifying them based on the resulting features, the classification result of the sample audio is obtained. The adversarial loss between the classification result and the classification label of the sample audio is determined to judge whether the sample coding features extracted by the model meet expectations. This allows for adjustment of model parameters to ensure the extracted sample coding features achieve the desired result. Combining adversarial loss and synthesis loss to construct a target loss function takes into account the impact of multiple factors on the model, such as the loss in extracting sample coding features and the overall loss in song synthesis. Training the model with multiple losses further improves the accuracy of the model in coding features, thereby improving the accuracy of song synthesis.

[0189] In one embodiment, the method further includes: extracting sample spectral features of the sample audio and determining the spectral loss between the predicted spectral features and the sample spectral features;

[0190] Construct a target loss function based on the synthesis loss between the predicted song and the sample audio, including:

[0191] Determine the synthesis loss between the predicted song and the sample audio; construct the target loss function based on the spectral loss and the synthesis loss.

[0192] The spectral loss characterizes the difference between the spectral features predicted by the model and the actual spectral features. The sample spectral features serve as the true spectral labels.

[0193] Specifically, the computer device can extract features from the sample audio to obtain the corresponding sample spectral features. Using these sample spectral features as the true spectral labels, the computer device calculates the spectral loss between the predicted spectral features and the sample spectral features, as well as the synthesis loss between the predicted song and the sample audio. Based on the spectral loss and synthesis loss, a target loss function is constructed. Further, the computer device can sum the spectral loss and synthesis loss to obtain the target loss function. Alternatively, the computer device can obtain the weights corresponding to the spectral loss and synthesis loss, and perform a weighted summation of the spectral loss, synthesis loss, and their respective weights to obtain the target loss function.

[0194] In this embodiment, the spectral loss reflects the difference between the model's predicted spectral features and the actual spectral features. Sample spectral features are extracted from the sample audio, and the spectral loss between the predicted and sample spectral features is determined to assess whether the model's predicted spectral features meet expectations. This allows for adjustment of model parameters to ensure the predicted spectral features achieve the desired outcome. Constructing a target loss function by combining spectral loss and synthesis loss takes into account the impact of multiple factors on the model, such as local losses in spectral prediction and overall losses in song synthesis. Training the model using both local and global losses further improves the model's accuracy in spectral prediction, thereby enhancing the accuracy of song synthesis.

[0195] In one embodiment, constructing a target loss function based on spectral loss and synthesis loss includes: constructing a target loss function based on phoneme duration loss, spectral loss, and synthesis loss.

[0196] In one embodiment, constructing a target loss function based on spectral loss and synthesis loss includes: constructing a target loss function based on syllable duration loss, spectral loss, and synthesis loss.

[0197] In one embodiment, constructing a target loss function based on spectral loss and synthesis loss includes: constructing a target loss function based on phoneme duration loss, syllable duration loss, spectral loss, and synthesis loss.

[0198] In one embodiment, the initial song synthesis model includes an initial duration model, an initial acoustic model, and an initial vocoder; training the initial song synthesis model based on timbre features yields a song synthesis model, including:

[0199] Based on the timbre characteristics of the target sound source, the parameters of the initial duration model, the initial acoustic model, and the initial vocoder are adjusted to obtain a song synthesis model that matches the timbre of the target sound source.

[0200] Specifically, the initial song synthesis model includes an initial duration model, an initial acoustic model, and an initial vocoder. The initial duration model is used to predict the phoneme duration of each phoneme. The initial acoustic model is used to expand the duration features and extract the acoustic features to output the corresponding spectral features. The initial vocoder is used to synthesize the corresponding song based on the spectral features. The trained song synthesis model includes the duration model, the acoustic model, and the vocoder.

[0201] In this embodiment, the computer device can acquire multiple singing audios corresponding to the target sound source, as well as the source lyrics duration information corresponding to each singing audio, and the source musical score information corresponding to each singing audio. The singing audio can be recorded from the initial sample set, or it can be audio that does not belong to the initial sample set or the augmented sample set.

[0202] In one embodiment, the computer device acquires the audio corresponding to the target sound source, the lyrics duration information corresponding to the audio, and the musical score information corresponding to the audio; extracts timbre features based on the audio of the target sound source, extracts lyrics features and lyrics duration features based on the lyrics duration information, and extracts musical score features based on the musical score information.

[0203] An initial duration model is used to predict duration based on lyric features, musical score features, and timbre features, resulting in the predicted phoneme duration for each phoneme in the lyric duration information. The phoneme duration loss between the predicted phoneme duration and the actual phoneme duration in the lyric duration information is then determined.

[0204] The initial duration model determines the predicted syllable duration for each syllable in the lyrics duration information based on the predicted phoneme duration for each phoneme in the lyrics duration information. The syllable duration loss between the predicted syllable duration for each syllable and the syllable duration for each syllable in the lyrics duration information is then determined.

[0205] The initial acoustic model encodes features based on the lyrics duration and musical score information of the target sound source audio, obtaining encoded features. The initial acoustic model then expands the encoded features based on the lyrics duration features, resulting in duration-extended encoded features. Finally, the initial acoustic model concatenates the timbre features and duration-extended encoded features for acoustic feature extraction, yielding predicted spectral features. The spectral features of the audio are extracted, and the spectral loss between the predicted and actual spectral features is determined.

[0206] A predicted song is synthesized based on the predicted spectral features using an initial vocoder. The synthesis loss between the predicted song and the audio of the target sound source is determined.

[0207] Based on phoneme duration loss, syllable duration loss, spectrum loss, and synthesis loss, a target loss function is constructed for the initial song synthesis model. The initial song synthesis model is then trained based on the target loss function to obtain a song synthesis model that matches the timbre of the target sound source.

[0208] In this embodiment, the model is pre-trained using a large number of training samples, including initial and augmented samples. Based on the initial model for song synthesis, the parameters of the initial duration model, initial acoustic model, and initial vocoder are adjusted according to the timbre characteristics of the target sound source. The model parameters can be fine-tuned using personal singing data, so that the resulting song synthesis model matches the individual's timbre. This allows for the synthesis of songs sung in a personal timbre using any lyrics and sheet music, thus achieving personal timbre customization.

[0209] In this embodiment, the training process for training the initial song synthesis model based on the timbre features of the target sound source is basically the same as the pre-training process. The difference is that pre-training uses a large number of samples from different sound sources for training, while the initial song synthesis model is trained using samples from a single target object to obtain a song synthesis model that matches the timbre of the target object. The target object is the corresponding target sound source.

[0210] In one embodiment, a method for training a song synthesis model is provided, applied to a computer device, comprising:

[0211] Obtain an initial sample set, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio.

[0212] Next, the recorded audio of multiple initial samples is subjected to pitch adjustment processing to obtain augmented audio corresponding to each recorded audio. The duration of the source lyrics of each recorded audio of multiple initial samples is used as the duration of the augmented lyrics corresponding to the augmented audio. According to the pitch adjustment processing of the recorded audio of multiple initial samples, the notes of the source musical score information of multiple recorded audio are subjected to scale adjustment processing to obtain augmented musical score information corresponding to each augmented audio. Based on each augmented audio, the duration of the augmented lyrics of each augmented audio, and the augmented musical score information of each augmented audio, corresponding augmented samples are formed.

[0213] Optionally, the recorded audio of multiple initial samples is divided to obtain audio segments corresponding to each recorded audio. For each recorded audio, the audio segments of the corresponding recorded audio are spliced ​​together in adjacent order to obtain multiple augmented audios of the corresponding recorded audio. Based on the division of the recorded audio of multiple initial samples, the source lyrics duration information of multiple initial samples is divided to obtain lyrics duration information segments corresponding to each audio segment. Based on the splicing process of each audio segment, the lyrics duration information segments of each audio segment are spliced ​​together to obtain augmented lyrics duration information corresponding to each augmented audio. Based on the division of the recorded audio of multiple initial samples, the source musical score information of multiple initial samples is divided to obtain musical score information segments corresponding to each audio segment. Based on the splicing process of each audio segment, the musical score information segments of each audio segment are spliced ​​together to obtain augmented musical score information corresponding to each augmented audio. Based on each augmented audio, the augmented lyrics duration information of each augmented audio, and the augmented musical score information of each augmented audio, corresponding augmented samples are formed.

[0214] Furthermore, each augmented sample is used to form an augmented sample set. From the set consisting of the initial sample set and the augmented sample set, sample audio, the duration of the corresponding sample lyrics, and the corresponding sample sheet music are obtained. When the obtained sample audio is recorded audio, the duration of the source lyrics corresponding to the recorded audio is used as the duration of the sample lyrics corresponding to the sample audio, and the duration of the source sheet music corresponding to the recorded audio is used as the duration of the sample sheet music corresponding to the sample audio. When the obtained sample audio is augmented audio, the duration of the augmented lyrics corresponding to the augmented audio is used as the duration of the sample lyrics corresponding to the sample audio, and the duration of the augmented sheet music corresponding to the augmented audio is used as the duration of the sample sheet music corresponding to the sample audio.

[0215] Next, the model is pre-trained, including:

[0216] Based on the sample lyrics duration information corresponding to the sample lyrics features, the sample musical score features, and the sample timbre features of the sample audio, duration prediction processing is performed to obtain the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information; the phoneme duration loss between the predicted phoneme duration of each phoneme and the sample phoneme duration of each phoneme in the sample lyrics duration information is determined.

[0217] Furthermore, based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information, the predicted syllable duration corresponding to each syllable in the sample lyrics duration information is determined; the syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information is determined.

[0218] Next, feature encoding is performed based on the duration information of the sample lyrics and the sample musical score information of the sample audio to obtain the sample encoded features.

[0219] Next, gradient inversion is performed on the encoded features of the samples, and classification is performed based on the features obtained from the gradient inversion to obtain the classification results of the sample audio; the adversarial loss between the classification results and the classification labels of the sample audio is determined.

[0220] Furthermore, based on the duration features of the sample lyrics, the duration features of the sample coding features are extended to obtain the extended coding features of the sample duration; the sample timbre features and the extended coding features of the sample duration are concatenated and then acoustic features are extracted to obtain the predicted spectral features; the sample spectral features of the sample audio are extracted, and the spectral loss between the predicted spectral features and the sample spectral features is determined.

[0221] Next, a predicted song is synthesized based on the predicted spectral features, and the synthesis loss between the predicted song and the sample audio is determined.

[0222] Furthermore, a pre-trained target loss function is constructed based on phoneme duration loss, syllable duration loss, adversarial loss, spectral loss, and synthesis loss; the model is pre-trained based on the target loss function to obtain the initial model for song synthesis.

[0223] Next, the initial model for song synthesis is trained, including:

[0224] Obtain the audio corresponding to the target sound source, the lyrics duration information corresponding to the audio, and the musical score information corresponding to the audio; extract timbre features based on the audio of the target sound source, extract lyrics features and lyrics duration features based on the lyrics duration information, and extract musical score features based on the musical score information.

[0225] Duration prediction is performed based on lyric features, musical score features, and timbre features to obtain the predicted phoneme duration for each phoneme in the lyric duration information; the phoneme duration loss between the predicted phoneme duration for each phoneme and the phoneme duration for each phoneme in the lyric duration information is determined.

[0226] Based on the predicted phoneme duration corresponding to each phoneme in the lyrics duration information, determine the predicted syllable duration corresponding to each syllable in the lyrics duration information; determine the syllable duration loss between the predicted syllable duration of each syllable and the syllable duration of each syllable in the lyrics duration information.

[0227] Next, feature encoding is performed based on the lyrics duration information and musical score information of the target sound source audio to obtain encoded features; according to the lyrics duration feature of the lyrics duration information, the duration feature of the encoded features is extended to obtain duration extended encoded features; the timbre features and duration extended encoded features are concatenated and then acoustic features are extracted to obtain predicted spectral features; the spectral features of the audio are extracted, and the spectral loss between the predicted spectral features and the spectral features is determined.

[0228] Next, a predicted song is synthesized based on the predicted spectral features, and the synthesis loss between the predicted song and the target sound source audio is determined.

[0229] Furthermore, based on phoneme duration loss, syllable duration loss, spectrum loss, and synthesis loss, a target loss function is constructed for the initial song synthesis model. The initial song synthesis model is then trained based on the target loss function to obtain a song synthesis model that matches the timbre of the target sound source.

[0230] In this embodiment, pitch adjustment processing is performed on the recorded audio of multiple initial samples to obtain augmented audio corresponding to each recorded audio. This allows the augmented audio to cover a wider range of pitches, thereby automatically expanding the number of audio samples. Since the pitch adjustment processing only adjusts pitch and has no effect on lyrics or duration, the source lyric duration information of each recorded audio from the multiple initial samples is directly used as the augmented lyric duration information for the corresponding augmented audio, ensuring the accuracy of the lyrics and duration of the augmented audio. Following the pitch adjustment processing of the recorded audio of the multiple initial samples, scale adjustment processing is performed on the notes of the source musical score information of the multiple recorded audio samples. This ensures that the musical score adjustment of the recorded audio corresponds to the pitch adjustment of that recorded audio, thus ensuring that each augmented audio corresponds to the correct musical score information. This effectively guarantees the accuracy of the mapping between augmented audio, augmented lyric duration information, and augmented musical score information, as well as the mapping relationship among the three, thereby improving the effectiveness and accuracy of data augmentation.

[0231] The recorded audio from multiple initial samples is divided into segments, each corresponding to a specific audio fragment. For each recorded audio, these segments are then concatenated in adjacent order to generate multiple augmented audio files. This process of segmenting and concatenating audio fragments allows for the generation of more augmented audio files, effectively achieving audio augmentation. Following the segmentation process, the source lyrics duration information of the initial samples is divided, ensuring a one-to-one correspondence between each audio fragment and a specific lyrics duration information fragment. The lyrics duration information fragments of each audio fragment are then concatenated, ensuring a one-to-one correspondence between the augmented audio files and the augmented lyrics duration information. Similarly, the source musical score information of the initial samples is divided, ensuring an accurate mapping between each audio fragment and a specific musical score information fragment. Finally, the musical score information fragments of each audio fragment are concatenated, ensuring an accurate mapping between the augmented audio files and the augmented musical score information, thus effectively augmenting the samples and obtaining more training samples. Furthermore, the augmented samples obtained through segmentation and splicing can acquire more contextual information, thereby enabling the song synthesis model to have greater robustness when receiving input information.

[0232] By extracting sample audio, corresponding sample lyric duration information, and corresponding sample musical score information from the initial and augmented sample sets, the model can be trained using both initial and augmented samples. This allows for training with a richer set of training data, improving the model's robustness. Furthermore, training with augmented samples significantly expands the training corpus for song synthesis, enhancing the model's stability, expressiveness, and practicality when faced with different combinations of lyrics and melodies.

[0233] Phoneme duration loss reflects the difference between the model's predicted duration and the actual duration of each phoneme; syllable duration loss reflects the difference between the model's predicted duration and the actual duration of each syllable; adversarial loss reflects the difference between the sample coding features extracted by the model and the sample coding features expected to be extracted; spectral loss reflects the difference between the model's predicted spectral features and the actual spectral features; and synthesis loss reflects the overall loss between the model's predicted song and the sample audio. Combining phoneme duration loss, syllable duration loss, adversarial loss, spectral loss, and synthesis loss to construct a target loss function takes into account the influence of multiple local factors such as phoneme duration prediction, syllable duration prediction, coding feature extraction, and spectral feature prediction, as well as global factors of song synthesis. Therefore, training the model using multiple local and global losses can further improve the model's prediction accuracy in various aspects, thereby improving the accuracy of the initial song synthesis model.

[0234] Based on the initial model of song synthesis, the initial model of song synthesis is fine-tuned using personal sound source data to accurately obtain a song synthesis model that matches the target timbre of the target sound source, thereby effectively achieving the timbre customization effect of song synthesis.

[0235] In one embodiment, a pitch-based sample augmentation method and a sequence splicing-based sample augmentation method are provided. Sample augmentation, also known as data augmentation, includes:

[0236] Data preprocessing:

[0237] First, a certain amount of multi-person singing dataset, i.e., the initial sample set, was purchased from an external vendor. Each person's initial sample for each song included a recorded audio recording of their vocals, a duration annotation file, and a MusicXML file representing the musical score. The duration annotation file contains the source lyrics duration information of the recorded audio, including the lyrics and duration annotations for each lyric, specifically including annotations for the duration of each phoneme. The MusicXML file represents the source musical score information of the recorded audio.

[0238] For this initial sample, the audio is segmented according to the pause points in the recorded audio, resulting in audio segments of approximately 2 to 5 seconds in length. For each audio segment, corresponding duration annotation data and musical score data are obtained, namely, lyric duration information segments and musical score information segments. These segmented audio segments, lyric duration information segments, and musical score information segments are called the base data, and the results trained using this base data are called the base results.

[0239] Pitch-based data augmentation methods:

[0240] During the audio processing stage, the original key's vocal recordings are first aligned using the SOX audio tool to uniformly raise the pitch by a semitone, resulting in the corresponding augmented audio. Similarly, to increase the quantity and richness of audio, the original key's vocal recordings can also be uniformly lowered by a semitone to obtain the corresponding augmented audio.

[0241] Data augmentation for source lyric duration information and source musical score information:

[0242] Since the audio tool only modifies the pitch of the recorded audio and does not change the duration of each lyric, there is no need to modify the lyrics and the duration of each lyric in the source lyric duration information. You can directly copy the duration marked in the source.

[0243] Because the recorded audio was processed by raising and lowering a semitone, the notes in the source score also need to be modified to obtain the corresponding augmented score information for each augmented audio.

[0244] Since the representation of a semitone sharp and semitone flat is not unique, this embodiment specifies a set of modification rules to ensure the diversity of different note combinations through randomness, thereby improving the richness of musical notation and obtaining musical notation information after a semitone sharp and semitone flat.

[0245] The singing synthesis model proposed in this embodiment is based on Transformer, a model entirely based on fully connected networks without recurrent neural networks. It differs from the recursive, ordered modeling of recurrent neural networks and the local modeling of convolutional neural networks. Transformer is a deep learning model entirely based on self-attention mechanisms, which allow it to perform global modeling and use positional encoding to distinguish different positions within a sequence. The singing synthesis model in this embodiment is a generative model based on Transformer. For this type of model, its computational complexity is quadratic with the sequence length. Furthermore, even if two sequences of different lengths belong to the same subsequence, the feature representations obtained after processing by this model will be different. Based on this characteristic of the Transformer structure, this embodiment designs a data augmentation method based on sequence length concatenation:

[0246] Since the basic data is segmented entirely based on pause times, the basic data can be spliced ​​and combined. The splicing method can be to define three ranges: 0-5 seconds, 5-8 seconds, and 8-13 seconds. Then, audio segments in the basic data are combined sequentially to obtain augmented audio within these three ranges. Using the same segmentation and combination method, the source lyrics duration information and source musical score information are also segmented and combined to obtain augmented lyrics duration information and augmented musical score information within the same three ranges. This results in augmented lyrics duration information and augmented musical score information for each augmented audio. Similarly, audio obtained by raising or lowering a semitone can also be segmented and spliced ​​to obtain more augmented audio. The processing of the lyrics duration information and musical score information corresponding to the audio obtained by raising or lowering a semitone is similar.

[0247] Based on the newly obtained three augmented sample sets, since the input phonemes and scores of each augmented audio have different contextual information, the three augmented sample sets are combined with the basic dataset to train the singing synthesis model. Compared with using only the basic data, the model trained on the large dataset after data augmentation can learn very rich contextual information, thus enabling the singing synthesis model to have higher robustness when receiving new input information.

[0248] Data augmentation based on sequence concatenation can be applied to song synthesis, and all generative models based on the Transformer architecture can use this data augmentation method to improve the context richness problem encountered by the model during training, thereby improving the robustness of the model when applied.

[0249] The data augmentation in this embodiment can be applied not only to singing synthesis, but also to generative models based on fully connected structures like Transformers or classification models of variable-length sequences. For example, it can be used for song recognition and song classification.

[0250] like Figure 6 The diagram shown illustrates the overall architecture for model pre-training in one embodiment. After obtaining the augmented sample set, the model can be pre-trained using both the augmented and initial sample sets to obtain an average model, i.e., the initial model for song synthesis. This model is then fine-tuned on a single-person singing dataset. Specific loss functions are required during both training and fine-tuning. The resulting song synthesis model can then be used to synthesize the final target song. The following sections will describe the model structure and training process:

[0251] The model structure consists of three parts: a duration model, an acoustic model, and a vocoder.

[0252] (1) Duration Model

[0253] Unlike Automatic Speech Recognition (ASR), which allows for diverse speaking styles, song synthesis requires the duration of each note to adhere to the constraints of the musical score. Since song synthesis often only provides lyrics and score, without specifying the precise duration of each phoneme, a duration model is needed to predict the duration of each phoneme. This embodiment uses a Bidirectional Long-Short Term Memory (Bi-directional LSTM) network to predict the phoneme duration. The input to this duration model includes phonemes, notes, beats, and the speaker embedding; the output is log-level phoneme durations (Log Durations). Phonemes belong to the sample lyrics features, notes and beats belong to the sample score features, and the speaker embedding belongs to the sample timbre features of the sample audio. During the application phase of the model, the predicted duration needs to be calculated exponentially (e). Here, duration refers to the number of frames, with each frame representing 10 milliseconds. 24kHz audio is represented in a computer as 24,000 16-bit integers per second. When extracting features from the audio, the audio sampling points need to be divided into frame-level units. In this embodiment, a 10ms sampling point represents one frame.

[0254] Extract sample phoneme durations and sample syllable durations from the duration annotation file or the Musicxml file of the music score as the true phoneme duration and true syllable duration, i.e., GT Phone-level & Syllable-level Durations. Calculate the phoneme duration loss by combining the predicted phoneme duration Log Durations and the corresponding sample phoneme duration GTPhone-level Durations for each phoneme predicted by the duration model.

[0255] The phonemes constituting each syllable are identified. The predicted phoneme durations corresponding to each phoneme are summed to form the predicted syllable duration for the corresponding syllable. The syllable duration loss is calculated by combining the predicted syllable duration and the actual syllable-level duration (GTSyllable-levelDurations). The phoneme duration loss and the syllable duration loss together form the multi-scale rhythm loss.

[0256] (2) Acoustic model

[0257] The acoustic model is constructed based on a multi-layer Transformer encoder FFT, a duration extension unit (Length Regulator), another multi-layer Transformer encoder FFT, and a CBHG-based post-processing network (CBHG Post-net). The CBHG Post-net is the back-end processing network used to extract sequence features; it can be called the sequence feature extraction unit. The Transformer encoder FFT can be called a self-attention encoder.

[0258] The acoustic model's input includes phonemes and notes. During the training phase, the duration expansion unit expands the sample coding features output by the self-attention encoder based on the actual phoneme durations (GTphone-level Durations). The state before and after expansion changes from phoneme-level to frame-level, resulting in sample duration-expanded coding features. For example, if the first self-attention encoder's FFT output is ABC, and the actual phoneme duration is 123, then ABC and 123 are used as input to the duration expansion unit. The duration expansion unit expands ABC based on 1, 2, and 3, resulting in sample duration-expanded coding features ABBCCC.

[0259] The second self-attention encoder, FFT, concatenates the output of the extended duration unit with the speaker embedding of the sample timbre features to extract acoustic features, outputting intermediate spectral features. A CBHG-based post-processing network further smooths these intermediate spectral features, enabling the acoustic model to output high-quality predicted spectral features (LPC features). The predicted spectral features (LPC features) and corresponding pitches from the acoustic model are then compared with the corresponding spectral labels (GT BFCCs, Bark-scale frequency cepstral coefficients) and pitch labels (Pitches) to calculate the spectral loss (BFCC loss) and pitch loss. The progressive pitch-weighted loss includes both BFCC loss and pitch loss. The spectral labels are sample spectral features extracted from the sample audio.

[0260] Since pre-training requires singing data with multiple timbres, an adversarial speaker classifier is added to the end of the self-attention encoder's FFT to avoid overfitting or severe imbalance. The speaker classifier performs gradient reversal processing on the encoded features of the samples, and a speaker classifier is then used to classify the features obtained from the gradient reversal process, yielding the corresponding classification results. This classifier enables the encoder to operate independently of the speaker, thereby enhancing its robustness to different corpora and unaffected by singing timbre. The speaker adversarial loss (speakeradversial loss) is calculated using the speaker classification results and classification labels from the classifier. This adversarial classifier ensures that the features output by the self-attention encoder's FFT do not contain personal characteristics, meaning that the features obtained from the self-attention encoder's FFT encoding do not include personal features.

[0261] (3) Vocoder Model

[0262] The vocoder is based on the LPC Net architecture. The LPC Net takes the predicted spectral features as input and outputs a high-quality 24kHz predicted song. The vocoder loss is calculated by combining the output 24kHz high-quality predicted song with the sample audio GT Song Segments.

[0263] The LPC Net model predicts all sample points of a song based on spectral feature regression. The principle is to upsample the LPC features by a factor of 240. Since the spectral features are extracted every 10ms, and for a 24kHz song, there are 24,000 sample points per second, this translates to 240 sample points every 10ms. Therefore, the spectral features first need to be upsampled by 240 times, and then prediction is performed using an RNN (Recurrent Neural Network). The output of the RNN at each time step represents one sample point.

[0264] In this embodiment, model pre-training can be divided into two parts: the duration model and the acoustic model can be trained together, while the vocoder model can be trained separately. During training, the duration model uses a log-level L1 loss as the objective function, employing both phoneme-level and spell-level durations (GT Phone-level & Syllable-level Durations) as true duration labels to guide parameter updates. The acoustic model also uses L1 loss as the objective function, employing a Mel-level spectrum BFCC (Below-the-Face Classification) more suitable for human hearing as the target spectrum to guide parameter updates. Notably, the duration expansion unit of the acoustic model uses the true duration of the phonemes as the basis for expansion during training. The vocoder model uses the cross-entropy classification loss function as the objective function (vocoder loss), i.e., the synthesis loss. The input to the vocoder model is the Mel-level spectrum BFCC, using the value of each sample point of the true waveform of the sample audio GT Song Segments as the target category index.

[0265] The application inference phase is the application phase. Song synthesis models can employ two different application strategies in different application scenarios:

[0266] 1) Use the duration of real human voices as the reasoning duration.

[0267] For synthesis scenarios requiring strict accompaniment matching, the actual duration of the provided accompaniment is used directly as the basis for the duration extension unit of the acoustic model, instead of the model's predicted duration. This is done to ensure the duration accurately aligns with the timing of the accompaniment, resulting in a more pleasing overall sound. For example, if the song the user wants to synthesize exists in the sample set, the corresponding lyric duration features can be directly obtained. Alternatively, if the user provides not only the target lyrics and sheet music but also the lyric duration information, the lyric duration features can be used directly as the extension basis. In this case, the duration model is idle and does not need to predict the duration.

[0268] 2) Use the model prediction time as the inference time.

[0269] For synthesis scenarios that do not require matching accompaniment or do not provide the actual duration, such as improvisational a cappella singing, the phoneme durations predicted by the duration model can be directly used as the basis for duration expansion units. This eliminates the need to pre-provide durations and allows for the generation of a song directly based on the target lyrics and sheet music information.

[0270] Understandably, regardless of the reasoning strategy employed, the song synthesis model can predict a sentence each time and then assemble them sequentially to form the target song.

[0271] Traditional song synthesis training samples rely on manual recording and annotation, requiring high manpower costs and lengthy recording and annotation cycles. Furthermore, under cost constraints, models trained on manually recorded and annotated data exhibit instability in areas with limited distribution of high or low notes and sustain. The data augmentation in this embodiment reduces the required corpus size, significantly lowering the cost of manual recording and annotation. Moreover, data augmentation expands the quantity and richness of the corpus, increasing the diversity of different melodic combinations, thereby improving the robustness of vocal synthesis and achieving end-to-end multi-timbre song synthesis effects.

[0272] The method in this embodiment can be used in any vocal synthesis or other generative models based on Transformer, expanding the amount and richness of data required for training the model. Furthermore, this method can customize a user's timbre based on a short excerpt of their existing vocal recording, thus providing a more comprehensive singing ability. Simultaneously, this method can also be used to cultivate virtual idols, providing entertainment value to users anytime, anywhere.

[0273] In other embodiments, the performance of the model can be improved by incorporating users' everyday speech data, which can further reduce the cost of acquiring singing data.

[0274] In one embodiment, such as Figure 7 As shown, a song synthesis method is provided, which can be applied to... Figure 1 Computer equipment (computer equipment can be) Figure 1 Taking a terminal or server as an example, the following steps are included:

[0275] Step S702: Obtain target lyrics and target musical score information, and perform feature encoding based on the target lyrics and target musical score information to obtain encoded features.

[0276] The target lyrics can be the lyrics corresponding to the audio of the song to be synthesized, and the target musical score information can be the melody information corresponding to the audio of the song to be synthesized, including information such as notes, note values, rhythm, legato, and sustain. The duration information of the target lyrics refers to the duration of the lyrics, including the phoneme duration of each phoneme, and may also include the syllable duration of each syllable.

[0277] Specifically, the computer device acquires the target lyrics for song synthesis, the corresponding duration of the lyrics, and the target musical score. It then extracts features from both the target lyrics and the musical score to obtain the lyric features and the musical score features. The lyric features may include phonemes and syllables. The musical score features include notes and beats.

[0278] The computer device concatenates the lyric features and the musical score features, and then encodes the concatenated features to obtain the corresponding encoded features.

[0279] In this embodiment, the target lyrics, the corresponding lyrics duration information, and the target musical score information are all provided by the user.

[0280] Step S704: Obtain the duration features of the target lyrics, and extend the duration features of the encoding features according to the duration features of the lyrics to obtain the duration extended encoding features.

[0281] The duration features of the target lyrics include the duration features of each phoneme corresponding to the target lyrics, and may also include the duration features of each syllable corresponding to the target lyrics.

[0282] Specifically, the computer device extracts features from the duration information of the target lyrics to obtain the corresponding duration features. Based on the duration features of the target lyrics, the computer device expands the duration features of the encoded features to obtain the corresponding duration-extended encoded features.

[0283] In this embodiment, the lyric duration feature includes phoneme duration. The computer device then expands the duration feature of each phoneme in the lyric duration information according to the phoneme duration of each phoneme to obtain the corresponding duration extended encoding feature.

[0284] In other embodiments, the lyrics duration feature includes syllable duration. The computer device then extends the duration feature of each syllable in the lyrics duration information to obtain the corresponding duration extended encoding feature.

[0285] Step S706: Determine the timbre features of the target timbre, concatenate the timbre features and duration extension coding features, and then extract the acoustic features to obtain the target spectral features.

[0286] Specifically, the computer equipment determines the target timbre required for song synthesis and obtains the corresponding timbre features. The computer equipment then concatenates the timbre features of the target timbre with the duration-extended coding features, and performs acoustic feature extraction on the concatenated features to extract spectral features, thus obtaining the target spectral features.

[0287] The computer device can determine the target timbre selected by the user from multiple candidate timbres and obtain the timbre features corresponding to the target timbre. In other embodiments, the computer device can perform feature extraction on the target audio provided by the user to obtain corresponding timbre features, which characterize the target timbre of the target audio.

[0288] Step S708: Synthesize a target song based on the target spectral features. The target song is matched with the target lyrics, target musical score information and target timbre.

[0289] Specifically, computer equipment can synthesize a target song based on the target spectral characteristics. The target song consists of target lyrics and target musical score information and is sung with the target timbre.

[0290] In this embodiment, the target lyrics and target musical score information for song synthesis are feature-encoded to obtain encoded features containing both lyric and musical score features. The duration features of the target lyrics are obtained, and the duration features of the encoded features are expanded according to these lyric duration features. Using the actual duration of the target lyrics as the basis for expansion makes the resulting extended duration encoded features more accurate. Furthermore, for synthesis scenarios requiring strict accompaniment matching, using the actual duration as the basis for expansion, rather than using predicted duration, ensures that the duration accurately aligns with the timing of the accompaniment, resulting in a more harmonious and natural sound from the synthesized song's lyrics and melody. The timbre features of the target timbre are determined, and the timbre features and extended duration encoded features are concatenated for acoustic feature extraction, yielding target spectral features containing the target timbre. Based on these target spectral features, a target song matching the target lyrics, target musical score information, and target timbre is synthesized, effectively achieving a customized target timbre effect. Furthermore, given the existing lyrics duration information, the system combines lyrics, sheet music, and timbre to synthesize a target song with a specific timbre, enabling the song synthesis to have a timbre customization function, thereby improving the naturalness of the synthesized song.

[0291] In one embodiment, the method further includes:

[0292] Based on the lyrical features of the target lyrics, the musical score features of the target musical score, and the timbre features of the target timbre, duration prediction processing is performed to obtain the phoneme duration corresponding to each phoneme in the target lyrics; the duration features of the extended coding features are obtained according to the phoneme duration of each phoneme to obtain the duration extended coding features.

[0293] Specifically, the computer equipment acquires the target lyrics and target musical score information for song synthesis, and extracts features from both to obtain lyric features corresponding to the target lyrics and musical score features corresponding to the target musical score. Lyric features may include phonemes and syllables. Musical score features include notes and may include rhythm.

[0294] The computer device can determine the target timbre selected by the user from multiple candidate timbres and obtain the timbre features corresponding to the target timbre. In other embodiments, the computer device can perform feature extraction on the target audio provided by the user to obtain corresponding timbre features, which characterize the target timbre of the target audio.

[0295] The computer device performs duration prediction processing based on lyric features, musical score features, and timbre features to obtain the predicted phoneme duration for each phoneme in the target lyrics. Further, the computer device performs duration prediction processing based on phoneme, note, meter, and timbre features to obtain the predicted phoneme duration for each phoneme in the target lyrics.

[0296] In this embodiment, when the user does not provide lyric duration information, duration prediction processing can be performed based on lyric features, musical score features, and the timbre features of the target timbre to accurately predict the duration of each phoneme in the target lyrics. This predicted phoneme duration can then be used as the basis for duration expansion. Following the duration features of the extended coded features for each phoneme, the extended duration coded features and the timbre features of the target timbre are concatenated and acoustic features are extracted to obtain the corresponding target spectral features. Based on these target spectral features, a target song sung with the target timbre, consisting of the target lyrics and the target musical score, is synthesized. This eliminates the need to pre-provide lyric duration information and allows for the direct synthesis of a target song with a specific timbre based on the lyrics, musical score, and timbre. This method is particularly effective for synthesis scenarios that do not require accompaniment, such as improvisational a cappella performances, and can meet the song synthesis needs of various scenarios.

[0297] In one embodiment, the song synthesis method is executed through a song synthesis model, the overall architecture of which is shown in the diagram below. Figure 8 As shown, the song synthesis model consists of three parts: a duration model, an acoustic model, and a vocoder model.

[0298] The song synthesis model uses a duration model to predict duration based on the lyric features of the target lyrics, the spectral features of the target musical score, and the timbre features of the target timbre, obtaining the phoneme duration corresponding to each phoneme in the target lyrics. The acoustic model includes two self-attention encoders, a duration expansion unit, and a sequence feature extraction unit. The first self-attention encoder encodes the lyric features of the target lyrics and the spectral features of the target musical score, obtaining encoded features. These encoded features and the phoneme durations corresponding to each phoneme are used as input to the duration expansion unit, which expands the duration features of the encoded features according to the phoneme durations of each phoneme, resulting in extended duration encoded features. The timbre features and the extended duration encoded features are used as input to the second self-attention encoder, which then inputs these features to the sequence feature extraction unit. The sequence feature extraction unit concatenates the timbre features and the extended duration encoded features and performs acoustic feature extraction to obtain the target spectral features. The vocoder synthesizes the target song based on the target spectral features, which matches the target lyrics, target musical score, and target timbre.

[0299] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0300] Based on the same inventive concept, this application also provides a training apparatus for a song synthesis model to implement the training method for the song synthesis model described above. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of one or more training apparatus embodiments for song synthesis models provided below can be found in the limitations of the training method for song synthesis models described above, and will not be repeated here.

[0301] In one embodiment, such as Figure 9 As shown, a training device 900 for a song synthesis model is provided, including: an acquisition module 902, an augmentation module 904, a pre-training module 906, an extraction module 908, and a training module 910, wherein:

[0302] The acquisition module 902 is used to acquire an initial sample set, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio.

[0303] The augmentation module 904 is used to augment samples based on the audio transformation of the recorded audio in the initial sample to obtain an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after audio transformation, the augmented lyrics duration information of the augmented audio, and the augmented musical score information of the augmented audio.

[0304] The pre-training module 906 is used to pre-train the model based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis.

[0305] The extraction module 908 is used to acquire the audio of the target sound source and extract timbre features based on the audio of the target sound source.

[0306] Training module 910 is used to train the initial model for song synthesis based on timbre features to obtain the song synthesis model.

[0307] In this embodiment, an initial sample set consisting of initial samples from multiple sound sources is obtained. The recorded audio, the duration of the source lyrics, and the source musical score information included in the initial samples are used as the base data for sample augmentation. Based on the audio transformation of the recorded audio in the initial samples, an augmented sample set is obtained. This augmented sample set includes augmented audio obtained after audio transformation, the duration of the augmented lyrics, and the augmented musical score information. This allows for the acquisition of a large amount of training data, expanding the quantity and richness of the training corpus. Furthermore, sample augmentation can obtain more training data on top of the base data, reducing the requirement for a large amount of base data and significantly reducing the cost of manual recording and annotation of the base data. Pre-training the model using the large amount of training data contained in the initial and augmented sample sets improves the robustness of the pre-trained song synthesis initial model. The system extracts timbre features from the target sound source and trains an initial song synthesis model based on these features. It can fine-tune the pre-trained initial song synthesis model based on individual audio data, accurately obtaining a song synthesis model that matches the target timbre of the target sound source, thereby effectively achieving customized timbre effects in song synthesis.

[0308] In one embodiment, the augmentation module 904 is further configured to transform the recorded audio of multiple initial samples according to an audio transformation method to obtain augmented audio corresponding to each recorded audio; determine the augmented lyrics duration information corresponding to each augmented audio based on the source lyrics duration information of the multiple initial samples; adjust the source score information of the multiple initial samples through a score transformation method that matches the audio transformation method to obtain augmented score information corresponding to each augmented audio; and form an augmented sample set based on each augmented audio, the augmented lyrics duration information of each augmented audio, and the augmented score information of each augmented audio.

[0309] In this embodiment, the recorded audio of multiple initial samples is transformed according to an audio transformation method to obtain augmented audio corresponding to each recorded audio, enabling the processing of existing audio to obtain a greater number of audio samples. Based on the source lyric duration information of multiple initial samples, the augmented lyric duration information corresponding to each augmented audio is determined, ensuring that each augmented audio corresponds to the correct lyric duration annotation information, guaranteeing the accuracy of the lyrics and lyric duration in the transformed audio. By using a music score transformation method that matches the audio transformation method, the source music score information of multiple initial samples is adjusted, automatically modifying the existing music score information according to the music score transformation method that matches the audio transformation, realizing music score augmentation, and ensuring that each augmented audio corresponds to the correct music score information. Based on each augmented audio, each augmented lyric duration information, and each augmented music score information, corresponding augmented samples are formed, effectively increasing the number of training samples and improving the diversity of training samples. Furthermore, augmenting existing samples reduces the cost of manual sample collection and improves the efficiency of training sample collection.

[0310] In one embodiment, the augmentation module 904 is further configured to perform pitch adjustment processing on the recorded audio of the multiple initial samples respectively to obtain augmented audio corresponding to each recorded audio; use the source lyrics duration information of each recorded audio of the multiple initial samples as the augmented lyrics duration information corresponding to the corresponding augmented audio; and perform scale adjustment processing on the notes of the source musical score information of the multiple recorded audio according to the pitch adjustment processing of the recorded audio of the multiple initial samples to obtain augmented musical score information corresponding to each augmented audio.

[0311] In this embodiment, pitch adjustment processing is performed on the recorded audio of multiple initial samples to obtain augmented audio corresponding to each recorded audio. This allows the augmented audio to cover a wider range of pitches, thereby automatically expanding the number of audio samples. Since the pitch adjustment processing only adjusts pitch and has no effect on lyrics or duration, the source lyric duration information of each recorded audio from the multiple initial samples is directly used as the augmented lyric duration information for the corresponding augmented audio, ensuring the accuracy of the lyrics and duration of the augmented audio. Following the pitch adjustment processing of the recorded audio of the multiple initial samples, scale adjustment processing is performed on the notes of the source musical score information of the multiple recorded audio samples. This ensures that the musical score adjustment of the recorded audio corresponds to the pitch adjustment of that recorded audio, thus ensuring that each augmented audio corresponds to the correct musical score information. This effectively guarantees the accuracy of the mapping between augmented audio, augmented lyric duration information, and augmented musical score information, as well as the mapping relationship among the three, thereby improving the effectiveness and accuracy of data augmentation.

[0312] In one embodiment, the augmentation module 904 is further configured to: divide the recorded audio of multiple initial samples to obtain audio segments corresponding to each recorded audio; for each recorded audio, concatenate the audio segments of the corresponding recorded audio in adjacent order to obtain multiple augmented audios of the corresponding recorded audio; divide the source lyric duration information of the multiple initial samples according to the division processing of the recorded audio of the multiple initial samples to obtain lyric duration information segments corresponding to each audio segment; concatenate the lyric duration information segments of each audio segment according to the concatenation processing of each audio segment to obtain augmented lyric duration information corresponding to each augmented audio; divide the source musical score information of the multiple initial samples according to the division processing of the recorded audio of the multiple initial samples to obtain musical score information segments corresponding to each audio segment; concatenate the musical score information segments of each audio segment according to the concatenation processing of each audio segment to obtain augmented musical score information corresponding to each augmented audio.

[0313] In this embodiment, the recorded audio of multiple initial samples is divided into segments to obtain audio segments corresponding to each recorded audio. For each recorded audio, the audio segments are spliced ​​together in adjacent order to obtain multiple augmented audio segments. By dividing the recorded audio and splicing the audio segments, more augmented audio segments can be obtained, effectively achieving audio augmentation. Following the division of the recorded audio of multiple initial samples, the source lyrics duration information of the multiple initial samples is divided, ensuring a one-to-one correspondence between each audio segment and each lyrics duration information segment. Following the splicing process of each audio segment, the lyrics duration information segments of each audio segment are spliced ​​together, ensuring a one-to-one correspondence between the augmented audio and the augmented lyrics duration information. Following the division of the recorded audio of multiple initial samples, the source musical score information of the multiple initial samples is divided, ensuring an accurate mapping relationship between each audio segment and each musical score information segment. By splicing together the audio segments, the musical score information segments of each audio segment are concatenated, ensuring an accurate mapping between the augmented audio and the augmented musical score information. This effectively augments the samples, resulting in more training samples. Furthermore, the augmented samples obtained through segmentation and splicing acquire more contextual information, thereby enhancing the robustness of the song synthesis model when receiving input information.

[0314] In one embodiment, the pre-training module 906 is further configured to: obtain sample audio, sample lyrics duration information corresponding to the sample audio, and sample musical score information corresponding to the sample audio from the set consisting of the initial sample set and the augmented sample set; perform feature encoding based on the sample lyrics duration information and sample musical score information of the sample audio to obtain sample encoded features; expand the duration features of the sample encoded features according to the sample lyrics duration features of the sample lyrics duration information to obtain sample duration extended encoded features; extract sample timbre features of the sample audio, concatenate the sample timbre features and sample duration extended encoded features, and then perform acoustic feature extraction to obtain predicted spectral features; synthesize a predicted song based on the predicted spectral features, construct a target loss function based on the synthesis loss between the predicted song and the sample audio; and perform model pre-training based on the target loss function to obtain an initial model for song synthesis.

[0315] In this embodiment, sample audio, corresponding sample lyrics duration information, and corresponding sample musical score information are obtained from a set consisting of an initial sample set and an augmented sample set. This allows the initial and augmented samples to be used as training samples for model training, enabling the model to be trained using richer training samples and improving its robustness. Feature encoding is performed based on the sample lyrics duration information and sample musical score information of the sample audio to obtain sample encoded features. The duration features of the sample encoded features are then expanded according to the sample lyrics duration features to obtain sample duration extended encoded features. Sample timbre features are extracted from the sample audio. The sample timbre features and sample duration extended encoded features are concatenated and then used for acoustic feature extraction to obtain predicted spectral features. A predicted song is synthesized based on the predicted spectral features, thus obtaining the song predicted by the model. A target loss function is constructed based on the synthesis loss between the predicted song and the real sample audio. Model pre-training is performed based on the target loss function to accurately obtain the initial model for song synthesis. Furthermore, training with augmented samples can greatly expand the training corpus for singing synthesis, thereby improving the model's stability, expressiveness, and practicality when faced with different combinations of lyrics and melodies.

[0316] In one embodiment, the pre-training module 906 is further configured to perform duration prediction processing based on the sample lyric features of the sample lyric duration information, the sample musical score features of the sample musical score information, and the sample timbre features to obtain the predicted phoneme duration corresponding to each phoneme in the sample lyric duration information; determine the phoneme duration loss between the predicted phoneme duration of each phoneme and the sample phoneme duration of each phoneme in the sample lyric duration information; determine the synthesis loss between the predicted song and the sample audio; and construct a target loss function based on the phoneme duration loss and the synthesis loss.

[0317] In this embodiment, the phoneme duration loss characterizes the difference between the model's predicted duration and the actual duration of each phoneme. Duration prediction is performed based on the sample lyrics features of the sample lyrics duration information, the sample musical score features of the sample musical score information, and the sample timbre features. This yields the predicted phoneme duration for each phoneme in the sample lyrics duration information. Therefore, the model's loss in predicting the duration of each phoneme can be determined based on the difference between the predicted duration and the actual duration of each phoneme in the sample lyrics duration information. A target loss function is constructed by combining the phoneme duration loss and the synthesis loss. This takes into account the influence of local factors in phoneme duration prediction and global factors in song synthesis on the model. Training the model using both local and global losses further improves the model's accuracy in phoneme duration prediction, thereby enhancing the accuracy of the initial song synthesis model.

[0318] In one embodiment, the pre-training module 906 is further configured to determine the predicted syllable duration corresponding to each syllable in the sample lyrics duration information based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information; determine the syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information; and construct a target loss function based on the phoneme duration loss, syllable duration loss, and synthesis loss.

[0319] In this embodiment, based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information, the predicted syllable duration corresponding to each syllable in the sample lyrics duration information is determined. The syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information is determined. A target loss function is constructed by combining phoneme duration loss, syllable duration loss, and synthesis loss. This can take into account the influence of multiple local factors such as phoneme duration prediction and syllable duration prediction, as well as global factors of song synthesis on the model. Thus, by combining multiple local losses and global losses to train the model, the accuracy of the model in phoneme duration prediction and syllable duration prediction can be further improved, thereby improving the accuracy of the initial model for song synthesis.

[0320] In one embodiment, the pre-training module 906 is further configured to perform gradient inversion processing on the sample encoded features, and classify the features obtained from the gradient inversion processing to obtain the classification result of the sample audio; determine the adversarial loss between the classification result and the classification label of the sample audio; determine the synthesis loss between the predicted song and the sample audio; and construct a target loss function based on the adversarial loss and the synthesis loss.

[0321] In this embodiment, adversarial loss reflects the difference between the sample coding features extracted by the model and the expected sample coding features. By performing gradient inversion on the sample coding features and classifying them based on the resulting features, the classification result of the sample audio is obtained. The adversarial loss between the classification result and the classification label of the sample audio is determined to judge whether the sample coding features extracted by the model meet expectations. This allows for adjustment of model parameters to ensure the extracted sample coding features achieve the desired result. Combining adversarial loss and synthesis loss to construct a target loss function takes into account the impact of multiple factors on the model, such as the loss in extracting sample coding features and the overall loss in song synthesis. Training the model with multiple losses further improves the accuracy of the model in coding features, thereby improving the accuracy of song synthesis.

[0322] In one embodiment, the pre-training module 906 is further configured to extract sample spectral features of the sample audio and determine the spectral loss between the predicted spectral features and the sample spectral features; determine the synthesis loss between the predicted song and the sample audio; and construct a target loss function based on the spectral loss and the synthesis loss.

[0323] In this embodiment, the spectral loss reflects the difference between the model's predicted spectral features and the actual spectral features. Sample spectral features are extracted from the sample audio, and the spectral loss between the predicted and sample spectral features is determined to assess whether the model's predicted spectral features meet expectations. This allows for adjustment of model parameters to ensure the predicted spectral features achieve the desired outcome. Constructing a target loss function by combining spectral loss and synthesis loss takes into account the impact of multiple factors on the model, such as local losses in spectral prediction and overall losses in song synthesis. Training the model using both local and global losses further improves the model's accuracy in spectral prediction, thereby enhancing the accuracy of song synthesis.

[0324] In one embodiment, the training module 910 is used to adjust the parameters of the initial duration model, the initial acoustic model, and the initial vocoder based on the timbre characteristics of the target sound source, so as to obtain a song synthesis model that matches the timbre of the target sound source.

[0325] In this embodiment, the model is pre-trained using a large number of training samples, including initial and augmented samples. Based on the initial model for song synthesis, the parameters of the initial duration model, initial acoustic model, and initial vocoder are adjusted according to the timbre characteristics of the target sound source. The model parameters can be fine-tuned using personal singing data, so that the resulting song synthesis model matches the individual's timbre. This allows for the synthesis of songs sung in a personal timbre using any lyrics and sheet music, thus achieving personal timbre customization.

[0326] Based on the same inventive concept, this application also provides a song synthesis apparatus for implementing the song synthesis method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more song synthesis apparatus embodiments provided below can be found in the limitations of the song synthesis method described above, and will not be repeated here.

[0327] In one embodiment, such as Figure 10 As shown, a song synthesis device 1000 is provided, including: an encoding module 1002, an expansion module 1004, a determination module 1006, and a synthesis module 1008, wherein:

[0328] The encoding module 1002 is used to acquire target lyrics and target musical score information, and to perform feature encoding based on the target lyrics and musical score information to obtain encoded features.

[0329] The extension module 1004 is used to obtain the duration features of the target lyrics and extend the duration features of the encoding features according to the duration features of the lyrics to obtain the duration extended encoding features.

[0330] The determination module 1006 is used to determine the timbre characteristics of the target timbre. After concatenating the timbre characteristics and the duration extension coding characteristics, acoustic features are extracted to obtain the target spectral characteristics.

[0331] The synthesis module 1008 is used to synthesize a target song based on the target spectral features, and the target song is matched with the target lyrics, target musical score information and target timbre.

[0332] In this embodiment, the target lyrics and target musical score information for song synthesis are feature-encoded to obtain encoded features containing both lyric and musical score features. The duration features of the target lyrics are obtained, and the duration features of the encoded features are expanded according to these lyric duration features. Using the actual duration of the target lyrics as the basis for expansion makes the resulting extended duration encoded features more accurate. Furthermore, for synthesis scenarios requiring strict accompaniment matching, using the actual duration as the basis for expansion, rather than using predicted duration, ensures that the duration accurately aligns with the timing of the accompaniment, resulting in a more harmonious and natural sound from the synthesized song's lyrics and melody. The timbre features of the target timbre are determined, and the timbre features and extended duration encoded features are concatenated for acoustic feature extraction, yielding target spectral features containing the target timbre. Based on these target spectral features, a target song matching the target lyrics, target musical score information, and target timbre is synthesized, effectively achieving a customized target timbre effect. Furthermore, given the existing lyrics duration information, the system combines lyrics, sheet music, and timbre to synthesize a target song with a specific timbre, enabling the song synthesis to have a timbre customization function, thereby improving the naturalness of the synthesized song.

[0333] In one embodiment, the device further includes a duration prediction module, which is used to perform duration prediction processing based on the lyric features of the target lyrics, the musical score features of the target musical score information, and the timbre features of the target timbre to obtain the phoneme duration corresponding to each phoneme in the target lyrics; and to obtain the duration extended coding features according to the duration features of the phoneme duration extended coding features of each phoneme.

[0334] In this embodiment, when the user does not provide lyric duration information, duration prediction processing can be performed based on lyric features, musical score features, and the timbre features of the target timbre to accurately predict the duration of each phoneme in the target lyrics. This predicted phoneme duration can then be used as the basis for duration expansion. Following the duration features of the extended coded features for each phoneme, the extended duration coded features and the timbre features of the target timbre are concatenated and acoustic features are extracted to obtain the corresponding target spectral features. Based on these target spectral features, a target song sung with the target timbre, consisting of the target lyrics and the target musical score, is synthesized. This eliminates the need to pre-provide lyric duration information and allows for the direct synthesis of a target song with a specific timbre based on the lyrics, musical score, and timbre. This method is particularly effective for synthesis scenarios that do not require accompaniment, such as improvisational a cappella performances, and can meet the song synthesis needs of various scenarios.

[0335] The training device and various modules of the aforementioned song synthesis model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0336] In one embodiment, a computer device is provided, which may be a terminal or a server. Taking a terminal as an example, its internal structure diagram can be as follows: Figure 11As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a training method for a song synthesis model and a song synthesis method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0337] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0338] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0339] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0340] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0341] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0342] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0343] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0344] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a song synthesis model, characterized in that, The method includes: An initial sample set is obtained, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio. The source lyrics duration information is obtained by duration annotation of the source lyrics of the recorded audio, and the source lyrics duration information includes duration annotation information of each phoneme in the source lyrics. Based on the audio transformation of the recorded audio in the initial samples, sample augmentation is performed to obtain an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after the audio transformation, the augmented lyrics duration information of the augmented audio, and the augmented musical score information of the augmented audio. The recorded audio and the augmented audio serve as sample audio, the original lyrics duration information and the augmented lyrics duration information serve as sample lyrics duration information, and the original musical score information and the augmented musical score information serve as sample musical score information. The audio transformation method includes at least one of pitch adjustment processing of the recorded audio or splicing processing of audio segments of the recorded audio. Multiple initial samples are augmented based on different audio transformation methods of the recorded audio. The transformation method of the recorded audio of the initial sample matches the duration transformation method of the original lyrics duration information of the initial sample and the musical score transformation method of the original musical score information of the initial sample. The model is pre-trained based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis. The audio of the target sound source is acquired, and timbre features are extracted based on the audio of the target sound source; the initial model for song synthesis is trained based on the timbre features to obtain the song synthesis model. The process of augmenting the audio samples based on the audio transformation recorded in the initial samples to obtain an augmented sample set includes at least one of the following: The recorded audios of multiple initial samples are subjected to pitch adjustment processing to obtain augmented audios corresponding to each recorded audio. The duration of the source lyrics for each recorded audio of the multiple initial samples is used as the duration of the augmented lyrics for the corresponding augmented audio. According to the pitch adjustment processing of the recorded audios of the multiple initial samples, the notes of the source musical score information of the multiple recorded audios are subjected to scale adjustment processing to obtain augmented musical score information corresponding to each augmented audio. The scale raising processing is matched with the pitch sharpening processing, and the scale lowering processing is matched with the pitch lowering processing. An augmented sample set is formed based on each augmented audio, the duration of the augmented lyrics for each augmented audio, and the augmented musical score information for each augmented audio. The recorded audio of multiple initial samples is divided into segments to obtain audio segments corresponding to each recorded audio. For each recorded audio, the audio segments are concatenated in adjacent order to obtain multiple augmented audio segments of the corresponding recorded audio. Based on the division of the recorded audio of the multiple initial samples, the source lyrics duration information of the multiple initial samples is divided to obtain lyrics duration information segments corresponding to each audio segment. Based on the concatenation of the audio segments, the lyrics duration information segments of each audio segment are concatenated. Next, augmented lyric duration information corresponding to each augmented audio is obtained; according to the division processing of the recorded audio of multiple initial samples, the source score information of multiple initial samples is divided to obtain score information fragments corresponding to each audio segment; according to the splicing processing of each audio segment, the score information fragments of each audio segment are spliced ​​to obtain augmented score information corresponding to each augmented audio; an augmented sample set is formed based on each augmented audio, the augmented lyric duration information of each augmented audio, and the augmented score information of each augmented audio.

2. The method according to claim 1, characterized in that, The step of pre-training the model based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis includes: From the set consisting of the initial sample set and the augmented sample set, obtain sample audio, sample lyrics duration information corresponding to the sample audio, and sample musical score information corresponding to the sample audio; Based on the sample lyrics duration information and sample musical score information of the sample audio, feature encoding is performed to obtain sample encoding features; Based on the sample lyrics duration features of the sample lyrics duration information, the duration features of the sample encoding features are extended to obtain the sample duration extended encoding features; Extract the sample timbre features of the sample audio, concatenate the sample timbre features and the sample duration extended coding features, and then extract the acoustic features to obtain the predicted spectral features; Based on the predicted spectral features, a predicted song is synthesized, and a target loss function is constructed according to the synthesis loss between the predicted song and the sample audio. Based on the target loss function, the model is pre-trained to obtain the initial model for song synthesis.

3. The method according to claim 2, characterized in that, The method further includes: Based on the sample lyric duration information, the sample musical score information, and the sample timbre features, duration prediction processing is performed to obtain the predicted phoneme duration for each phoneme in the sample lyric duration information. Determine the phoneme duration loss between the predicted phoneme duration of each phoneme and the sample phoneme duration of each phoneme in the sample lyrics duration information; The step of constructing the target loss function based on the synthesis loss between the predicted song and the sample audio includes: Determine the synthesis loss between the predicted song and the sample audio; Based on the phoneme duration loss and the synthesis loss, a target loss function is constructed.

4. The method according to claim 3, characterized in that, The method further includes: Based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information, determine the predicted syllable duration corresponding to each syllable in the sample lyrics duration information; Determine the syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information; The step of constructing a target loss function based on the phoneme duration loss and the synthesis loss includes: A target loss function is constructed based on the phoneme duration loss, the syllable duration loss, and the synthesis loss.

5. The method according to claim 2, characterized in that, The method further includes: The sample encoding features are subjected to gradient inversion processing, and the features obtained from the gradient inversion processing are used for classification to obtain the classification result of the sample audio. Determine the adversarial loss between the classification result and the classification label of the sample audio; The step of constructing the target loss function based on the synthesis loss between the predicted song and the sample audio includes: Determine the synthesis loss between the predicted song and the sample audio; Based on the adversarial loss and the synthetic loss, a target loss function is constructed.

6. The method according to claim 2, characterized in that, The method further includes: Extract the sample spectral features of the sample audio and determine the spectral loss between the predicted spectral features and the sample spectral features; The step of constructing the target loss function based on the synthesis loss between the predicted song and the sample audio includes: Determine the synthesis loss between the predicted song and the sample audio; Based on the spectral loss and the synthetic loss, a target loss function is constructed.

7. The method according to any one of claims 1 to 6, characterized in that, The initial model for song synthesis includes an initial duration model, an initial acoustic model, and an initial vocoder; The process of training the initial song synthesis model based on the timbre features to obtain the song synthesis model includes: Based on the timbre characteristics of the target sound source, the parameters of the initial duration model, the initial acoustic model, and the initial vocoder are adjusted to obtain a song synthesis model that matches the timbre of the target sound source.

8. The method according to any one of claims 1 to 6, characterized in that, The audio of the target sound source is obtained, and timbre features are extracted based on the audio of the target sound source; The initial song synthesis model is trained based on the timbre features to obtain a song synthesis model, including: The singing data of multiple target objects are acquired, and the initial model for song synthesis is trained using the singing data of each target object to obtain a sub-model that matches the timbre of each target object. The sub-models are then integrated into the song synthesis model. The song synthesis model provides multiple candidate timbres, and the multiple candidate timbres are matched one by one with each sub-model.

9. The method according to claim 8, characterized in that, The process of training the initial song synthesis model using the singing data of each target object to obtain a sub-model matching the timbre of each target object, and integrating the sub-models into a song synthesis model, includes: The parameters of the initial duration model, initial acoustic model, and initial vocoder in the initial song synthesis model are adjusted using the singing data of each target object to obtain a sub-model that matches the timbre of each target object, and the sub-models are integrated into the song synthesis model.

10. A method for synthesizing a song, comprising: Obtain target lyrics and target sheet music information, and perform feature encoding based on the target lyrics and target sheet music information to obtain encoded features; Obtain the lyrics duration feature of the target lyrics, and extend the duration feature of the encoding feature according to the lyrics duration feature to obtain the duration extended encoding feature; Determine the timbre features of the target timbre, concatenate the timbre features and the duration-extended coding features, and then extract acoustic features to obtain the target spectral features; A target song is synthesized based on the target spectral features, and the target song is matched with the target lyrics, the target musical score information, and the target timbre. The song synthesis method is executed through a song synthesis model, and the training method of the song synthesis model includes: An initial sample set is obtained, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio. The source lyrics duration information is obtained by duration annotation of the source lyrics of the recorded audio, and the source lyrics duration information includes duration annotation information of each phoneme in the source lyrics. Based on the audio transformation of the recorded audio in the initial samples, sample augmentation is performed to obtain an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after the audio transformation, the augmented lyrics duration information of the augmented audio, and the augmented musical score information of the augmented audio. The recorded audio and the augmented audio serve as sample audio, the original lyrics duration information and the augmented lyrics duration information serve as sample lyrics duration information, and the original musical score information and the augmented musical score information serve as sample musical score information. The audio transformation method includes at least one of pitch adjustment processing of the recorded audio or splicing processing of audio segments of the recorded audio. Multiple initial samples are augmented based on different audio transformation methods of the recorded audio. The transformation method of the recorded audio of the initial sample matches the duration transformation method of the original lyrics duration information of the initial sample and the musical score transformation method of the original musical score information of the initial sample. The model is pre-trained based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis. The audio of the target sound source is acquired, and timbre features are extracted based on the audio of the target sound source; the initial model for song synthesis is trained based on the timbre features to obtain the song synthesis model. The process of augmenting the audio samples based on the audio transformation recorded in the initial samples to obtain an augmented sample set includes at least one of the following: The recorded audios of multiple initial samples are subjected to pitch adjustment processing to obtain augmented audios corresponding to each recorded audio. The duration of the source lyrics for each recorded audio of the multiple initial samples is used as the duration of the augmented lyrics for the corresponding augmented audio. According to the pitch adjustment processing of the recorded audios of the multiple initial samples, the notes of the source musical score information of the multiple recorded audios are subjected to scale adjustment processing to obtain augmented musical score information corresponding to each augmented audio. The scale raising processing is matched with the pitch sharpening processing, and the scale lowering processing is matched with the pitch lowering processing. An augmented sample set is formed based on each augmented audio, the duration of the augmented lyrics for each augmented audio, and the augmented musical score information for each augmented audio. The recorded audio of multiple initial samples is divided into segments to obtain audio segments corresponding to each recorded audio. For each recorded audio, the audio segments are concatenated in adjacent order to obtain multiple augmented audio segments of the corresponding recorded audio. Based on the division of the recorded audio of the multiple initial samples, the source lyrics duration information of the multiple initial samples is divided to obtain lyrics duration information segments corresponding to each audio segment. Based on the concatenation of the audio segments, the lyrics duration information segments of each audio segment are concatenated. Next, augmented lyric duration information corresponding to each augmented audio is obtained; according to the division processing of the recorded audio of multiple initial samples, the source score information of multiple initial samples is divided to obtain score information fragments corresponding to each audio segment; according to the splicing processing of each audio segment, the score information fragments of each audio segment are spliced ​​to obtain augmented score information corresponding to each augmented audio; an augmented sample set is formed based on each augmented audio, the augmented lyric duration information of each augmented audio, and the augmented score information of each augmented audio.

11. The method according to claim 10, characterized in that, The method further includes: Based on the lyric features of the target lyrics, the musical score features of the target musical score information, and the timbre features of the target timbre, duration prediction processing is performed to obtain the phoneme duration corresponding to each phoneme in the target lyrics; The duration feature of the coding feature is extended according to the phoneme duration of each phoneme to obtain the duration-extended coding feature.

12. A training device for a song synthesis model, characterized in that, The device includes: The acquisition module is used to acquire an initial sample set, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio. The source lyrics duration information is obtained by duration annotation of the source lyrics of the recorded audio, and the source lyrics duration information includes duration annotation information of each phoneme in the source lyrics. An augmentation module is used to augment samples based on audio transformations of the recorded audio in the initial samples, obtaining an augmented sample set. The augmented samples in the augmented sample set include augmented audio obtained after the audio transformation, augmented lyrics duration information of the augmented audio, and augmented musical score information of the augmented audio. The recorded audio and the augmented audio serve as sample audio, the original lyrics duration information and the augmented lyrics duration information serve as sample lyrics duration information, and the original musical score information and the augmented musical score information serve as sample musical score information. The audio transformation method includes at least one of pitch adjustment processing of the recorded audio or splicing processing of audio segments of the recorded audio. Multiple initial samples are augmented based on different audio transformation methods of the recorded audio. The transformation method of the recorded audio of the initial samples matches the duration transformation method of the original lyrics duration information of the initial samples and the musical score transformation method of the original musical score information of the initial samples. The pre-training module is used to pre-train the model based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis. An extraction module is used to acquire the audio of a target sound source and extract timbre features based on the audio of the target sound source; The training module is used to train the initial song synthesis model based on the timbre features to obtain the song synthesis model; The augmentation module is also used for at least one of the following: The recorded audios of multiple initial samples are subjected to pitch adjustment processing to obtain augmented audios corresponding to each recorded audio. The duration of the source lyrics for each recorded audio of the multiple initial samples is used as the duration of the augmented lyrics for the corresponding augmented audio. According to the pitch adjustment processing of the recorded audios of the multiple initial samples, the notes of the source musical score information of the multiple recorded audios are subjected to scale adjustment processing to obtain augmented musical score information corresponding to each augmented audio. The scale raising processing is matched with the pitch sharpening processing, and the scale lowering processing is matched with the pitch lowering processing. An augmented sample set is formed based on each augmented audio, the duration of the augmented lyrics for each augmented audio, and the augmented musical score information for each augmented audio. The recorded audio of multiple initial samples is divided into segments to obtain audio segments corresponding to each recorded audio. For each recorded audio, the audio segments are concatenated in adjacent order to obtain multiple augmented audio segments of the corresponding recorded audio. Based on the division of the recorded audio of the multiple initial samples, the source lyrics duration information of the multiple initial samples is divided to obtain lyrics duration information segments corresponding to each audio segment. Based on the concatenation of the audio segments, the lyrics duration information segments of each audio segment are concatenated. Next, augmented lyric duration information corresponding to each augmented audio is obtained; according to the division processing of the recorded audio of multiple initial samples, the source score information of multiple initial samples is divided to obtain score information fragments corresponding to each audio segment; according to the splicing processing of each audio segment, the score information fragments of each audio segment are spliced ​​to obtain augmented score information corresponding to each augmented audio; an augmented sample set is formed based on each augmented audio, the augmented lyric duration information of each augmented audio, and the augmented score information of each augmented audio.

13. The apparatus according to claim 12, characterized in that, The device further includes: a pre-training module, configured to obtain sample audio, sample lyric duration information corresponding to the sample audio, and sample musical score information corresponding to the sample audio from the set consisting of the initial sample set and the augmented sample set; perform feature encoding based on the sample lyric duration information and sample musical score information of the sample audio to obtain sample encoding features; extend the duration features of the sample encoding features according to the sample lyric duration features of the sample lyric duration information to obtain sample duration extended encoding features; extract sample timbre features of the sample audio, concatenate the sample timbre features and the sample duration extended encoding features, and then perform acoustic feature extraction to obtain predicted spectral features; synthesize a predicted song based on the predicted spectral features, construct a target loss function based on the synthesis loss between the predicted song and the sample audio; and perform model pre-training based on the target loss function to obtain an initial model for song synthesis.

14. The apparatus according to claim 13, characterized in that, The pre-training module is also used to perform duration prediction processing based on the sample lyrics duration information, the sample musical score information, and the sample timbre features to obtain the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information. Determine the phoneme duration loss between the predicted phoneme duration of each phoneme and the sample phoneme duration of each phoneme in the sample lyrics duration information; Determine the synthesis loss between the predicted song and the sample audio; construct a target loss function based on the phoneme duration loss and the synthesis loss.

15. The apparatus according to claim 14, characterized in that, The pre-training module is also used to determine the predicted syllable duration corresponding to each syllable in the sample lyrics duration information based on the predicted phoneme duration corresponding to each phoneme in the sample lyrics duration information. Determine the syllable duration loss between the predicted syllable duration of each syllable and the sample syllable duration of each syllable in the sample lyrics duration information; construct a target loss function based on the phoneme duration loss, the syllable duration loss, and the synthesis loss.

16. The apparatus according to claim 13, characterized in that, The pre-training module further performs gradient inversion processing on the encoded features of the samples, and classifies them based on the features obtained from the gradient inversion processing to obtain the classification result of the sample audio; and determines the adversarial loss between the classification result and the classification label of the sample audio. Determine the synthesis loss between the predicted song and the sample audio; construct a target loss function based on the adversarial loss and the synthesis loss.

17. The apparatus according to claim 13, characterized in that, The pre-training module is further configured to extract sample spectral features of the sample audio and determine the spectral loss between the predicted spectral features and the sample spectral features; determine the synthesis loss between the predicted song and the sample audio; and construct a target loss function based on the spectral loss and the synthesis loss.

18. The apparatus according to any one of claims 12 to 17, characterized in that, The initial model for song synthesis includes an initial duration model, an initial acoustic model, and an initial vocoder; the training module is used to adjust the parameters of the initial duration model, the initial acoustic model, and the initial vocoder based on the timbre characteristics of the target sound source, so as to obtain a song synthesis model that matches the timbre of the target sound source.

19. The apparatus according to any one of claims 12 to 17, characterized in that, The training module is used to acquire singing data of multiple target objects, train the initial model of song synthesis using the singing data of each target object, obtain a sub-model that matches the timbre of each target object, and integrate the sub-models into the song synthesis model; the song synthesis model provides multiple candidate timbres, and the multiple candidate timbres are matched one by one with each sub-model.

20. The apparatus according to claim 19, characterized in that, The training module is used to adjust the parameters of the initial duration model, initial acoustic model and initial vocoder in the initial song synthesis model using the singing data of each target object, to obtain a sub-model that matches the timbre of each target object, and to integrate the sub-models into the song synthesis model.

21. A song synthesis device, characterized in that, The device includes: The encoding module is used to acquire target lyrics and target musical score information, and to perform feature encoding based on the target lyrics and the target musical score information to obtain encoded features; An extension module is used to obtain the lyrics duration features of the target lyrics, and extend the duration features of the encoding features according to the lyrics duration features to obtain duration extended encoding features; The determination module is used to determine the timbre features of the target timbre, and then perform acoustic feature extraction after concatenating the timbre features and the duration-extended coding features to obtain the target spectral features; A synthesis module is used to synthesize a target song based on the target spectral features, wherein the target song is matched with the target lyrics, the target musical score information and the target timbre; The target song is synthesized using a song synthesis model, and the training method for the song synthesis model includes: An initial sample set is obtained, which includes initial samples from multiple sound sources. The initial samples include recorded audio, source lyrics duration information of the recorded audio, and source musical score information of the recorded audio. The source lyrics duration information is obtained by duration annotation of the source lyrics of the recorded audio, and the source lyrics duration information includes duration annotation information of each phoneme in the source lyrics. Based on the audio transformation of the recorded audio in the initial samples, sample augmentation is performed to obtain an augmented sample set. The augmented samples in the augmented sample set include the augmented audio obtained after the audio transformation, the augmented lyrics duration information of the augmented audio, and the augmented musical score information of the augmented audio. The recorded audio and the augmented audio serve as sample audio, the original lyrics duration information and the augmented lyrics duration information serve as sample lyrics duration information, and the original musical score information and the augmented musical score information serve as sample musical score information. The audio transformation method includes at least one of pitch adjustment processing of the recorded audio or splicing processing of audio segments of the recorded audio. Multiple initial samples are augmented based on different audio transformation methods of the recorded audio. The transformation method of the recorded audio of the initial sample matches the duration transformation method of the original lyrics duration information of the initial sample and the musical score transformation method of the original musical score information of the initial sample. The model is pre-trained based on the initial sample set and the augmented sample set to obtain the initial model for song synthesis. The audio of the target sound source is acquired, and timbre features are extracted based on the audio of the target sound source; the initial model for song synthesis is trained based on the timbre features to obtain the song synthesis model. The process of augmenting the audio samples based on the audio transformation recorded in the initial samples to obtain an augmented sample set includes at least one of the following: The recorded audios of multiple initial samples are subjected to pitch adjustment processing to obtain augmented audios corresponding to each recorded audio. The duration of the source lyrics for each recorded audio of the multiple initial samples is used as the duration of the augmented lyrics for the corresponding augmented audio. According to the pitch adjustment processing of the recorded audios of the multiple initial samples, the notes of the source musical score information of the multiple recorded audios are subjected to scale adjustment processing to obtain augmented musical score information corresponding to each augmented audio. The scale raising processing is matched with the pitch sharpening processing, and the scale lowering processing is matched with the pitch lowering processing. An augmented sample set is formed based on each augmented audio, the duration of the augmented lyrics for each augmented audio, and the augmented musical score information for each augmented audio. The recorded audio of multiple initial samples is divided into segments to obtain audio segments corresponding to each recorded audio. For each recorded audio, the audio segments are concatenated in adjacent order to obtain multiple augmented audio segments of the corresponding recorded audio. Based on the division of the recorded audio of the multiple initial samples, the source lyrics duration information of the multiple initial samples is divided to obtain lyrics duration information segments corresponding to each audio segment. Based on the concatenation of the audio segments, the lyrics duration information segments of each audio segment are concatenated. Next, augmented lyric duration information corresponding to each augmented audio is obtained; according to the division processing of the recorded audio of multiple initial samples, the source score information of multiple initial samples is divided to obtain score information fragments corresponding to each audio segment; according to the splicing processing of each audio segment, the score information fragments of each audio segment are spliced ​​to obtain augmented score information corresponding to each augmented audio; an augmented sample set is formed based on each augmented audio, the augmented lyric duration information of each augmented audio, and the augmented score information of each augmented audio.

22. The apparatus according to claim 21, characterized in that, The device further includes: The duration prediction module is used to perform duration prediction processing based on the lyric features of the target lyrics, the musical score features of the target musical score information, and the timbre features of the target timbre, to obtain the phoneme duration corresponding to each phoneme in the target lyrics; and to extend the duration features of the encoding features according to the phoneme duration of each phoneme to obtain the duration extended encoding features.

23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images