Speech and singing voice synthesis method, training method, and related apparatus

Abstract

Disclosed in the present disclosure are a speech and singing voice synthesis method, a training method, and a related apparatus. The training method comprises: acquiring speech training data; acquiring singing voice training data; converting text data into text embedding, and converting speech data into a speech embedding; concatenating the text embedding and the speech embedding to form encoded speech training data; segmenting lyrics data into phrases and converting the phrases into phrase embeddings, segmenting singing voice data into singing voice segments corresponding to the phrases and converting the singing voice segments into singing voice segment embeddings, extracting, from musical score data, pitch sequences corresponding to the phrases and / or singing voice segments, and converting the pitch sequences into pitch embeddings; concatenating the phrase embeddings, the singing voice segment embeddings, and the pitch embeddings to form encoded singing voice training data; and inputting the encoded speech training data and the encoded singing voice training data into a speech and singing voice synthesis module of an initial model for training, to obtain a target model upon training.

Description

Speech and singing synthesis methods, training methods and related devices

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to Chinese Patent Application No. 2024106721873, filed on May 28, 2024, entitled "Speech and Singing Synthesis Method, Training Method and Apparatus, Model", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of artificial intelligence audio processing technology, and specifically to speech and singing synthesis methods, training methods, and related apparatus. Background Technology

[0004] In the field of music production, several vocal synthesis (SVS) software programs have been developed, such as Synthesizer V, Vocaloid, and ACE Studio. These programs offer highly customizable editing interfaces, such as a piano roll view, allowing users to manually adjust various attributes like melody and timbre to produce high-quality vocal effects. While these techniques are usable for vocal production, they are generally limited to singing and have a relatively high learning curve, restricting their widespread application.

[0005] Meanwhile, while SVS technology based on neural network models offers ease of use and improved results, it faces challenges such as insufficient vocal data and limited functionality.

[0006] The background description is provided for the purpose of understanding the relevant technologies in this field and is not intended as an admission of prior art.

[0007] Application content

[0008] Therefore, embodiments of this disclosure provide a speech and singing synthesis method, training method and apparatus, and model that can at least partially mitigate or resolve the aforementioned technical limitations.

[0009] This disclosure provides a training method, the training method comprising:

[0010] Acquire speech training data, which includes multiple matching text data and multiple speech data;

[0011] Acquire singing training data, which includes multiple matching lyric data, multiple singing data, and multiple sheet music data;

[0012] The text data is converted into text embedding encoding, and the speech data is converted into speech embedding encoding;

[0013] The text embedding encoding and the speech embedding encoding are concatenated to form speech training encoding data;

[0014] The lyrics data is divided into phrases and the phrases are converted into phrase embedding codes. The singing data is divided into singing segments corresponding to the phrases and the singing segments are converted into singing segment embedding codes. The pitch sequences corresponding to the phrases and / or singing segments are extracted from the music score data and the pitch sequences are converted into pitch embedding codes.

[0015] The sentence embedding code, the vocal segment embedding code, and the pitch embedding code are concatenated to form vocal training encoding data;

[0016] The speech training encoded data and the singing training encoded data are input into the speech and singing synthesis modules of the initial model for training, so as to obtain the target model including the trained speech and singing synthesis modules.

[0017] In some embodiments of this disclosure, extracting pitch sequences corresponding to the phrases and / or vocal segments from the musical score data and converting the pitch sequences into pitch embedding codes includes:

[0018] Pitch information is extracted from the musical score data;

[0019] Obtain the duration of each pitch from the extracted pitch information;

[0020] Obtain the timestamp of the corresponding sentence in the lyrics data;

[0021] The pitch embedding code and the sentence embedding code are aligned based on the duration of each pitch and the timestamp of the corresponding sentence.

[0022] In some embodiments of this disclosure, extracting pitch sequences corresponding to the phrases and / or vocal segments from the musical score data and converting the pitch sequences into pitch embedding codes includes:

[0023] Pitch information is extracted from the musical score data;

[0024] Obtain the duration of each pitch from the extracted pitch information;

[0025] Obtain the duration of the singing segment from the singing data;

[0026] The pitch embedding code and the song segment embedding code are aligned according to the pitch duration and the duration of each pitch segment.

[0027] In some embodiments of this disclosure, the sheet music data includes a MIDI file of the song.

[0028] In some embodiments of this disclosure, the step of inputting speech training encoded data and singing training encoded data into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module includes:

[0029] Confirm whether the input training encoding data includes pitch encoding information, and if the pitch encoding information exists, generate a hidden state sequence from the pitch encoding information to control the pitch range, rhythm and melody of the predicted audio waveform.

[0030] This disclosure also provides a method for synthesizing speech and singing, including:

[0031] Obtain the data to be synthesized, which includes at least text data;

[0032] Determine whether the data to be synthesized contains musical score data;

[0033] If the music score data is not included, the text data in the data to be synthesized is embedded and encoded as the first encoded data, and the speech synthesis encoded data is input into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target speech.

[0034] If the data includes musical score data, the text data in the data to be synthesized is embedded and encoded to obtain second encoded data, and the second encoded data is input into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target singing voice.

[0035] In some embodiments of this disclosure, the speech and singing synthesis composite model includes a target model trained according to the training method described in any one of the embodiments of this disclosure.

[0036] In some embodiments of this disclosure, the method further includes:

[0037] If the data includes musical score data, the musical score data in the data to be synthesized is embedded and encoded to obtain third encoded data, and the third encoded data is input into the speech and singing synthesis module of the trained speech and singing synthesis composite model, wherein the speech and singing synthesis module generates a hidden state sequence of the third encoded data to control the vocal range, rhythm and melody of the target singing voice.

[0038] In some embodiments of this disclosure, the method further includes:

[0039] The target vocals and the melody generated from the musical score data are mixed to obtain the song to be played.

[0040] This disclosure also provides a speech and singing synthesis composite model, the composite model including a target model trained according to the method described in any embodiment of this disclosure.

[0041] This disclosure also provides a speech synthesis model, which includes a target model trained according to any one of the embodiments of this disclosure.

[0042] This disclosure also provides a singing voice synthesis model, which includes a target model trained according to any one of the embodiments of this disclosure.

[0043] This disclosure also provides a training device, including:

[0044] The first acquisition module is configured to acquire voice training data, which includes multiple matching text data and multiple voice data.

[0045] The second acquisition module is configured to acquire singing training data, which includes multiple matching lyric data, multiple singing data, and multiple musical score data.

[0046] The first encoding module is configured to convert the text data into text embedding encoding and the speech data into speech embedding encoding.

[0047] The first splicing module is configured to splice the text embedding code and the speech embedding code into speech training encoding data;

[0048] The second encoding module is configured to segment the lyrics data into phrases and convert the phrases into phrase embedding codes, segment the singing data into singing segments corresponding to the phrases and convert the singing segments into singing segment embedding codes, and extract the pitch sequence corresponding to the phrases and / or singing segments from the score data and convert the pitch sequence into pitch embedding codes.

[0049] The second splicing module is configured to splice the sentence embedding code, the vocal segment embedding code, and the pitch embedding code into vocal training encoding data.

[0050] The model training module is configured to input speech training encoded data and singing training encoded data into the speech and singing synthesis module of the initial model for training, so as to obtain a target model including the trained speech and singing synthesis module.

[0051] This disclosure also provides a speech and singing synthesis apparatus, including:

[0052] The third acquisition module is configured to acquire data to be synthesized, wherein the data to be synthesized includes at least text data.

[0053] The judgment module is configured to determine whether the data to be synthesized contains musical score data;

[0054] The synthesis module is configured to embed and encode the text data in the data to be synthesized as the first encoded data if it does not contain musical score data, and input the speech synthesis encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target speech;

[0055] The synthesis module is further configured to, if it contains musical score data, embed and encode the text data in the data to be synthesized to obtain second encoded data, and input the second encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target singing voice.

[0056] This disclosure also provides an electronic device including a processor and a memory storing a computer program, the processor being configured to implement the methods described in any embodiment of this disclosure when the computer program is executed.

[0057] This disclosure also provides a storage medium storing a computer program that, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0058] The training method provided in this embodiment involves: acquiring speech training data; acquiring singing training data; converting the text data of the speech training data into text embedding codes, converting the speech data into speech embedding codes, and concatenating the text embedding codes and speech embedding codes into speech training encoded data; segmenting the lyrics data of the singing training data into sentences and converting the sentences into sentence embedding codes, segmenting the singing data into singing segments corresponding to the sentences and converting the singing segments into singing segment embedding codes, extracting the pitch sequences of the corresponding sentences and / or singing segments from the musical score data and converting the pitch sequences into pitch embedding codes, and concatenating the sentence embedding codes, singing segment embedding codes, and pitch embedding codes into singing training encoded data; and inputting the speech training encoded data and singing training encoded data into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module. On the one hand, the resulting target model can function as both a speech synthesis model, which, when used as a speech synthesis model, can fully utilize the singing data during training to improve the pitch and bass effects of speech, and a singing synthesis model, which, when used as a singing synthesis model, uses speech data to assist in singing training, solving the problem of limited singing data. On the other hand, the resulting target model can also function as a composite model of speech synthesis and singing synthesis, combining the advantages mentioned above to achieve the beneficial effect of being able to both speak and sing, with excellent results in both areas.

[0059] Accordingly, the speech and singing synthesis method provided in the embodiments of this disclosure can determine whether the data to be synthesized contains musical score data; and selectively output different data based on whether musical score data is included, thus achieving the beneficial effect of both speaking and singing, with excellent results.

[0060] Other optional features and technical effects of the embodiments of this disclosure are described in part below, and in part will be apparent from reading this document. Attached Figure Description

[0061] To more clearly illustrate the embodiments of this disclosure or the technical solutions in the prior art, the accompanying drawings used will be briefly described below. It should be understood that the drawings described below only cover some embodiments of this disclosure. Those skilled in the art can obtain other drawings based on these drawings without creative work. The purpose of these drawings is to better illustrate the technical details to aid in understanding the implementation of this disclosure, wherein:

[0062] Figure 1 shows a schematic flowchart of the training method according to an embodiment of the present disclosure;

[0063] Figure 2 shows a schematic process diagram of the training method of this disclosure during training;

[0064] Figure 3 shows a schematic flowchart of the training method according to an embodiment of the present disclosure;

[0065] Figure 4 shows a schematic flowchart of the training method according to an embodiment of the present disclosure;

[0066] Figure 5 shows an exemplary model structure diagram of the training method and synthesis method applicable to embodiments of this disclosure;

[0067] Figure 6 shows a schematic flowchart of the speech and singing synthesis method according to an embodiment of the present disclosure;

[0068] Figure 7 shows an exemplary structural diagram of a training device according to an embodiment of the present disclosure; and

[0069] Figure 8 shows an exemplary structural schematic diagram of an electronic device capable of implementing the methods according to embodiments of the present disclosure. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0071] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0072] In the embodiments of this disclosure, a "phoneme" is the smallest unit of speech defined based on the natural properties of speech, and one articulation action forms one phoneme. A "phoneme sequence" is a sequence composed of several phonemes.

[0073] Currently, singing voice synthesis (SVS) technology based on neural network models has made significant progress. However, it also faces challenges such as insufficient singing voice data available for training.

[0074] Furthermore, current singing voice synthesis (SVS) models are basically only capable of singing, making it necessary to provide an additional text-to-speech (TTS) model to enable AI to both speak and sing. Switching between the two models is difficult to operate and costly. Moreover, speech and singing voices have many differences, such as a wider vocal range, higher requirements for sound quality, and requirements for rhythm and melody, making it impractical to directly use a text-to-speech (TTS) model for singing voice synthesis (SVS).

[0075] Therefore, embodiments of this disclosure intend to provide a training method by which a model trained possesses combined capabilities of speech synthesis and vocal synthesis. Furthermore, the model trained using this method, when dedicated to vocal synthesis (SVS), also solves the problem of insufficient vocal data available for training. The inventors have also additionally discovered that a model trained using this method, when dedicated to text-to-speech (TTS), exhibits a surprisingly significant improvement in pitch and bass levels.

[0076] As shown in Figure 1, this disclosure provides a training method. The training method may include the following steps:

[0077] S110: Obtain speech training data, which includes multiple matching text data and multiple speech data.

[0078] In some embodiments of this disclosure, the speech training data may be conventional speech data that can be used to train a TTS model. In some exemplary embodiments, the speech training data may include text-speech pairs, but it is conceivable that the speech training data may be provided in a more raw form and then preprocessed to form text-speech pairs, which is not a limitation of this disclosure.

[0079] S120: Obtain singing training data, which includes multiple matching lyric data, multiple singing data, and multiple musical score data.

[0080] In some embodiments of this disclosure, the matching lyrics data and vocal data may include the lyrics of the entire song and the corresponding vocals. The vocals may be melodic, but may also be unmelodic, and this disclosure does not impose any restrictions on this.

[0081] In this embodiment of the disclosure, the singing data does not include music consisting solely of melodies. In another embodiment, the matching lyrics data and singing data may be lyrics segments already divided according to lyric phrases and their corresponding singing data.

[0082] In some embodiments of this disclosure, the lyrics data can be any suitable lyrics file. In one example, the lyrics data may include an LRC (Lyric Resource Computer) file, but this disclosure is not limited thereto. LRC files achieve synchronization with music by adding time stamps next to the lyrics text. The structure of an LRC file includes time stamps and lyrics text. The time stamps in an LRC file are enclosed in square brackets (e.g., [00:01.00]) and indicate the specific time points at which the lyrics should be displayed, corresponding to the playback time of the music file. The lyrics text that should be displayed at the corresponding time point is immediately followed by the time stamp.

[0083] In some embodiments of this disclosure, the sheet music data may include a MIDI file of a song, but this disclosure is not limiting in this regard; for example, the sheet music data may include MusicXML.

[0084] MIDI, or Musical Instrument Digital Interface, is a real-time communication protocol used between electronic musical instruments, synthesizers, and other performance devices for the transmission of real-time performance data between hardware. In this embodiment, the MIDI file refers to a file conforming to the MIDI protocol file format. A MIDI file may contain several MIDI messages for transmitting real-time musical performance information, presented as instructions containing information such as pitch, tempo, and effects parameters. By way of example, and not limitation, MIDI messages include note on and note off.

[0085] S130: Convert the text data into text embedding code and the speech data into speech embedding code.

[0086] In this embodiment of the disclosure, as illustrated in Figure 2, text segments can be segmented to generate text embeddings. For example, string matching-based or rule-based methods can be used, such as forward maximum matching segmentation algorithms, reverse maximum matching segmentation algorithms, and statistical methods. Before segmentation, some embodiments also include text normalization of the text segments, which involves converting character sets (e.g., uniformly using Unicode encoding), eliminating case differences, and removing irrelevant characters (e.g., HTML tags). The text embedding process may include constructing a vocabulary, performing word frequency statistics, filtering words to remove high-frequency stop words and low-frequency words, and assigning a unique numerical index to each token in the vocabulary. Token encoding may use One-Hot encoding to convert each token into a high-dimensional sparse vector, limiting the vector size as necessary, and using special tokens to replace uncommon words. The process of learning dense vector representations may involve selecting pre-trained models such as Word2Vec, GloVe, FastText, or BERT, or training a custom model for a specific task, to learn the vector representation of the tokens through context. These vectors can capture complex relationships between words, such as semantic similarity and grammatical rules. Subsequent dimensionality reduction, such as PCA or t-SNE, can be performed to optimize the vector space, making similar words appear closer together. However, the text embedding in this disclosure is not limited to this.

[0087] In this embodiment of the disclosure, as schematically shown in FIG2, useful information or features, such as frequency components, energy, rhythm, etc., can be extracted from the original audio signal to generate speech embedding. These features capture important properties of the audio, but are usually still continuous numerical values. This information and features are then discretized, which involves mapping feature vectors to a high-dimensional space. The embedding process can be implemented using any suitable neural network model, such as a deep learning model. However, the speech embedding in this embodiment of the disclosure is not limited to this.

[0088] S140: The text embedding code and the speech embedding code are concatenated to form speech training code data.

[0089] S150: The lyrics data is divided into sentences and the sentences are converted into sentence embedding codes. The vocal data is divided into vocal segments corresponding to the sentences and the vocal segments are converted into vocal segment embedding codes. The pitch sequence corresponding to the sentences and / or vocal segments is extracted from the musical score data and the pitch sequence is converted into pitch embedding codes.

[0090] In this embodiment of the disclosure, lyrics can be divided into multiple sentences based on characteristics such as punctuation marks. After obtaining the sentences, they can be embedded using the text embedding method described above.

[0091] In this embodiment of the disclosure, the corresponding vocal segment can be obtained based on the timestamp (time tag) of a lyrics file, such as an LRC file. After obtaining the vocal segment, vocal segment embedding can be performed using the voice embedding method described above.

[0092] In this embodiment of the disclosure, as schematically shown by the dashed line in FIG2, compared with the embeddings of the voice training data, the pitch information of the score file, such as the MIDI file, can also be extracted for the singing training data.

[0093] Taking MIDI messages in a MIDI file as an example, the pitch value of each note can be recorded in the note trigger (NOTEON) event. Correspondingly, MIDI information can be extracted from the MIDI messages of each note in the MIDI file, thereby obtaining the pitch sequence corresponding to the song. After appropriate processing, the pitch sequence corresponding to each phrase or vocal segment can be obtained. The pitch sequence includes the pitch information of each note, describing the high and low points of the melody. Furthermore, the extracted pitch information (pitch sequence) can be converted into pitch embedding code using a suitable embedding method. For clarification, the pitch sequence or pitch embedding code converted from the pitch sequence described herein is not the vector dimension that may represent pitch in audio embedding. In the embodiments of this disclosure, any suitable embedding method can be used to convert the pitch sequence into pitch embedding code. As an example, the above or other text embedding methods can be reasonably modified to obtain the pitch embedding code.

[0094] In a preferred embodiment of this disclosure, as illustrated in FIG5, pitch duration information can be aligned with the corresponding vocal segments and / or lyric phrases.

[0095] In some embodiments, pitch duration information can be aligned with lyric phrases.

[0096] As shown in Figure 3, step S150 may include:

[0097] S151: Extract pitch information from the musical score data.

[0098] S152: Obtain the duration of each pitch of the extracted pitch information.

[0099] S153: Obtain the timestamp of the corresponding sentence in the lyrics data.

[0100] S154: Align the pitch embedding code and the sentence embedding code according to the duration of each pitch and the timestamp of the corresponding sentence.

[0101] Specifically, MIDI files store musical information in digital form, including not only pitch but also the duration of each pitch. Accordingly, in step S151, pitch information, such as the pitch of the corresponding note, is extracted from the MIDI file (MIDI message); and in step S152, the extracted pitch duration information, more precisely, the note duration information, is obtained. In some embodiments, the duration of each note's pitch can be determined by the note's on / off state. Accordingly, the pitch and duration information in the MIDI file are extracted to obtain the melody of the song and the duration of each note.

[0102] As a further explanation, and not a limitation, regarding MIDI file timing information, time in MIDI is typically represented by "ticks." These ticks, combined with the song's BPM (beats per minute), can be converted into actual duration. This can be understood as time divided by notes, specifically TPQN (Ticks Per Quarter-Note), which indicates how many ticks are contained in a quarter note. Based on this, tick values ​​in a MIDI file can be converted into actual time. For example, converting ticks to seconds or milliseconds depends on the tempo (BPM) setting in the MIDI file.

[0103] In step S153, the timestamp of the corresponding clause can be obtained. For example, in an LRC file, the timestamp information of the corresponding clause can be obtained by retrieving the timestamp.

[0104] This allows for the alignment of pitch duration information with the corresponding lyrics based on the duration of each lyric. As a result, compared to conventional text-audio pairs, each text-audio pair obtained from singing training data is supplemented with corresponding pitch and pronunciation length information.

[0105] In a further preferred embodiment, alignment of lyric phrases with vocal segments can also be provided.

[0106] In an alternative embodiment, pitch duration information can be aligned with the vocal segment.

[0107] As shown in Figure 4, step S150 may include:

[0108] S151': Extract pitch information from the musical score data.

[0109] S152': Obtain the duration of each pitch of the extracted pitch information.

[0110] S153': Obtain the duration of the singing segment from the singing data.

[0111] S154': Align the pitch embedding code and the song segment embedding code according to the pitch duration and the duration of the song segment.

[0112] Steps S151' and S152' can be referred to steps S151 and S152.

[0113] After extracting pitch and duration information from the MIDI using step S152' to obtain the melody of the song and the duration of each note, the pitch sequence can be aligned with the vocal segment by fine-tuning the time axis of the vocal segment. One possible alignment method is Dynamic Time Warping (DTW).

[0114] In a further preferred embodiment, alignment of lyric phrases with vocal segments can also be provided.

[0115] In the embodiments shown in Figure 3 or Figure 4, alignment is performed before embedding or direct alignment of pitch embeddings and lyric phrases and / or vocal segment embeddings is included.

[0116] Therefore, by means of the alignment, the latent states of pitch transition information of the song melody can be learned during training. Accordingly, in the training step S160 below, a sequence of latent states for controlling the range, rhythm, and melody of the predicted audio waveform can be generated from the pitch coding information.

[0117] S160: The sentence embedding code, the vocal segment embedding code, and the pitch embedding code are concatenated to form vocal training encoding data.

[0118] In the embodiments of this disclosure, the order of the spliced ​​vocal training encoding data can be any suitable form, and this disclosure does not limit it.

[0119] S170: Input the speech training encoding data and the singing training encoding data into the speech and singing synthesis module of the initial model for training, so as to obtain the target model including the trained speech and singing synthesis module.

[0120] In embodiments of this disclosure, the speech and vocal synthesis module can be any suitable TTS model that has been adapted. In one exemplary embodiment, the speech and vocal synthesis module can be based on the FastSpeech model architecture, but this disclosure is not limited thereto, and any suitable TTS or SVS base model or a suitably adapted modified model can be used.

[0121] FastSpeech is a novel feedforward network that offers performance improvements in several areas compared to traditional TTS models. FastSpeech can quickly generate stable, controllable, and high-quality Mel spectrograms.

[0122] In some embodiments of this disclosure, as shown in FIG5, the FastSpeech mainly includes the following parts:

[0123] a. Feed-Forward Transformer Block (FFT Block): It includes a self-attention module and a 1D convolution module, which can convert the input phoneme sequence into a Mel spectrogram sequence to prepare for subsequent speech synthesis steps.

[0124] b. Length Regulator: Since the length of the phoneme sequence is less than the length of the Mel spectrogram sequence, one phoneme corresponds to multiple Mel spectrograms. The number of Mel spectrograms aligned with a phoneme is called the phoneme duration. The length regulator extends the hidden sequence of the phoneme according to the duration to match the length of the Mel spectrogram sequence. The phoneme duration can be increased or decreased proportionally to adjust the tempo.

[0125] c. Linear Layer: A linear layer is a basic neural network layer configured to perform linear transformations. In deep learning and neural networks, it is also often called a fully connected layer or a dense layer, and can be configured to convert the network's intermediate representations into the final output format, i.e., a Mel-spectral graph.

[0126] As shown in Figure 5, compared to the conventional FastSpeech architecture, in this embodiment of the disclosure, the FastSpeech architecture is modified to be suitable for inputting variable encoding formats. For example, it can optionally accept concatenated data of text embedding + audio embedding, or concatenated data of text embedding + audio embedding + pitch embedding. In some embodiments, the position encoding module can be configured in any suitable manner to enable it to receive variable-length encodings. In another embodiment, the concatenation (steps S140 and S160) can be configured such that when concatenating speech training encoding data, an empty encoding vector segment corresponding to the pitch embedding encoding length is reserved. However, this disclosure is not limited to this.

[0127] During training, the predicted Mel spectra are compared with the speech or singing labels to iteratively update the Fastspeech-based speech and singing synthesis modules.

[0128] During the inference process, the predicted Mel spectrum is input into a waveform generation neural network model, such as a dedicated vocoder (WaveNet, WaveGlow, and WaveRNN, etc.), to convert the sound feature Mel spectrum into the final target audio waveform. Depending on whether the input data contains MIDI or pitch information, the target audio waveform is either a target speech waveform or a target singing waveform.

[0129] As previously described, step S160 may optionally include: confirming whether the input training encoded data includes pitch encoding information, and if the pitch encoding information is present, generating a hidden state sequence from the pitch encoding information to control the pitch range, rhythm, and melody of the predicted audio waveform. The speech and singing synthesis modules of the initial model can be adapted as needed to introduce hidden state sequences to control the input text / speech / singing embedding codes (or other encoded sequences converted from them).

[0130] Accordingly, embodiments of this disclosure may also provide a method for selectively synthesizing speech and singing.

[0131] As shown in Figure 6, this embodiment of the present disclosure provides a method for synthesizing speech and singing voice, including the following steps:

[0132] S610: Obtain the data to be synthesized, wherein the data to be synthesized includes at least text data;

[0133] S620: Determine whether the data to be synthesized contains musical score data;

[0134] If the sheet music data is not included, then execute S630 and S640:

[0135] S630: Embedded and encoded the text data in the data to be synthesized as the first encoded data;

[0136] S640: Input the speech synthesis encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target speech.

[0137] If sheet music data is included, execute S650 and S660:

[0138] S650: Embedding and encoding the text data in the data to be synthesized to obtain second encoded data;

[0139] S660: Input the second encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model.

[0140] The speech and singing synthesis composite model preferably includes the target model trained according to any one of the training methods in the embodiments of this disclosure.

[0141] In an optional embodiment of this disclosure, during inference, it is not necessary to extract pitch information from the score data; the score file can be used to determine whether it is a singing voice synthesis or a speech synthesis.

[0142] However, in a preferred embodiment, pitch information can also be extracted from the score data to assist in vocal synthesis during inference.

[0143] In some embodiments of this disclosure, if musical score data is included, steps S670 and S680 may also be executed optionally:

[0144] S670: Embed the musical score data in the data to be synthesized to obtain third encoded data;

[0145] S680: Input the third encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model.

[0146] Specifically, the speech and singing synthesis module generates a hidden state sequence from the third encoded data to control the vocal range, rhythm, and melody of the target singing voice.

[0147] In some embodiments of this disclosure, the method further includes: mixing the target vocalization with a melody generated from the score data to obtain a song to be played. In this embodiment, the score data can be processed by any suitable audio decoder to generate a melody, and then mixed with the generated vocalization to obtain a song to be played with a melody.

[0148] Furthermore, the target model trained by the training method described in the embodiments of this disclosure can also be configured for various different purposes.

[0149] In some embodiments, a speech and singing synthesis composite model is provided, which is a target model trained according to the training method described in any one of the embodiments of this disclosure. Optionally, the speech and singing synthesis composite model may include a trained speech and singing synthesis module. The speech and singing synthesis composite model may optionally include an encoding module configured to encode speech data and / or singing data (including sheet music files) into corresponding encoding sequences, which may, for example, employ a similar encoding module used in the training method. This speech and singing synthesis composite model, while possessing the advantages of improved pitch and bass effects in speech and addressing the problem of small training data volume in singing, enables the model to achieve the beneficial effect of being able to both speak and sing, with excellent results in both areas.

[0150] In other embodiments, a speech synthesis model is provided, which may include a target model trained according to the training method described in any one of the embodiments of this disclosure. The trained speech and vocal synthesis modules of this speech synthesis model may be configured only to synthesize speech. Optionally, the speech synthesis model may have a conventional speech synthesis coding module, i.e., a coding module without pitch sequence coding capabilities. This speech synthesis model can fully utilize the vocal data from training to improve the pitch and bass effects of the speech.

[0151] In other embodiments, a singing voice synthesis model is provided, which may include a target model trained according to the training method described in any one of the embodiments of this disclosure. The trained speech and singing voice synthesis modules of this singing voice synthesis model may be configured solely to synthesize singing voices. Optionally, the singing voice synthesis model may have a conventional singing voice synthesis encoding module, or it may have an encoding module similar to one configured to encode singing voice data (including sheet music files) into corresponding encoding sequences. Optionally, the singing voice synthesis model may also include additional filters that can filter out input data that does not conform to the input format, such as input data without sheet music information. During training, the singing voice synthesis model uses speech data to assist in singing voice training, addressing the problem of small training data volume for singing voices.

[0152] As shown in Figure 7, this embodiment of the present disclosure provides a training device 700, which may include: a first acquisition module 710, a second acquisition module 720, a first encoding module 730, a first splicing module 740, a second encoding module 750, a second splicing module 760, and a model training module 770.

[0153] In this embodiment, the first acquisition module 710 is configured to acquire speech training data, which includes multiple matching text data and multiple speech data. The second acquisition module 720 is configured to acquire singing training data, which includes multiple matching lyric data, multiple singing data, and multiple musical score data. The first encoding module 730 is configured to convert the text data into text embedding codes and the speech data into speech embedding codes. The first splicing module 740 is configured to splice the text embedding codes and the speech embedding codes into speech training encoded data. The second encoding module 750 is configured to segment the lyric data into phrases and convert the phrases into phrase embedding codes, segment the singing data into singing segments corresponding to the phrases and convert the singing segments into singing segment embedding codes, extract pitch sequences corresponding to the phrases and / or singing segments from the musical score data and convert the pitch sequences into pitch embedding codes. The second splicing module 760 is configured to splice the sentence embedding code, the vocal segment embedding code, and the pitch embedding code into vocal training encoding data. The model training module 770 is configured to input the speech training encoding data and the vocal training encoding data into the speech and vocal synthesis module of the initial model for training, so as to obtain a target model including the trained speech and vocal synthesis module.

[0154] This disclosure also provides a speech and singing synthesis device, including a third acquisition module, a judgment module, and a synthesis module.

[0155] In this embodiment of the disclosure, the third acquisition module is configured to acquire data to be synthesized, the data to be synthesized including at least text data; the judgment module is configured to determine whether the data to be synthesized contains musical score data; the synthesis module is configured to, if it does not contain musical score data, embed and encode the text data in the data to be synthesized as first encoded data, and input the speech synthesis encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target speech; and the synthesis module is further configured to, if it contains musical score data, embed and encode the text data in the data to be synthesized to obtain second encoded data, and input the second encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target singing voice.

[0156] The apparatus, components, configurations, and features according to embodiments of this disclosure can be incorporated into the methods according to embodiments of this disclosure in a non-contradictory manner. Conversely, the methods, steps, sub-steps, and features according to embodiments of this disclosure can be incorporated into the apparatus according to embodiments of this disclosure in a non-contradictory manner.

[0157] In this disclosure, an electronic device is also provided, including: a processor and a memory storing a computer program, the processor being configured to implement the relevant methods of any embodiment when running the computer program.

[0158] Figure 8 illustrates a schematic diagram of an electronic device 800 or an electronic device 800 that can be used to implement the methods of the embodiments of the present disclosure. In some embodiments, the number of electronic devices may be more or less than the number shown in the figure. In some embodiments, it may be implemented using a single or multiple electronic devices. It may also be implemented using cloud or distributed electronic devices in some embodiments.

[0159] As shown in Figure 8, the electronic device 800 includes a processor 810 and a memory 820. The processor is configured to execute programs stored in the memory, which, when executed by a computer, can implement the methods, steps, or functions described in the above embodiments. The processor 810 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 810 and the memory 820 are interconnected via a bus 830. Input / output (I / O) interfaces may also be connected to the bus 830.

[0160] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, smartphone, personal computer, laptop computer, in-vehicle human-machine interface device, personal digital assistant, media player, navigation device, game console, tablet computer, wearable device, smart TV, Internet of Things system, smart home, industrial computer, server, or a combination thereof.

[0161] Although not shown, in embodiments of this disclosure, a storage medium is provided that stores a computer program configured to be run to implement the methods of any embodiment.

[0162] Storage media in embodiments of this disclosure include articles that are permanent and non-permanent, removable and non-removable, capable of storing information by any method or technology. Examples of storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium configured to store information accessible by a computing device.

[0163] The methods, programs, systems, apparatuses, etc., of the embodiments of this disclosure can be executed or implemented in a single or multiple networked computers, or practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks can be performed by remote processing devices connected via a communication network.

[0164] Those skilled in the art will understand that the embodiments described in this specification can be provided as methods, systems, or computer program products. Therefore, those skilled in the art will realize that the functional modules / units or controllers and related method steps described in the above embodiments can be implemented in software, hardware, or a combination of both.

[0165] Unless explicitly stated otherwise, the actions or steps of the methods or procedures described in the embodiments of this disclosure do not necessarily have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0166] This document describes several embodiments of the present disclosure; however, for the sake of brevity, the descriptions of the embodiments are not exhaustive, and identical or similar features or portions between the embodiments may be omitted. In this document, "one embodiment," "some embodiments," "example," "specific example," or "some examples" refers to embodiments applicable to at least one, but not all, of the embodiments according to this disclosure. The above terms do not necessarily refer to the same embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples.

[0167] The exemplary systems and methods of this disclosure have been specifically shown and described with reference to the foregoing embodiments, and are merely examples of the best mode for implementing the systems and methods. Those skilled in the art will understand that various changes can be made to the embodiments of the systems and methods described herein without departing from the spirit and scope of this disclosure as defined in the appended claims when implementing the systems and / or methods. Industrial applicability

[0168] This disclosure provides a speech and singing synthesis method, training method, and related apparatus. The target model can be used as a speech synthesis model to fully utilize the singing data during training to improve the pitch of speech, or as a singing synthesis model to assist singing training with speech data to solve the problem of small singing data volume. It can also be used as a composite model of speech synthesis and singing synthesis to achieve the beneficial effect of being able to both speak and sing, while possessing the advantages mentioned above.

Claims

1. A training method characterized by, The training method comprises: obtaining voice training data, the voice training data comprising matched multiple text data and multiple voice data; obtaining singing voice training data, the singing voice training data comprising matched multiple lyrics data, multiple singing voice data and multiple score data; converting the text data into text embedding codes and converting the voice data into voice embedding codes; splicing the text embedding codes and the voice embedding codes into voice training code data; segmenting the lyrics data into sub-sentences and converting the sub-sentences into sub-sentence embedding codes, segmenting the singing voice data into singing voice segments corresponding to the sub-sentences and converting the singing voice segments into singing voice segment embedding codes, extracting pitch sequences corresponding to the sub-sentences and / or singing voice segments from the score data and converting the pitch sequences into pitch embedding codes; splicing the sub-sentence embedding codes, the singing voice segment embedding codes and the pitch embedding codes into singing voice training code data; inputting the voice training code data and the singing voice training code data into a voice and singing voice synthesis module of an initial model for training to obtain a target model comprising a trained voice and singing voice synthesis module.

2. The method of claim 1, wherein, extracting pitch sequences corresponding to the sub-sentences and / or singing voice segments from the score data and converting the pitch sequences into pitch embedding codes comprises: extracting pitch information from the score data; obtaining the lengths of pitches of the extracted pitch information; obtaining timestamps of the corresponding sub-sentences of the lyrics data; aligning the pitch embedding codes and the sub-sentence embedding codes according to the lengths of the pitches and the timestamps of the corresponding sub-sentences.

3. The training method of claim 1, wherein, extracting pitch sequences corresponding to the sub-sentences and / or singing voice segments from the score data and converting the pitch sequences into pitch embedding codes comprises: extracting pitch information from the score data; obtaining the lengths of pitches of the extracted pitch information; obtaining the lengths of pitches of the extracted pitch information; obtaining the lengths of pitches of the extracted pitch information; 4. Training method according to any of claims 1-3, characterized in that, aligning the pitch embedding codes and the sub-sentence embedding codes according to the lengths of the pitches and the timestamps of the corresponding sub-sentences.

5. The training method according to any one of claims 1 to 4, characterized in that, The score data comprises a MIDI file of a song. The training method comprises:

6. A method of speech and singing synthesis, characterized by, confirming whether the input training code data comprises pitch code information, and generating a hidden state sequence for controlling the range, rhythm and melody of a predicted audio waveform from the pitch code information when the pitch code information exists. The method comprises: obtaining to-be-synthesized data, the to-be-synthesized data comprising at least text data; judging whether the to-be-synthesized data comprises score data; if the to-be-synthesized data does not comprise score data, embedding coding the text data in the to-be-synthesized data as first code data, and inputting the voice synthesis code data into a voice and singing voice synthesis module of the trained voice and singing voice synthesis composite model to synthesize target voice; If the score data is included, the text data in the to-be-synthesized data is embedded and encoded to obtain second encoded data, and the second encoded data is input into a voice and singing synthesis module of the trained voice and singing synthesis composite model to synthesize the target singing.

7. The method of synthesizing speech and singing according to claim 6, wherein, The voice and singing synthesis composite model comprises a target model trained according to the training method in any one of claims 1 to 5.

8. The speech and singing synthesis method according to claim 6 or 7, characterized by, The method further comprises: If the score data is included, the score data in the to-be-synthesized data is embedded and encoded to obtain third encoded data, and the third encoded data is input into a voice and singing synthesis module of the trained voice and singing synthesis composite model, wherein the third encoded data is used to generate a hidden state sequence for controlling a vocal range, rhythm and melody of the target singing by the voice and singing synthesis module.

9. The method according to claim 6 or 7, characterized in that, The method further comprises: The target singing and the melody generated from the score data are mixed to obtain a to-be-played song.

10. A speech and singing synthesis composite model characterized by, The composite model comprises a target model trained according to the training method in any one of claims 1 to 5.

11. A speech synthesis model, characterized by, The voice synthesis model comprises a target model trained according to the training method in any one of claims 1 to 5.

12. A singing voice synthesis model, characterized by, The singing synthesis model comprises a target model trained according to the training method in any one of claims 1 to 5.

13. A training device, characterized by Comprise: A first obtaining module configured to obtain voice training data, the voice training data comprising a plurality of matched text data and a plurality of voice data; A second obtaining module configured to obtain singing training data, the singing training data comprising a plurality of matched lyric data, a plurality of singing data and a plurality of score data; A first encoding module configured to convert the text data into text embedded encoding and convert the voice data into voice embedded encoding; A first splicing module configured to splice the text embedded encoding and the voice embedded encoding into voice training encoded data; A second encoding module configured to divide the lyric data into sub-sentences and convert the sub-sentences into sub-sentence embedded encoding, divide the singing data into singing segments corresponding to the sub-sentences and convert the singing segments into singing segment embedded encoding, and extract a pitch sequence corresponding to the sub-sentences and / or singing segments from the score data and convert the pitch sequence into pitch embedded encoding; A second splicing module configured to splice the sub-sentence embedded encoding, the singing segment embedded encoding and the pitch embedded encoding into singing training encoded data; A model training module configured to input the voice training encoded data and the singing training encoded data into a voice and singing synthesis module of an initial model for training to obtain a target model comprising a trained voice and singing synthesis module.

14. A speech and singing synthesis apparatus characterized by comprising: Comprise: A third obtaining module configured to obtain to-be-synthesized data, the to-be-synthesized data comprising at least text data; A judging module configured to judge whether the to-be-synthesized data comprises score data; The synthesizing module is configured to, if the score data is not included, perform embedding coding on the text data in the data to be synthesized as first coded data, and input the speech and singing synthesizing coded data to a speech and singing synthesizing module of the trained speech and singing synthesizing composite model to synthesize target speech; The synthesizing module is further configured to, if the score data is included, perform embedding coding on the text data in the data to be synthesized to obtain second coded data, and input the second coded data to the speech and singing synthesizing module of the trained speech and singing synthesizing composite model to synthesize target singing.

15. An electronic device, comprising: The electronic device comprises a processor and a memory storing a computer program, the processor being configured to implement the method of any one of claims 1-5, or the method of any one of claims 6-9, when running the computer program.

16. A storage medium, characterized by The storage medium stores a computer program, the computer program being executed by a processor to implement the method of any one of claims 1-5, or the method of any one of claims 6-9.

Images