Speech and singing synthesis method, training method and device, model
By converting the voice and singing training data into embedded encoding and training models, combining the music score data to extract pitch information, and generating a target model that can both speak and sing, the problem of insufficient data and high difficulty in getting started in singing synthesis technology is solved, and the problem of insufficient data and high difficulty in getting started is achieved. It has an efficient effect of speaking and singing skills.
Patent Information
- Application Number
- CN202410672187.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-05-28
AI Technical Summary
The existing singing vocal synthesis technology faces the problems of insufficient singing data and single functions, and it is difficult to both speak and sing, and it is difficult to get started, which limits its wide application.
By obtaining speech and singing training data, converting it into embedded encoding, and inputting it into the initial model for training, it generates a target model that can both synthesize speech and sing, and extracting pitch information using the score data to improve the melody and range.
It achieves the advantageous effect of being able to speak and sing, solves the problems of small amount of singing data and high difficulty in getting started, and improves the high and low sound effects of voice synthesis.
Smart Images

Figure CN118411977B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of artificial intelligence audio processing, and in particular to a speech and singing synthesis method, a training method and related devices. Background Art
[0002] In the field of music production, several singing voice synthesis (SVS) software have been proposed, such as Synthesizer V, Vocaloid, and ACE Studio. These software allow users to manually adjust various attributes such as melody and timbre to produce high-quality singing effects by providing highly customized editing interfaces, such as piano roll view. Although these technologies are useful in singing effects, they can usually only be used for singing and are relatively difficult to use, which limits their widespread application.
[0003] At the same time, although SVS technologies based on neural network models provide ease of use and improved results, they face challenges such as insufficient singing data and single function.
[0004] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention
[0005] Therefore, the embodiments of the present application provide a speech and singing synthesis method, training method and device, and model, which can at least partially alleviate or solve the above-mentioned technical limitations.
[0006] In a first aspect, a training method is provided, the training method comprising:
[0007] Acquire speech training data, wherein the speech training data includes a plurality of matched text data and a plurality of speech data;
[0008] Acquire singing training data, wherein the singing training data includes a plurality of lyrics data, a plurality of singing data and a plurality of music score data that are matched;
[0009] Converting the text data into text embedding code, and converting the voice data into voice embedding code;
[0010] splicing the text embedding code and the speech embedding code into speech training code data;
[0011] The lyrics data is divided into sentences and the sentences are converted into sentence embedding codes, the singing data is divided into singing segments corresponding to the sentences and the singing segments are converted into singing segment embedding codes, and the pitch sequence corresponding to the sentences and / or singing segments is extracted from the music score data and the pitch sequence is converted into pitch embedding codes;
[0012] Splicing the sentence embedding code, the singing segment embedding code and the pitch embedding code into singing training code data;
[0013] The speech training coded data and the singing training coded data are input into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module.
[0014] In some embodiments of the present application, extracting a pitch sequence corresponding to the phrase and / or singing segment from the score data and converting the pitch sequence into a pitch embedding code includes:
[0015] Extracting pitch information from the music score data;
[0016] Obtain the length of each pitch of the extracted pitch information;
[0017] Obtaining the timestamp of the corresponding sentence of the lyrics data;
[0018] The pitch embedding code and the sentence embedding code are aligned according to the tone length of each pitch and the timestamp of the corresponding sentence.
[0019] In some embodiments of the present application, extracting a pitch sequence corresponding to the phrase and / or singing segment from the score data and converting the pitch sequence into a pitch embedding code includes:
[0020] Extracting pitch information from the music score data;
[0021] Obtain the length of each pitch of the extracted pitch information;
[0022] Obtaining the duration of the singing segment of the singing data;
[0023] According to the tone length of each pitch and the duration of the singing segment, the pitch embedding code and the singing segment embedding code are aligned.
[0024] In some embodiments of the present application, the music score data includes a MIDI file of the song.
[0025] In some embodiments of the present application, the inputting of the speech training coded data and the singing training coded data into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module comprises:
[0026] It is determined whether the input training encoding data includes pitch encoding information, and if the pitch encoding information exists, a hidden state sequence for controlling the range, rhythm and melody of the predicted audio waveform is generated from the pitch encoding information.
[0027] In a second aspect, an embodiment of the present application provides a method for synthesizing speech and singing, comprising:
[0028] Acquire data to be synthesized, wherein the data to be synthesized at least includes text data;
[0029] Determining whether the data to be synthesized includes music score data;
[0030] If the music score data is not included, embedding and encoding the text data in the data to be synthesized as the first encoded data, and inputting the speech synthesis encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target speech;
[0031] If the music score data is included, the text data in the data to be synthesized is embedded and encoded to obtain second encoded data, and the second encoded data is input into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target singing.
[0032] In some embodiments of the present application, the composite model of speech and singing synthesis includes a target model trained according to the training method described in any one of the embodiments of the present application.
[0033] In some embodiments of the present application, the method further includes:
[0034] If the music score data is included, the music score data in the data to be synthesized is embedded and encoded to obtain third encoded data, and the third encoded data is input into the speech and singing synthesis module of the trained speech and singing synthesis composite model, wherein the third encoded data is generated by the speech and singing synthesis module to generate a hidden state sequence for controlling the range, rhythm and melody of the target song.
[0035] In some embodiments of the present application, the method further includes:
[0036] The target singing voice and the melody generated by the music score data are mixed to obtain the song to be played.
[0037] In a third aspect, a composite model for speech and singing synthesis is provided, wherein the composite model includes a target model trained according to the method described in any embodiment of the present application.
[0038] In a fourth aspect, a speech synthesis model is provided, wherein the speech synthesis model includes a target model trained according to the method described in any one of the embodiments of the present application.
[0039] In a fifth aspect, a singing synthesis model is provided, the singing synthesis model comprising a target model trained according to the method described in any one of the embodiments of the present application.
[0040] In a sixth aspect, a training device is provided, comprising:
[0041] A first acquisition module, used to acquire voice training data, wherein the voice training data includes a plurality of matched text data and a plurality of voice data;
[0042] A second acquisition module is used to acquire singing training data, wherein the singing training data includes a plurality of lyrics data, a plurality of singing data and a plurality of music score data that are matched;
[0043] A first encoding module, used to convert the text data into a text embedding code and convert the voice data into a voice embedding code;
[0044] A first splicing module, used for splicing the text embedding code and the speech embedding code into speech training code data;
[0045] A second encoding module is used to divide the lyrics data into sentences and convert the sentences into sentence embedded codes, divide the singing data into singing segments corresponding to the sentences and convert the singing segments into singing segment embedded codes, extract the pitch sequence corresponding to the sentences and / or singing segments from the music score data and convert the pitch sequence into pitch embedded codes;
[0046] A second splicing module is used to splice the sentence embedding code, the singing segment embedding code and the pitch embedding code into singing training code data;
[0047] The model training module is used to input the speech training coded data and the singing training coded data into the speech and singing synthesis module of the initial model for training, so as to obtain a target model including the trained speech and singing synthesis module.
[0048] In a seventh aspect, an electronic device is provided, comprising a processor and a memory storing a computer program, wherein the processor is configured to implement the method described in any embodiment of the present application when running the computer program.
[0049] In an eighth aspect, a storage medium is provided, wherein the storage medium stores a computer program, and the computer program is configured to implement the method described in any embodiment of the present application when executed.
[0050] The training method provided in the embodiment of the present application is obtained by acquiring speech training data; acquiring singing training data; converting the text data of the speech training data into text embedding code, converting the speech data into speech embedding code, and splicing the text embedding code and the speech embedding code into speech training code data; dividing the lyrics data of the singing training data into sentences and converting the sentences into sentence embedding codes, dividing the singing data into singing segments corresponding to the sentences and converting the singing segments into singing segment embedding codes, extracting the pitch sequence of the corresponding sentences and / or singing segments from the music score data and converting the pitch sequence into pitch embedding code, and splicing the sentence embedding code, the singing segment embedding code and the pitch embedding code into singing training code data; inputting the speech training code data and the singing training code data into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module. On the one hand, the target model thus obtained can be used as a speech synthesis model. When used as a speech synthesis model, it can make full use of the singing data during training to improve the high and low pitch effects of the speech. It can also be used as a singing synthesis model. When used as a singing synthesis model for training, the singing training is assisted by speech data to solve the problem of small singing data volume. On the other hand, the target model thus obtained can also be used as a speech synthesis and singing synthesis model. With the above advantages, the model can achieve the beneficial effects of both speaking and singing, and the effects are both excellent.
[0051] Accordingly, according to the speech and singing synthesis method provided in the embodiment of the present application, it can determine whether the data to be synthesized contains music score data; and perform selective different outputs based on whether the music score data is included, thereby having the advantageous effects of both speaking voice and singing, and the effects are both excellent.
[0052] Other optional features and technical effects of the embodiments of the present application are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use are briefly introduced below. It should be clear that the drawings in the following description only cover some embodiments of the present application, and ordinary technicians in this field can obtain other drawings based on these drawings without performing creative work. The purpose of these drawings is to better illustrate the technical details to help understand the implementation methods of the present application, among which:
[0054] Figure 1 A schematic flow chart showing a training method according to an embodiment of the present application;
[0055] Figure 2 A schematic process diagram showing the training method of an embodiment of the present application during training;
[0056] Figure 3 A schematic flow chart showing a training method according to an embodiment of the present application;
[0057] Figure 4 A schematic flow chart showing a training method according to an embodiment of the present application;
[0058] Figure 5 An exemplary model structure diagram showing a training method and a synthesis method applicable to an embodiment of the present application;
[0059] Figure 6 A schematic flow chart showing a method for synthesizing speech and singing voice according to an embodiment of the present application;
[0060] Figure 7 An exemplary structural diagram showing a training device according to an embodiment of the present application; and
[0061] Figure 8 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0062] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0063] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0064] In the embodiment of the present application, a "phoneme" is the smallest unit of speech divided according to the natural properties of speech, and one pronunciation action forms a phoneme. A "phoneme sequence" is a sequence composed of a number of phonemes.
[0065] Currently, singing voice synthesis (SVS) technology based on neural network models has made great progress. However, they face challenges such as insufficient singing data available for training.
[0066] In addition, the current singing voice synthesis (SVS) model can basically only sing, so in order to enable AI to have both speaking and singing capabilities, an additional speech synthesis (TTS) model is required. Switching between the two models is difficult to operate and the cost is high. Moreover, there are many differences between speech and singing, such as singing has a wider range, higher sound quality requirements, and also requirements for rhythm and melody. It is not feasible to directly use the speech synthesis (TTS) model for singing voice synthesis (SVS).
[0067] To this end, the present application embodiment intends to provide a training method, by which the model trained can have the composite capability of speech synthesis and singing synthesis. Moreover, if the model trained by the training method is dedicated to singing synthesis (SVS), the problem of insufficient singing data available for training is also solved. The inventors have also additionally found that if the model trained by the training method is dedicated to speech synthesis (TTS), it has a surprising effect of improving high and low notes.
[0068] like Figure 1 As shown, the embodiment of the present application provides a training method. The training method may include the following steps:
[0069] S110: Acquire speech training data, where the speech training data includes a plurality of matching text data and a plurality of speech data.
[0070] In some embodiments of the present application, the speech training data may be conventional speech data that can be used to train a TTS model. In some exemplary embodiments, the speech training data may include text-speech pairs, but it is conceivable that the speech training data is provided in a more original form and then preprocessed to form text-speech pairs, which is not limited by the present application.
[0071] S120: Acquire singing training data, where the singing training data includes a plurality of matching lyrics data, a plurality of singing data, and a plurality of music score data.
[0072] In some embodiments of the present application, the matched lyrics data and singing data may include the lyrics of the entire song and the corresponding singing, and the singing is preferably with melody, but may also be without melody, which is not limited by the present application. In an embodiment of the present application, the singing data does not include music with pure melody. In another embodiment, the matched lyrics data and singing data may be the lyrics phrases and corresponding singing data that have been segmented according to the lyrics phrases.
[0073] In some embodiments of the present application, the lyrics data can select any suitable lyrics file. In one example, the lyrics data can include an LRC (Lyric Resource Computer) file, but the present application is not limited to this. The LRC file is synchronized with the music by adding a time tag next to the lyrics text. The structure of the LRC file includes a time tag and a lyrics text. The time tag of the LRC file is surrounded by square brackets (for example: [00:01.00]), indicating which specific time point the lyrics should be displayed, and these time points correspond to the playback time of the music file. The time tag is followed by the lyrics text that should be displayed at the corresponding time point.
[0074] In some embodiments of the present application, the music score data includes a MIDI file of a song, but the present application is not limited to this. For example, the music score data may include MusicXML.
[0075] MIDI protocol, or Musical Instrument Digital Interface, is an instant communication protocol between electronic musical instruments, synthesizers and other performance devices, used for real-time performance data transmission between hardware. In the embodiment of the present application, the MIDI file refers to a file in a file format that conforms to the MIDI protocol. A MIDI file may contain a number of MIDI messages, which are used to transmit real-time music performance information, presented as instructions containing information such as pitch, tempo, and effector parameters. By way of example and not limitation, MIDI messages include the triggering (NOTE ON) and termination (NOTE OFF) of notes.
[0076] S130: Convert the text data into text embedded code, and convert the voice data into voice embedded code.
[0077] In the embodiments of the present application, Figure 2It is schematically shown that in order to generate text embedding, the text segment can be segmented. As an example, a string matching-based method or a rule-based method can be used, such as a forward maximum matching segmentation algorithm, a reverse maximum matching segmentation algorithm, and a statistical-based method. Before segmentation, some embodiments also include text normalization of the text segment, which involves converting character sets (such as uniformly using Unicode encoding), eliminating case differences, and removing irrelevant characters (such as HTML tags). For the embedding process of the text, it may include building a vocabulary, in which word frequency statistics, vocabulary screening are performed to remove high-frequency stop words and low-frequency words, and each token in the vocabulary is assigned a unique numerical index. Token encoding may use One-Hot encoding to convert each token into a high-dimensional sparse vector, and limit the vector size when necessary, and use special tokens instead of uncommon words. The process of learning dense vector representation may involve selecting pre-trained models such as Word2Vec, GloVe, FastText, or BERT or training a custom model according to a specific task, and learning the vector representation of the token through context. These vectors can capture complex relationships between words, such as semantic similarity and grammatical rules. Dimension reduction can also be performed later, such as PCA or t-SNE, to optimize the vector space so that similar words are closer in the vector space. However, the text embedding of the embodiment of the present application is not limited to this.
[0078] In the embodiments of the present application, Figure 2 As schematically shown, in order to generate speech embedding, useful information or features, such as frequency components, energy, rhythm, etc., can be extracted from the original audio signal. These features capture the important properties of the audio, but are usually still continuous values. This information and features are then discretized, which involves mapping the feature vector to a high-dimensional space. The embedding process can be implemented by selecting any suitable neural network model, such as a deep learning model. However, the speech embedding of the embodiment of the present application is not limited to this.
[0079] S140: Concatenate the text embedding code and the speech embedding code into speech training code data.
[0080] S150: Divide the lyrics data into sentences and convert the sentences into sentence embedded codes, divide the singing data into singing segments corresponding to the sentences and convert the singing segments into singing segment embedded codes, extract the pitch sequence corresponding to the sentences and / or singing segments from the music score data and convert the pitch sequence into pitch embedded codes.
[0081] In the embodiment of the present application, the lyrics can be divided into multiple sentences according to the characteristics of the lyrics, for example, according to punctuation marks. After the sentences are obtained, the sentences can be embedded using the text embedding method described above.
[0082] In the embodiment of the present application, the corresponding singing segment can be obtained based on the timestamp (time tag) of the lyrics file, such as the LRC file. After obtaining the singing segment, the singing segment can be embedded in the manner of speech embedding described above.
[0083] In the embodiments of the present application, Figure 2 The dotted line portion in the figure schematically shows that, compared with the embeddings of the speech training data, for the singing training data, the pitch information of the music score files, such as MIDI files, can also be extracted.
[0084] Taking the MIDI messages in a MIDI file as an example, the pitch value of the note can be recorded in the note trigger (NOTEON) event of each note. Accordingly, MIDI information can be extracted from the MIDI messages of each note in the MIDI file, thereby obtaining the pitch sequence corresponding to the song, and after reasonable processing, the pitch sequence corresponding to each phrase or singing segment can be obtained. The pitch sequence includes the pitch information of each note, which describes the high and low trend of the melody. And, the extracted pitch information (pitch sequence) can be converted into a pitch embedding code by a suitable embedding method. As an explanation, the pitch sequence described in this article or the pitch embedding code converted from the pitch sequence is not a vector dimension that may represent the pitch in the audio embedding. In an embodiment of the present application, any suitable embedding method can be used to embed the pitch converted from the pitch sequence. As an example, for example, the pitch embedding code can be obtained by reasonable modification using the above or other text embedding methods.
[0085] In a preferred embodiment of the present application, Figure 5 Schematically, the pitch duration information can be aligned with the corresponding singing segment and / or lyrics phrase.
[0086] In some embodiments, the pitch duration information may be aligned with the lyrics phrases.
[0087] like Figure 3 As shown, the step S150 may include:
[0088] S151: Extracting pitch information from the music score data.
[0089] S152: Acquire the length of each pitch of the extracted pitch information.
[0090] S153: Obtain the timestamp of the corresponding sentence of the lyrics data.
[0091] S154: Align the pitch embedding code and the sentence embedding code according to the tone length of each pitch and the timestamp of the corresponding sentence.
[0092] Specifically, the music information stored in the digital form of the MIDI file includes not only the pitch (Pitch) but also the duration of the pitch. Accordingly, in step S151, the pitch information is extracted from the MIDI file (MIDI message), such as which pitch the corresponding note is at; and in step S152, the duration information of the extracted pitch is obtained, more precisely, the duration information of the note. In some embodiments, the duration of the respective pitches corresponding to the notes can be determined by the triggering (NOTE ON) and termination (NOTE OFF) of the notes mentioned above. Accordingly, the pitch and duration information in the MIDI are extracted to obtain the melody of the song and the pronunciation length of each note.
[0093] As a further explanation and not limitation, MIDI file time information, the time in MIDI is usually expressed in "ticks", these ticks combined with the BPM (beats per minute) of the song can be converted into actual time length. This can be understood as the time divided by notes, which can be specifically called TPQN (Ticks Per Quarter-Note), that is, how many ticks are contained in a quarter note. Based on this, the tick value in the MIDI file can be converted into actual time. For example, converting ticks to seconds or milliseconds can depend on the tempo (BPM) set in the MIDI file.
[0094] In step S153, the timestamp of the corresponding sentence may be obtained accordingly. For example, in the LRC file, the timestamp information of the corresponding sentence may be obtained by obtaining the time tag.
[0095] In this way, the pitch duration information and the corresponding lyrics can be aligned according to the duration of each line of lyrics. Therefore, compared with conventional text-audio pairs, each text-audio pair obtained from the singing training data is attached with corresponding pitch information and pronunciation length information.
[0096] In a further preferred embodiment, alignment of lyrics phrases and singing segments can also be provided.
[0097] In an alternative embodiment, the pitch duration information may be aligned with the singing segment.
[0098] like Figure 4 As shown, the step S150 may include:
[0099] S151': extracting pitch information from the music score data.
[0100] S152': Acquire the length of each pitch of the extracted pitch information.
[0101] S153': Obtain the duration of the singing segment of the singing data.
[0102] S154': aligning the pitch embedding code and the singing segment embedding code according to the tone length of each pitch and the duration of the singing segment.
[0103] Steps S151 ′ and S152 ′ may refer to steps S151 and S152 .
[0104] After extracting the pitch and duration information in MIDI using step S152' and obtaining the melody of the song and the pronunciation length of each note, the pitch sequence can be aligned with the singing segment by fine-tuning the singing segment time axis. An optional alignment method is dynamic time warping (DTW).
[0105] In a further preferred embodiment, alignment of lyrics phrases and singing segments can also be provided.
[0106] exist Figure 3 or Figure 4 In the illustrated embodiment, alignment followed by embedding processing or direct alignment of pitch embeddings and lyrics phrases and / or vocal segment embeddings is covered.
[0107] Thus, by means of the alignment, the hidden state of the pitch transition information of the song melody can be learned during training. Accordingly, in the training step of S160 below, a hidden state sequence for controlling the range, rhythm and melody of the predicted audio waveform can be generated from the pitch encoding information.
[0108] S160: splicing the sentence embedding code, the singing segment embedding code and the pitch embedding code into singing training coding data.
[0109] In the embodiments of the present application, the order of the spliced singing training coding data can be in any suitable form, and the present application does not impose any limitation on this.
[0110] S170: Input the speech training coded data and the singing training coded data into the speech and singing synthesis module of the initial model for training, so as to obtain a target model including the trained speech and singing synthesis module.
[0111] In the embodiment of the present application, the speech and singing synthesis module can be any suitable TTS model that has been adapted. In an exemplary embodiment, the speech and singing synthesis module can be based on the FastSpeech model architecture, but the present application is not limited thereto, and any suitable TTS or SVS basic model or a modified model that has been appropriately adapted can be used.
[0112] The FastSpeech architecture is a new feed-forward network that improves performance in many aspects compared to traditional TTS models. FastSpeech can quickly generate stable, controllable, and high-quality mel-spectrograms.
[0113] In some embodiments of the present application, Figure 5 As shown, the FastSpeech mainly includes the following parts:
[0114] a. Feed-Forward Transformer Block: It includes a self-attention module and a 1D convolution module, which can convert the input phoneme sequence into a Mel-spectrogram sequence to prepare for the subsequent speech synthesis step.
[0115] b. Length Regulator: Since the length of the phoneme sequence is smaller than the length of the Mel-spectrogram sequence, one phoneme corresponds to multiple Mel-spectrograms. The number of Mel-spectrograms aligned with a phoneme is called the phoneme duration. The length regulator expands the hidden sequence of the phoneme according to the duration to match the length of the Mel-spectrogram sequence. The duration of the phoneme can be increased or decreased proportionally to adjust the speed of the sound.
[0116] c. Linear Layer: The linear layer is a basic neural network layer used to implement linear transformation. In deep learning and neural networks, it is often called a fully connected layer or a dense layer, which can be used to convert the intermediate representation of the network into the final output format, i.e., a mel-spectrogram.
[0117] like Figure 5As shown, compared to the conventional FastSpeech architecture, in the embodiment of the present application, the FastSpeech architecture is transformed into a variable coding form suitable for input, for example, the splicing data of text embedding+audio embedding can be optionally accepted, or the splicing data of text embedding+audio embedding+pitch embedding can be optionally accepted. In some embodiments, the position encoding module can be set in any suitable manner to enable it to receive variable length coding. In other embodiments, the splicing (steps S140 and S160) can be set so that when splicing speech training coding data, an empty coding vector segment corresponding to the pitch embedding coding length is reserved. However, the present application is not limited thereto.
[0118] During the training process, the predicted Mel-spectrogram will be compared with the speech or singing label to iteratively update the Fastspeech-based speech and singing synthesis module.
[0119] During the inference process, the predicted Mel-spectrogram will be input into a waveform generation neural network model, such as a dedicated vocoder (WaveNet, WaveGlow, and WaveRNN, etc.), to convert the sound feature Mel-spectrogram into the final target audio waveform, which is a target speech waveform or a target singing waveform depending on whether the input data contains MIDI or pitch information.
[0120] As mentioned above, step S160 may optionally include: confirming whether the input training coding data includes pitch coding information, and when the pitch coding information exists, generating a hidden state sequence for controlling the range, rhythm and melody of the predicted audio waveform from the pitch coding information. The speech and singing synthesis modules of the initial model can be adapted as needed to introduce hidden state sequences to control the input text / speech / singing embedding coding (or other coding sequences after their conversion).
[0121] Accordingly, the embodiment of the present application can also provide a method for selectively synthesizing speech and singing.
[0122] like Figure 6 As shown, the embodiment of the present application provides a method for synthesizing speech and singing, comprising the following steps:
[0123] S610: Acquire data to be synthesized, where the data to be synthesized at least includes text data;
[0124] S620: Determine whether the data to be synthesized includes music score data;
[0125] If the score data is not included, then S630 and S640 are executed:
[0126] S630: embedding and encoding the text data in the data to be synthesized as first encoded data;
[0127] S640: Input the speech synthesis coding data into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target speech.
[0128] If the music score data is included, execute S650 and S660:
[0129] S650: performing embedding coding on the text data in the data to be synthesized to obtain second coded data;
[0130] S660: Input the second encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model.
[0131] Among them, the composite model of speech and singing synthesis preferably includes a target model trained according to the training method described in any one of the embodiments of the present application.
[0132] In an optional embodiment of the present application, when inferring, there is no need to extract pitch information from the score data, and the score file can be used to determine whether it is singing synthesis or speech synthesis.
[0133] However, in a preferred embodiment, pitch information may also be extracted from the score data to assist in singing synthesis during inference.
[0134] In some embodiments of the present application, if music score data is included, S670 and S680 are optionally performed:
[0135] S670: performing embedding coding on the music score data in the data to be synthesized to obtain third coded data;
[0136] S680: Input the third encoded data into the speech and singing synthesis module of the trained speech and singing synthesis composite model.
[0137] Among them, the third encoded data is generated by the speech and singing synthesis module to generate a hidden state sequence for controlling the range, rhythm and melody of the target singing.
[0138] In some embodiments of the present application, the method further comprises: mixing the target singing voice and the melody generated by the music score data to obtain a song to be played. In this embodiment, the music score data can be processed by any suitable audio decoder to generate a melody, and mixed with the generated singing voice to obtain a song to be played with a melody.
[0139] Furthermore, the target model trained by the training method described in the embodiments of the present application can also be used for various purposes.
[0140] In some embodiments, a composite model for speech and singing synthesis is provided, which is a target model obtained by training according to the training method described in any one of the embodiments of the present application. Optionally, the composite model for speech and singing synthesis may include trained speech and singing synthesis modules. The composite model for speech and singing synthesis may optionally include a coding module for encoding speech data and / or singing data (including music score files) into corresponding coding sequences, which may, for example, adopt a similar coding module used in the training method. The speech synthesis and singing synthesis conform to the model, which, while having the advantages of improving the high and low pitch effects of the above-mentioned speech and solving the problem of small amount of singing training data, enables the model to obtain the advantageous effect of being able to speak speech and sing, and the effect is excellent.
[0141] In other embodiments, a speech synthesis model is provided, which may include a target model trained according to any one of the training methods described in the embodiments of the present application. The trained speech and singing synthesis modules of the speech synthesis model may be used only for synthesizing speech. Optionally, the speech synthesis model may have a conventional speech synthesis coding module, i.e., a coding module that does not have the ability to encode pitch sequences. The speech synthesis model can make full use of the singing data during training to improve the high and low pitch effects of speech.
[0142] In other embodiments, a singing synthesis model is provided, which may include a target model trained according to the training method described in any one of the embodiments of the present application. The trained speech and singing synthesis modules of the singing synthesis model can be used only for synthesizing singing. Optionally, the singing synthesis model may have a conventional singing synthesis encoding module, or may have an encoding module similar to that for encoding singing data (including music score files) into corresponding encoding sequences. Optionally, the singing synthesis model may also include an additional filter, which may filter out input data that does not conform to the input format, such as input data without music score information. When the singing synthesis model is trained, the singing training is assisted by speech data to solve the problem of small amount of singing training data.
[0143] like Figure 7 As shown, an embodiment of the present application provides a training device 700, which may include: a first acquisition module 710, a second acquisition module 720, a first encoding module 730, a first splicing module 740, a second encoding module 750, a second splicing module 760 and a model training module 770.
[0144] In an embodiment of the present application, the first acquisition module 710 is configured to acquire speech training data, and the speech training data includes a plurality of matching text data and a plurality of voice data. The second acquisition module 720 is configured to acquire singing training data, and the singing training data includes a plurality of matching lyrics data, a plurality of singing data, and a plurality of music score data. The first encoding module 730 is configured to convert the text data into text embedding code, and convert the voice data into voice embedding code. The first splicing module 740 is configured to splice the text embedding code and the voice embedding code into speech training code data. The second encoding module 750 is configured to divide the lyrics data into sentences and convert the sentences into sentence embedding codes, divide the singing data into singing segments corresponding to the sentences and convert the singing segments into singing segment embedding codes, extract the pitch sequence corresponding to the sentences and / or singing segments from the music score data and convert the pitch sequence into pitch embedding codes. The second splicing module 760 is configured to splice the sentence embedding code, the singing segment embedding code and the pitch embedding code into singing training code data. The model training module 770 is configured to input the speech training code data and the singing training code data into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module.
[0145] The device according to the embodiment of the present application and its components, configurations and features can be combined into the method according to the embodiment of the present application in a non-contradictory manner. Conversely, the method according to the embodiment of the present application and its steps, sub-steps and features can be combined into the device according to the embodiment of the present application in a non-contradictory manner.
[0146] In an embodiment of the present application, an electronic device is also provided, including: a processor and a memory storing a computer program, wherein the processor is configured to implement a related method of any embodiment when running the computer program.
[0147] Figure 8 A schematic diagram of an electronic device 800 of an embodiment of the present application or an electronic device 800 that can be used to implement a method of an embodiment of the present application is shown. In some embodiments, the number of electronic devices may be more or less than the number shown. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, cloud or distributed electronic devices may also be used for implementation.
[0148] like Figure 8As shown, the electronic device 800 includes a processor 810 and a memory 820. The processor is used to execute programs stored in the memory, which can implement the methods, steps or functions described in the above embodiments when executed by a computer. The processor 810 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. The processor 810 and the memory 820 are interconnected via a bus 830. An input / output (I / O) interface, etc. may also be connected to the bus 830.
[0149] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.
[0150] Although not shown, in an embodiment of the present application, a storage medium is provided, the storage medium storing a computer program, the computer program being configured to implement a method of any embodiment when executed.
[0151] The storage media in the embodiments of the present application include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0152] The methods, programs, systems, devices, etc. of the embodiments of the present application may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of the present specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.
[0153] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.
[0154] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present application do not have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0155] In this article, multiple embodiments of the present application are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are meant to be applicable to at least one embodiment or example according to the present application, rather than all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. Under non-contradictory emotions, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.
[0156] The exemplary systems and methods of the present application have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present application as defined in the appended claims.
Claims
1. A training method, characterized in that: The training method comprises: Acquire speech training data, wherein the speech training data includes a plurality of matched text data and a plurality of speech data; Acquire singing training data, wherein the singing training data includes a plurality of lyrics data, a plurality of singing data and a plurality of music score data that are matched; Converting the text data into text embedding code, and converting the voice data into voice embedding code; splicing the text embedding code and the speech embedding code into speech training code data; The lyrics data is divided into sentences and the sentences are converted into sentence embedding codes, the singing data is divided into singing segments corresponding to the sentences and the singing segments are converted into singing segment embedding codes, and the pitch sequence corresponding to the sentences and / or singing segments is extracted from the music score data and the pitch sequence is converted into pitch embedding codes; Splicing the sentence embedding code, the singing segment embedding code and the pitch embedding code into singing training code data; The speech training coded data and the singing training coded data are input into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module.
2. The method according to claim 1, characterized in that Extracting a pitch sequence corresponding to the phrase and / or singing segment from the music score data and converting the pitch sequence into a pitch embedding code comprises: Extracting pitch information from the music score data; Obtain the length of each pitch of the extracted pitch information; Obtain the timestamp of the corresponding sentence of the lyrics data; The pitch embedding code and the sentence embedding code are aligned according to the tone length of each pitch and the timestamp of the corresponding sentence.
3. The training method according to claim 1, characterized in that: Extracting a pitch sequence corresponding to the phrase and / or singing segment from the music score data and converting the pitch sequence into a pitch embedding code comprises: Extracting pitch information from the music score data; Obtain the length of each pitch of the extracted pitch information; Obtaining the duration of the singing segment of the singing data; According to the tone length of each pitch and the duration of the singing segment, the pitch embedding code and the singing segment embedding code are aligned.
4. The training method according to claim 1, characterized in that: The music score data includes a MIDI file of the song.
5. The training method according to any one of claims 1 to 4, characterized in that: The method of inputting the speech training coded data and the singing training coded data into the speech and singing synthesis module of the initial model for training to obtain a target model including the trained speech and singing synthesis module comprises: It is determined whether the input training encoding data includes pitch encoding information, and if the pitch encoding information exists, a hidden state sequence for controlling the range, rhythm and melody of the predicted audio waveform is generated from the pitch encoding information.
6. A method for synthesizing speech and singing, characterized in that: include: Acquire data to be synthesized, wherein the data to be synthesized at least includes text data; Determining whether the data to be synthesized includes music score data; If the music score data is not included, embedding and encoding the text data in the data to be synthesized as first encoded data, and inputting the first encoded data into a speech and singing synthesis module of a trained speech and singing synthesis composite model to synthesize the target speech; If the music score data is included, the text data in the data to be synthesized is embedded and encoded to obtain second encoded data, and the second encoded data is input into the speech and singing synthesis module of the trained speech and singing synthesis composite model to synthesize the target singing.
7. The method for synthesizing speech and singing according to claim 6, characterized in that: The speech and singing synthesis composite model includes a target model trained according to the training method described in any one of claims 1 to 5.
8. The method for synthesizing speech and singing according to claim 6 or 7, characterized in that: The method further comprises: If the music score data is included, the music score data in the data to be synthesized is embedded and encoded to obtain third encoded data, and the third encoded data is input into the speech and singing synthesis module of the trained speech and singing synthesis composite model, wherein the third encoded data is generated by the speech and singing synthesis module to generate a hidden state sequence for controlling the range, rhythm and melody of the target song.
9. The method according to claim 6 or 7, characterized in that: The method further comprises: The target singing voice and the melody generated by the music score data are mixed to obtain the song to be played.
10. A composite model for speech and singing synthesis, characterized in that: The composite model includes a target model trained according to the training method according to any one of claims 1 to 5.
11. A speech synthesis model, characterized in that: The speech synthesis model includes a target model trained according to the training method described in any one of claims 1 to 5.
12. A singing synthesis model, characterized in that: The singing synthesis model includes a target model trained according to the training method described in any one of claims 1 to 5.
13. A training device, characterized in that: include: A first acquisition module, used to acquire voice training data, wherein the voice training data includes a plurality of matched text data and a plurality of voice data; A second acquisition module is used to acquire singing training data, wherein the singing training data includes a plurality of lyrics data, a plurality of singing data and a plurality of music score data that are matched; A first encoding module, used to convert the text data into a text embedding code and convert the voice data into a voice embedding code; A first splicing module, used for splicing the text embedding code and the speech embedding code into speech training code data; A second encoding module is used to divide the lyrics data into sentences and convert the sentences into sentence embedded codes, divide the singing data into singing segments corresponding to the sentences and convert the singing segments into singing segment embedded codes, extract the pitch sequence corresponding to the sentences and / or singing segments from the music score data and convert the pitch sequence into pitch embedded codes; A second splicing module is used to splice the sentence embedding code, the singing segment embedding code and the pitch embedding code into singing training code data; The model training module is used to input the speech training coded data and the singing training coded data into the speech and singing synthesis module of the initial model for training, so as to obtain a target model including the trained speech and singing synthesis module.
