Singing sound synthesis model training method, singing sound synthesis method, singing sound synthesis device and singing sound synthesis equipment
By extracting the semantic features of lyrics and musical score features and updating the vocal synthesis model, the problem of poor vocal synthesis effect in the existing technology is solved, and a more realistic vocal synthesis effect is achieved.
Patent Information
- Application Number
- CN202411294231.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-29
AI Technical Summary
The prior art uses only the pitch information in the music score when training the singing vocal synthesis model, resulting in poor singing vocal synthesis effect.
By obtaining the lyrics text and music scores of the sample songs, extracting the semantic features and music score features of the lyrics, extracting the expression characteristics of the songs based on attention characteristics, and updating the vocal synthesis model with the music score features to generate a more realistic synthetic vocal vocal.
It improves the effect of singing vocal synthesis, so that the emotions of the lyrics are integrated into the singing, and improves the naturalness and fluency of singing vocal synthesis.
Smart Images

Figure CN119049434A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a singing synthesis model training method, a singing synthesis method, a device and equipment. Background Art
[0002] The existing technical solution only uses the pitch information in the music score when training the singing synthesis model, resulting in poor singing synthesis effect.
[0003] Therefore, how to improve the singing synthesis effect is a technical problem that needs to be solved by those skilled in the art. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a singing synthesis model training method, a singing synthesis method, a device and an equipment to solve the technical problem of poor format synthesis effect in the prior art.
[0005] To solve the above technical problems, the present invention provides a singing synthesis model training method, comprising:
[0006] Acquire a sample song, wherein the sample song includes lyrics and music scores;
[0007] Extracting semantic features of lyrics of the lyrics text through the singing voice synthesis model to be trained, and extracting music score features of the music score through the singing voice synthesis model to be trained;
[0008] By using the singing synthesis model to be trained, attention features are extracted based on the semantic features of the lyrics and the features of the music score to obtain song expressiveness features;
[0009] Obtaining a singing voice coding feature according to the song expressiveness feature and the music score feature;
[0010] Decoding the singing voice encoding features by the singing voice synthesis model to be trained to obtain the synthesized singing voice features for the sample song;
[0011] The singing synthesis model to be trained is updated based on the synthesized singing features and the original singing features of the sample song to obtain a target singing synthesis model. The target singing synthesis model is used to output target synthesized singing features according to the input target lyrics text and target music score. The target synthesized singing features are used to generate corresponding synthesized singing audio.
[0012] Optionally, extracting lyrics semantic features of the lyrics text through the singing synthesis model to be trained includes:
[0013] Determining the initial lyrics semantic features of the lyrics text by the singing synthesis model to be trained;
[0014] Determine whether there is a semantic feature belonging to the same word in the initial lyrics semantic features;
[0015] When it exists, repeated processing is performed based on the semantic features belonging to the same word to obtain
[0016] Repeating semantic features;
[0017] The repeated semantic features are encoded to obtain the lyrics semantic features.
[0018] Optionally, extracting the music score features of the music score through the singing synthesis model to be trained includes:
[0019] Performing symbol recognition on the music score to determine note features;
[0020] Determining a phoneme, a pitch, and a phoneme duration based on the note features;
[0021] The phonemes and the pitches are subjected to text encoding processing by the singing synthesis model to be trained to obtain text encoding data, and the text encoding data and the phoneme duration are subjected to phoneme expansion to obtain the score features.
[0022] Optionally, the song synthesis model to be trained is used to extract attention features based on the semantic features of the lyrics and the features of the music score to obtain song expressiveness features, including:
[0023] The dimension of the lyrics semantic features is transformed using a linear layer or a convolutional layer to obtain
[0024] The semantic features of lyrics in preset dimensions;
[0025] The semantic features of the lyrics of the preset dimension and the features of the music score are correlated based on the self-attention mechanism to obtain the expressive features of the song.
[0026] Optionally, updating the singing voice synthesis model to be trained based on the synthesized singing voice features and the original singing voice features of the sample song to obtain a target singing voice synthesis model includes:
[0027] Determining a Mel-spectrogram loss function based on the synthesized singing voice features and the original singing voice features of the sample song;
[0028] The singing voice synthesis model to be trained is trained by using the Mel spectrum loss function until the Mel spectrum loss function converges, thereby obtaining the target singing voice synthesis model.
[0029] Optionally, the singing voice synthesis model to be trained is trained by using the Mel spectrum loss function until the Mel spectrum loss function converges to obtain the target singing voice synthesis model, including:
[0030] Determine the predicted fundamental frequency of the synthesized singing voice feature using a fundamental frequency predictor, and determine a fundamental frequency loss function based on the predicted fundamental frequency and the original fundamental frequency of the sample song;
[0031] After updating the singing voice synthesis model to be trained based on the fundamental frequency loss function and the Mel-spectrogram loss function, the training is continued until the training is completed to obtain the target singing voice synthesis model.
[0032] The present application also provides a singing synthesis method, comprising:
[0033] Obtain target lyrics text and target music score;
[0034] The target song synthesis model is used to process the target lyrics text and the target music score to obtain target synthesized song features; wherein the target song synthesis model is the above-mentioned target song synthesis model;
[0035] The target synthesized singing voice features are processed by using a vocoder to obtain the target synthesized singing voice.
[0036] The present application also provides a singing synthesis model training device, comprising:
[0037] A sample song acquisition module is used to acquire sample songs, wherein the sample songs include lyrics and music scores;
[0038] A feature extraction module, used to extract the semantic features of the lyrics of the lyrics text through the singing synthesis model to be trained, and to extract the music score features of the music score through the singing synthesis model to be trained;
[0039] An attention feature extraction module, used to extract attention features based on the lyrics semantic features and the music score features through the singing synthesis model to be trained, so as to obtain song expressiveness features;
[0040] A singing coding feature determination module, used to obtain singing coding features according to the song expressiveness features and the music score features;
[0041] A decoding module, used for decoding based on the singing coding features by the singing synthesis model to be trained to obtain the synthesized singing features for the sample song;
[0042] A model training module is used to update the singing synthesis model to be trained based on the synthesized singing features and the original singing features of the sample song to obtain a target singing synthesis model. The target singing synthesis model is used to output target synthesized singing features according to the input target lyrics text and target music score. The target synthesized singing features are used to generate corresponding synthesized singing audio.
[0043] The present application also provides a singing synthesis device, comprising:
[0044] An input information acquisition module is used to acquire target lyrics text and target music score;
[0045] A synthetic singing voice feature determination module, used to process the target lyrics text and the target music score using the target singing voice synthesis model to obtain a target synthetic singing voice feature; wherein the target singing voice synthesis model is the above-mentioned target singing voice synthesis model;
[0046] The singing voice synthesis module is used to process the target synthesized singing voice features by using a vocoder to obtain the target synthesized singing voice.
[0047] The present application also provides an electronic device, including:
[0048] Memory for storing computer programs;
[0049] A processor is used to implement the steps of the above method when executing the computer program.
[0050] The present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0051] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned singing synthesis model training method and singing synthesis method.
[0052] It can be seen that the present invention obtains sample songs, and the sample songs include lyrics text and music score; extracts lyrics semantic features of the lyrics text through the singing synthesis model to be trained, and extracts music score features of the music score through the singing synthesis model to be trained; extracts attention features based on the lyrics semantic features and music score features through the singing synthesis model to be trained to obtain song expressive features; obtains singing coding features based on the song expressive features and music score features; decodes based on the singing coding features through the singing synthesis model to be trained to obtain synthesized singing features for the sample songs; updates the singing synthesis model to be trained based on the synthesized singing features and the original singing features of the sample songs to obtain a target singing synthesis model, the target singing synthesis model is used to output target synthesized singing features according to the input target lyrics text and target music score, and the target synthesized singing features are used to generate corresponding synthesized singing audio.
[0053] The beneficial effects of the present application are as follows: when performing model training, attention features are extracted based on the semantic features of the lyrics and the features of the music score to obtain the expressive features of the song, so that when performing model training, the synthesized singing features obtained based on the expressive features of the song and the original singing features of the sample song are used to update the singing synthesis model to be trained, so that the singing synthesis model can integrate the emotions of the lyrics into the singing through the semantic features of the lyrics, thereby improving the effect of singing synthesis.
[0054] In addition, the present invention also provides a singing synthesis model training device, equipment and computer-readable storage medium, as well as a singing synthesis method, device and equipment, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0056] Figure 1 A flowchart of a singing synthesis model training method provided by an embodiment of the present invention;
[0057] Figure 2 A flowchart of a singing synthesis model training method provided by an embodiment of the present invention;
[0058] Figure 3 A schematic diagram of a singing synthesis model training framework provided by an embodiment of the present invention;
[0059] Figure 4 A schematic diagram of encoding lyrics text provided by an embodiment of the present invention;
[0060] Figure 5 A flow chart of a singing voice synthesis method provided by an embodiment of the present invention;
[0061] Figure 6 A structural schematic diagram of a singing synthesis model training device provided by an embodiment of the present invention;
[0062] Figure 7 A structural schematic diagram of a singing voice synthesis device provided by an embodiment of the present invention;
[0063] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0064] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.
[0065] During the description of the embodiments of the present application, some nouns or terms are applicable to the following explanations:
[0066] Singing synthesis: Using speech synthesis related technologies to enable a computer to produce beautiful and pleasant singing voices like humans. As a new application field of speech synthesis technology, singing synthesis has considerable application value and prospects in fields such as virtual singers, record production, and digital music creation.
[0067] Phoneme: The smallest unit of Chinese pronunciation (the smallest unit of writing is grapheme). For example, for the Chinese character "好", the pronunciation units are "h" and "ao".
[0068] Syllable: The smallest speech unit formed by the combination of a single vowel phoneme and a consonant phoneme in a language. A single vowel phoneme can also form a syllable by itself. A Chinese syllable is a speech unit formed by the combination of an initial consonant and a final vowel, and a single final vowel can also form a syllable by itself.
[0069] Fundamental frequency: When a sounding body emits sound due to vibration, the sound can generally be decomposed into many simple sine waves. That is to say, all natural sounds are basically composed of many sine waves with different frequencies. Among them, the sine wave with the lowest frequency is the fundamental tone, and the other sine waves with higher frequencies are overtones.
[0070] Breath: Humans produce breath in the lungs to drive the vocal cords to vibrate and produce sound. The vibration of the vocal cords generates a periodic signal, while the breath generates an aperiodic signal. Breath is a very important singing technique in singing. The sound based on breath is elastic; while the sound without breath is dull.
[0071] Mel spectrum: In signal processing, the Mel-Frequency Cepstrum (MFC) is a spectrum that can be used to represent short-term audio. Its principle is based on the logarithmic spectrum represented by a non-linear Mel scale and its linear cosine transform. The frequency band representation method of the Mel spectrum is closer to the non-linear human auditory system.
[0072] Music score: In the synthesis stage, lyrics and pitch information are required. Both of these information come from the music score. Common music score formats include midi, musicxml, etc. The lyrics content and pitch information can be extracted from these formats to create a synthesis template.
[0073] Semantic representation: contains information about vocabulary, syntax, context, entity recognition and relationships, sentiment and emotions in the text, which may exist in the language representation in an explicit or implicit way, depending on the representation method used. Sometimes, deeper processing (such as semantic analysis, sentiment analysis, etc.) may require additional techniques to extract relevant information from the representation.
[0074] Language Model: A language model is a statistical model used to understand, recognize, and generate human language. Fundamentally, it learns and understands the structure of a language, including syntax (grammar), semantics (meaning), and in some cases, context. With this understanding, a language model can generate new sentences, predict the next word in a sequence, or fill in missing words in a sentence. Some examples of language models include BERT (Bidirectional Encoder Representations from Transformers, a pre-trained deep learning model based on the Transformer architecture), GPT (Generative Pre-trained Transformer, a generative pre-trained model based on the Transformer architecture), and LSTM (Long Short-Term Memory, a special type of recurrent neural network).
[0075] Please refer to Figure 1 , Figure 1 A flowchart of a singing synthesis model training method provided by an embodiment of the present invention. The method may include:
[0076] S101, obtaining a sample song, where the sample song includes lyrics and music scores.
[0077] The executor of this embodiment is an electronic device. This embodiment does not limit the specific electronic device, and the electronic device can be a computer, a mobile phone, etc. The sample song in this embodiment is a representative, specific song for reference or analysis. The lyrics text in this embodiment is the content part of the song expressed in words, usually sung by the singer, conveying the theme, emotion and story of the song. The music score in this embodiment is a written form of recording music in the form of notes, symbols and lines. It usually includes the representation of musical elements such as melody, rhythm, harmony, etc. The music score in this embodiment may include phonemes, pitch and phoneme duration. Phonemes are the smallest unit of Chinese pronunciation, and pitch refers to various sounds with different pitches, that is, the height of the sound, a basic feature of the sound. The phoneme duration in this embodiment refers to the time required for the phoneme to be pronounced in a word, phrase or sentence.
[0078] S102, extracting semantic features of lyrics of the lyrics text through the singing voice synthesis model to be trained, and extracting music score features of the music score through the singing voice synthesis model to be trained.
[0079] This embodiment does not limit the method for extracting lyrics semantic features in the singing voice synthesis model to be trained. For example, the lyrics semantic feature extraction method in this embodiment can be a recurrent neural network (RNN); or the lyrics semantic feature extraction method in this embodiment can also be a long short-term memory network (LSTM) or a convolutional neural network (CNN); or the lyrics semantic feature extraction method in this embodiment can also be a Bert language model. The core of Bert semantic feature extraction is to use a pre-trained model to understand and process natural language. The pre-trained model is trained on a large-scale general corpus, which can capture various basic laws and patterns of language. The emotional semantic features of the lyrics in this embodiment can determine the emotion of the song. The score features in this embodiment are information that characterizes the characteristics of the sound. The framework of the singing voice synthesis model to be trained in this embodiment is evolved from the FastSpeech model. FastSpeech is an efficient speech synthesis model designed to solve the problems of slow reasoning speed and unstable synthesized speech in traditional speech synthesis systems.
[0080] It should be further explained that, in order to improve the naturalness and fluency of synthesized speech, the above-mentioned extraction of the lyrics semantic features of the lyrics text through the singing synthesis model to be trained can include: determining the initial lyrics semantic features of the lyrics text through the singing synthesis model to be trained; determining whether there are semantic features belonging to the same word in the initial lyrics semantic features; when there are, repeating the processing based on the semantic features belonging to the same word to obtain repeated semantic features; encoding the repeated semantic features to obtain lyrics semantic features. It can be understood that because the model input is at the phoneme level, it is necessary to make an extension here according to the mapping relationship of word2phn (word2phn is a tool commonly used in speech recognition and speech synthesis systems to convert words into phoneme sequences), and repeat the semantic features belonging to the same word. The repeated processing makes the characters and phonemes aligned, thereby improving the naturalness and fluency of synthesized speech. When using BERT to determine the initial lyric semantic features of the lyric text, since the generated semantic feature representation is distributed, specific information will not be directly encoded as a single dimension or element. Therefore, a feature encoder is needed to allow the model to learn to capture effective information.
[0081] If further explanation is needed, the above-mentioned extraction of the music score features of the music score through the singing synthesis model to be trained may include: performing symbol recognition on the music score to determine the note features; determining the phonemes, pitches and phoneme durations based on the note features; performing text encoding processing on the phonemes and pitches through the singing synthesis model to be trained to obtain text encoding data, performing phoneme expansion on the text encoding data and the phoneme duration to obtain the music score features. It can be understood that based on the note features, the specific phoneme category can be judged by model matching, and the phonemes, pitches and phoneme durations can be more accurately identified, so as to more accurately determine the music score features. It should be noted that this embodiment will perform phoneme expansion on the text encoding data and the phoneme duration. For example, there are 3 frames for "good", and it can be determined that the "good" after repeated processing is hhaoaoao based on the 3 frames. The phoneme expansion in this embodiment is a technology that matches the target Mel spectrum sequence by increasing the length of the phoneme sequence, which is used to improve the naturalness and accuracy of speech synthesis.
[0082] S103, using the singing synthesis model to be trained, extracting attention features based on the semantic features of the lyrics and the features of the music score to obtain the expressive features of the song.
[0083] The song expressiveness feature in this embodiment is the encoding that characterizes the emotion and melody of the lyrics, which refers to the energy and melody changes in the singing process. In this embodiment, the score features can be used as the Q (query, a vector used to query or retrieve information) of the attention mechanism in the singing synthesis model to be trained, and the lyrics semantic features and as K (key, key) and V (value, value), and the song expressiveness features are obtained through the expressiveness perception attention mechanism.
[0084] It should be further explained that the above-mentioned singing synthesis model to be trained performs attention feature extraction based on the semantic features of lyrics and the features of the music score to obtain the expressive features of the song, which may include: using a linear layer or a convolutional layer to convert the dimensions of the semantic features of the lyrics to obtain the semantic features of the lyrics of a preset dimension; performing correlation calculations on the semantic features of the lyrics and the features of the music score of the preset dimension based on the self-attention mechanism to obtain the expressive features of the song. The linear layer in this embodiment, also called the fully connected layer, is used to perform a linear transformation on the input data and map the input features to a new feature space. The convolutional layer in this embodiment can extract features from the input data. The specific preset dimension is not limited in this embodiment. For example, the preset dimension in this embodiment may be 256; or the preset dimension in this embodiment may also be 384. The formula of the self-attention mechanism corresponding to the above processing process is ,in, is the dimension of the feature in the self-attention mechanism, the score feature is Q, the lyrics semantic features are K and V, and T represents transposition. This embodiment determines the self-attention mechanism so that the lyrics semantic features and the score features can be correlated, thereby improving the expressiveness of the model.
[0085] S104, obtaining vocal coding features according to song expressiveness features and music score features.
[0086] In this embodiment, obtaining the singing coding feature according to the song expressiveness feature and the music score feature may refer to adding the song expressiveness feature and the music score feature to obtain the singing coding feature.
[0087] S105, decoding is performed based on the singing coding features by the singing synthesis model to be trained to obtain the synthesized singing features for the sample song.
[0088] In this embodiment, the decoder in the singing synthesis model to be trained can decode the singing coding features. The decoder in this embodiment decompresses and decodes the received coded data to restore it into playable audio and video signals.
[0089] S106, based on the synthesized singing features and the original singing features of the sample song, the singing synthesis model to be trained is updated to obtain a target singing synthesis model. The target singing synthesis model is used to output target synthesized singing features according to the input target lyrics text and target music score. The target synthesized singing features are used to generate corresponding synthesized singing audio.
[0090] This embodiment does not limit the specific process of updating the singing voice synthesis model to be trained based on the synthesized singing voice features and the original singing voice features of the sample song to obtain the target singing voice synthesis model. For example, this embodiment can determine the Mel spectrum loss function based on the synthesized singing voice features and the original singing voice features of the sample song, thereby updating the singing voice synthesis model to be trained and obtaining the target singing voice synthesis model; or this embodiment can also determine the predicted fundamental frequency of the synthesized singing voice features, and determine the fundamental frequency loss function based on the predicted fundamental frequency and the original fundamental frequency of the sample song, thereby updating the parameters of the singing voice synthesis model to be trained based on the fundamental frequency loss function and the Mel spectrum loss function.
[0091] It should be further explained that the above-mentioned updating of the singing voice synthesis model to be trained based on the synthesized singing voice features and the original singing voice features of the sample song to obtain the target singing voice synthesis model may include:
[0092] S1061, determining a mel-spectrogram loss function based on the synthesized singing voice features and the original singing voice features of the sample song;
[0093] S1062, training the singing voice synthesis model to be trained by using the Mel spectrum loss function until the Mel spectrum loss function converges, thereby obtaining a target singing voice synthesis model.
[0094] In this embodiment, the parameters of the singing synthesis model to be trained can be updated through the Mel spectrum loss function to obtain the target singing synthesis model. In this embodiment, the loss function of the singing synthesis model to be trained is the Mel spectrum loss function, that is, the loss function between the synthesized singing feature (predicted Mel spectrum) and the original singing feature (real Mel spectrum).
[0095] It should be further explained that the above-mentioned training of the singing voice synthesis model to be trained by the Mel spectrum loss function until the Mel spectrum loss function converges to obtain the target singing voice synthesis model can include: using the fundamental frequency predictor to determine the predicted fundamental frequency of the synthesized singing voice feature, and determining the fundamental frequency loss function based on the predicted fundamental frequency and the original fundamental frequency of the sample song; updating the singing voice synthesis model to be trained based on the fundamental frequency loss function and the Mel spectrum loss function and continuing the training until the training is completed to obtain the target singing voice synthesis model. The fundamental frequency loss function of this embodiment is a loss function between the predicted fundamental frequency (predicted fundamental frequency) and the real fundamental frequency (original fundamental frequency), such as the L2 loss function, also known as the Euclidean loss function or the mean square error (MSE) loss function. This embodiment does not limit the specific process of training the singing voice synthesis model to be trained by the fundamental frequency loss function and the Mel spectrum loss function. For example, this embodiment can first train the singing synthesis model to be trained based on the Mel spectrum loss function, and then train the singing synthesis model to be trained based on the fundamental frequency loss function; or this embodiment can first train the singing synthesis model to be trained based on the fundamental frequency loss function, and then train the singing synthesis model to be trained based on the Mel spectrum loss function. In this embodiment, through the fundamental frequency loss function, the emotions in the lyrics can be intuitively integrated into the details of the melody, such as the amplitude of the vibrato, the speed of the transition, etc.
[0096] The singing synthesis model training method provided by the embodiment of the present invention may include: S101, obtaining a sample song, the sample song including a lyric text and a music score; S102, extracting the lyrics semantic features of the lyrics text through the singing synthesis model to be trained, and extracting the music score features of the music score through the singing synthesis model to be trained; S103, extracting attention features based on the lyrics semantic features and the music score features through the singing synthesis model to be trained, and obtaining song expressive features; S104, obtaining singing coding features based on the song expressive features and the music score features; S105, decoding based on the singing coding features through the singing synthesis model to be trained, and obtaining synthesized singing features for the sample song; S106, updating the singing synthesis model to be trained based on the synthesized singing features and the original singing features of the sample song, and obtaining a target singing synthesis model, the target singing synthesis model is used to output target synthesized singing features according to the input target lyric text and the target music score, and the target synthesized singing features are used to generate corresponding synthesized singing audio. When performing model training, the present application performs attention feature extraction based on the semantic features of lyrics and the features of music score to obtain the expressive features of the song. Therefore, when performing model training, the synthesized singing features obtained using the expressive features of the song and the original singing features of the sample song are used to update the singing synthesis model to be trained, so that the singing synthesis model can integrate the emotions of the lyrics into the singing through the semantic features of the lyrics, thereby improving the effect of singing synthesis.
[0097] In order to make the present invention easier to understand, please refer to Figure 2 , Figure 2 A flow chart of a singing synthesis model training method provided by an embodiment of the present invention may specifically include:
[0098] S201, using a text encoder to process the phonemes and pitches of the sample song to obtain encoding information.
[0099] In this embodiment, the text encoder is a tool for converting text data into a numerical form that can be processed by a machine learning algorithm. The structural framework diagram corresponding to the singing synthesis model training method in this embodiment is as follows: Figure 3 As shown, Figure 3 A schematic diagram of a singing synthesis model training framework provided by an embodiment of the present invention. In this embodiment, when inputting a score (phonemes, pitches, and phoneme duration), an additional lyric text input is added, which is not perceived by the user and the product does not need to be modified. The training data in this embodiment includes the score, which can be processed to obtain phonemes, pitches, phoneme duration, and fundamental frequency (which are contained in the singing signal).
[0100] S202, perform phoneme expansion based on the coding information and the phoneme duration to obtain music score features as Q.
[0101] S203, using a semantic extractor to process the lyrics text to obtain lyrics semantic features, and using a semantic encoder to encode the lyrics semantic features to obtain lyrics semantic features as K and V.
[0102] The semantic extractor in this embodiment is a natural language processing tool for extracting meaningful information from text data. The semantic extractor in this embodiment corresponds to the semantic feature extraction mentioned above. The semantic feature extractor in this embodiment can be Bert or other language models, and then output word-level features. Because the model input is at the phoneme level, it is necessary to make an extension based on the mapping relationship of word2phn to repeat the semantic features belonging to the same word. Since the semantic representation generated by Bert is distributed, specific information will not be directly encoded as a single dimension or element, so a feature encoder is added to allow the model to learn to capture effective information. K and V in this embodiment are the same. For ease of understanding, please refer to Figure 4 , Figure 4 A schematic diagram of encoding a lyric text provided in an embodiment of the present invention is provided. First, semantic features of the lyric text are extracted, then repeated processing is performed, and finally, the lyric text is processed using a feature encoder to obtain the required encoding features K and V.
[0103] S204, using Q, K, and V to perform correlation calculation based on the self-attention mechanism to obtain the song expressiveness characteristics.
[0104] The expressiveness-aware attention mechanism in this embodiment corresponds to the self-attention mechanism described above. The formula for the expressiveness-aware attention mechanism in this embodiment is ,in, is the dimension of the feature in the attention mechanism, the music score phoneme is encoded as Q, the lyrics emotional semantics are encoded as K and V, and T represents transposition. The encoding features corresponding to the semantics in this embodiment need to be first passed through a linear layer or a convolution layer to convert its dimension to (Not shown in the figure) and then input into the expressiveness-aware attention mechanism as K and V. This embodiment can calculate the correlation between the score features and the song expressiveness features, improve the model expressiveness, add the lyrics semantic information to the information corresponding to the score, and pass through the decoder to obtain the Mel spectrum of the singing voice (synthetic singing voice features), and then pass through a separately trained vocoder (not shown in the figure) to obtain the singing voice audio, and a fundamental frequency predictor module is added before the decoder.
[0105] S205, obtaining singing coding features according to the song expressiveness features and the music score features, and decoding based on the singing coding features by the singing synthesis model to be trained to obtain synthesized singing features for the sample song.
[0106] S206, using a fundamental frequency predictor to determine a predicted fundamental frequency of the synthesized singing voice feature, and determining a fundamental frequency loss function based on the predicted fundamental frequency and an original fundamental frequency of the sample song.
[0107] This embodiment corresponds to Figure 3 The predicted fundamental frequency in this embodiment corresponds to Figure 3 The Pred (Predicted) fundamental frequency curve in this embodiment corresponds to Figure 3 The GT (Ground Truth) fundamental frequency curve in.
[0108] S207, determining a mel-spectrogram loss function based on the synthesized singing voice features and the original singing voice features of the sample song.
[0109] S208, after updating the singing voice synthesis model to be trained based on the fundamental frequency loss function and the mel-spectrogram loss function, continue training until the training is completed to obtain the target singing voice synthesis model.
[0110] The embodiment of the present invention adopts phoneme-level features because the expressiveness of singing voice is relatively detailed. The overall melody direction is determined by the input MIDI (Musical Instrument Digital Interface, used to transmit digital signals between music devices). However, based on the expressiveness under the melody framework, such as the fundamental frequency curve, the feature granularity is finer and more suitable for phoneme-level. This solution uses the attention mechanism to determine the potential relationship between the semantics of lyrics and the expressiveness of singers, and improve the effect of singing synthesis.
[0111] In order to make the present invention easier to understand, please refer to Figure 5 , Figure 5 A flowchart of a singing voice synthesis method provided by an embodiment of the present invention may specifically include:
[0112] S301, obtaining target lyrics text and target music score.
[0113] The target lyrics text and target music score in this embodiment are the lyrics text and music score corresponding to the song that needs to be synthesized.
[0114] S302, using a target singing voice synthesis model to process a target lyrics text and a target music score to obtain a target synthesized singing voice feature; wherein the target singing voice synthesis model is the above-mentioned target singing voice synthesis model.
[0115] The target singing voice synthesis model in this embodiment is a trained singing voice synthesis model obtained through the above training that can integrate the emotions of the lyrics into the singing voice.
[0116] It should be further explained that, in order to improve the accuracy of singing synthesis, the above-mentioned target music score features include target phonemes, target pitches and target phoneme durations. The above-mentioned target song synthesis model is used to process the target lyrics text and the target music score to obtain the target synthesized singing features, which may include:
[0117] S3021, use the text encoder in the target singing synthesis model to process the target phoneme and target pitch to obtain phoneme coding information, and perform phoneme expansion based on the phoneme coding information and the target phoneme duration to determine the target music score features.
[0118] S3022, extracting semantic features of the target lyrics text using the target singing voice synthesis model to obtain lyrics semantic features.
[0119] S3023, using the self-attention mechanism in the target singing synthesis model to process the target score features and lyrics semantic features to determine the expressiveness features of the target song;
[0120] S3024, using the target singing voice synthesis model to superimpose the target song expressiveness feature and the target music score feature to obtain the target singing voice coding feature;
[0121] S3025, decoding is performed based on the target singing coding features through the target singing synthesis model to obtain the target synthesized singing features.
[0122] S303, using a vocoder to process the target synthesized singing voice features to obtain the target synthesized singing voice.
[0123] The embodiment of the present invention uses the target music score features and the target lyrics text as the input of the target singing synthesis model, so that the emotion of the lyrics can be integrated into the melody during the singing synthesis process, thereby improving the effect of singing synthesis.
[0124] The singing voice synthesis model training device provided in an embodiment of the present invention is introduced below. The singing voice synthesis model training device described below and the singing voice synthesis model training method described above can be referenced to each other.
[0125] Please refer to Figure 6 , Figure 6 A structural diagram of a singing synthesis model training device provided by an embodiment of the present invention may include:
[0126] The sample song acquisition module 100 is used to acquire sample songs, wherein the sample songs include lyrics and music scores;
[0127] The feature extraction module 200 is used to extract the semantic features of the lyrics of the lyrics text through the singing synthesis model to be trained, and to extract the music score features of the music score through the singing synthesis model to be trained;
[0128] The attention feature extraction module 300 is used to extract attention features based on the lyrics semantic features and the music score features through the singing synthesis model to be trained to obtain song expressiveness features;
[0129] A singing coding feature determination module 400, used to obtain singing coding features according to the song expressiveness features and the music score features;
[0130] A decoding module 500 is used to decode the singing voice encoding features through the singing voice synthesis model to be trained to obtain the synthesized singing voice features for the sample song;
[0131] The model training module 600 is used to update the singing synthesis model to be trained based on the synthesized singing features and the original singing features of the sample song to obtain a target singing synthesis model. The target singing synthesis model is used to output target synthesized singing features according to the input target lyrics text and target music score. The target synthesized singing features are used to generate corresponding synthesized singing audio.
[0132] Further, based on the above embodiment, the feature extraction module 200 may include:
[0133] An initial lyrics semantic feature extraction unit, used to determine the initial lyrics semantic features of the lyrics text through the singing synthesis model to be trained;
[0134] A judgment unit, used to determine whether there is a semantic feature belonging to the same word in the initial lyrics semantic features;
[0135] A repeated semantic feature determination unit, used for performing repeated processing based on the semantic features belonging to the same character to obtain repeated semantic features when they exist;
[0136] The lyrics semantic feature determination unit is used to encode the repeated semantic features to obtain the lyrics semantic features.
[0137] Further, based on any of the above embodiments, the feature extraction module 200 may include:
[0138] A note feature determination unit, used to perform symbol recognition on the music score to determine note features;
[0139] A phoneme, pitch and phoneme duration determination unit, configured to determine the phoneme, pitch and phoneme duration based on the note feature;
[0140] The music score feature determination unit is used to perform text encoding processing on the phonemes and the pitches through the singing synthesis model to be trained to obtain text encoding data, and perform phoneme expansion on the text encoding data and the phoneme duration to obtain the music score features.
[0141] Further, based on any of the above embodiments, the attention feature extraction module 300 may include:
[0142] A lyrics semantic feature determination unit of a preset dimension, used for converting the dimension of the lyrics semantic feature by using a linear layer or a convolutional layer to obtain lyrics semantic features of a preset dimension;
[0143] The song expressiveness feature determination unit is used to calculate the correlation between the lyrics semantic features of the preset dimension and the music score features based on the self-attention mechanism to obtain the song expressiveness feature.
[0144] Further, based on any of the above embodiments, the model training module 600 may include:
[0145] A Mel spectrum loss function determining unit, used to determine a Mel spectrum loss function based on the synthesized singing voice features and the original singing voice features of the sample song;
[0146] The target singing synthesis model determination unit is used to train the singing synthesis model to be trained by using the Mel spectrum loss function until the Mel spectrum loss function converges to obtain the target singing synthesis model.
[0147] Further, based on the above embodiment, the above target singing synthesis model determination unit may include:
[0148] A fundamental frequency loss function determination subunit, used for determining the predicted fundamental frequency of the synthesized singing voice feature using a fundamental frequency predictor, and determining a fundamental frequency loss function based on the predicted fundamental frequency and the original fundamental frequency of the sample song;
[0149] The target singing synthesis model determination subunit is used to update the singing synthesis model to be trained based on the fundamental frequency loss function and the Mel spectrum loss function and then continue training until the training is completed to obtain the target singing synthesis model.
[0150] It should be noted that the order of the modules and units in the above-mentioned singing synthesis model training device can be changed without affecting the logic.
[0151] The singing synthesis model training device provided by the embodiment of the present invention may include: a sample song acquisition module 100, used to acquire a sample song, wherein the sample song includes a lyric text and a music score; a feature extraction module 200, used to extract the lyrics semantic features of the lyrics text through the singing synthesis model to be trained, and to extract the music score features of the music score through the singing synthesis model to be trained; an attention feature extraction module 300, used to extract the attention features based on the lyrics semantic features and the music score features through the singing synthesis model to be trained, and obtain the song expressiveness features; a singing coding feature determination module 400, used to extract the attention features based on the lyrics semantic features and the music score features through the singing synthesis model to be trained The invention relates to a method for obtaining a singing coding feature according to the song expressiveness feature and the music score feature; a decoding module 500, which is used to decode the singing coding feature through the singing synthesis model to be trained, and obtain the synthesized singing feature for the sample song; a model training module 600, which is used to update the singing synthesis model to be trained based on the synthesized singing feature and the original singing feature of the sample song, and obtain the target singing synthesis model, and the target singing synthesis model is used to output the target synthesized singing feature according to the input target lyrics text and the target music score, and the target synthesized singing feature is used to generate the corresponding synthesized singing audio. When the present application performs model training, attention feature extraction is performed based on the semantic features of lyrics and the features of music score to obtain the expressiveness feature of the song, so that when the model is trained, the synthesized singing feature obtained by using the expressiveness feature of the song and the original singing feature of the sample song are used to update the singing synthesis model to be trained, so that the singing synthesis model can integrate the emotion of the lyrics into the singing through the semantic features of the lyrics, thereby improving the effect of singing synthesis.
[0152] The singing voice synthesis device provided by an embodiment of the present invention is introduced below. The singing voice synthesis device described below and the singing voice synthesis method described above can be referred to each other.
[0153] Please refer to Figure 7 , Figure 7 A structural schematic diagram of a singing voice synthesis device provided by an embodiment of the present invention may include:
[0154] Input information acquisition module 700, used to acquire target lyrics text and target music score;
[0155] The synthetic singing voice feature determination module 800 is used to process the target lyrics text and the target music score using the target singing voice synthesis model to obtain the target synthetic singing voice feature; wherein the target singing voice synthesis model is the above-mentioned target singing voice synthesis model;
[0156] The singing voice synthesis module 800 is used to process the target synthesized singing voice features by using a vocoder to obtain the target synthesized singing voice.
[0157] It should be noted that the order of the modules and units in the above-mentioned singing synthesis device can be changed without affecting the logic.
[0158] The embodiment of the present invention uses the target music score features and the target lyrics text as the input of the target singing synthesis model, so that the emotion of the lyrics can be integrated into the melody during the singing synthesis process, thereby improving the effect of singing synthesis.
[0159] An electronic device provided by an embodiment of the present invention is introduced below. The electronic device described below and the singing synthesis model training method and the singing synthesis method described above can be referenced to each other.
[0160] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention may include:
[0161] A memory 10, used for storing computer programs;
[0162] The processor 20 is used to execute a computer program to implement the above-mentioned singing synthesis model training method and singing synthesis method.
[0163] The memory 10 , the processor 20 , and the communication interface 30 all communicate with each other via a communication bus 40 .
[0164] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:
[0165] Extract the semantic features of the lyrics to obtain the emotional semantic features of the lyrics;
[0166] The music score is processed to obtain music score features; wherein the music score is the music score corresponding to the lyrics; and the music score features are information representing the features of the sound;
[0167] The emotional semantic features of the lyrics are used as input and the score features are used as output to model the relationship between the lyrics, melody and expressiveness. The loss function of the singing synthesis model is determined based on the relationship between the lyrics, melody and expressiveness to obtain the target singing synthesis model; among which, the expressiveness refers to the richness of the melody changes in the singing synthesis process.
[0168] Alternatively, the memory 10 may store a program for implementing the following functions:
[0169] Extract the semantic features of the lyrics to be synthesized to obtain the emotional semantic features of the target lyrics;
[0170] Process the music score to be synthesized to obtain the target music score features;
[0171] The target music score features and the target lyrics emotional semantic features are used as the input of the target singing synthesis model to obtain the target synthesized singing; wherein, the target singing synthesis model is the singing synthesis model mentioned above.
[0172] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.
[0173] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.
[0174] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .
[0175] The communication interface 30 may be an interface of a communication module, and is used to connect to other devices or systems.
[0176] Of course, it should be noted that Figure 8 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Figure 8 More or fewer components than shown, or combinations of certain components.
[0177] The computer-readable storage medium provided in an embodiment of the present invention is introduced below. The computer-readable storage medium described below and the singing synthesis model training method and the singing synthesis method described above can be referenced to each other.
[0178] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned singing synthesis model training method and singing synthesis method are implemented.
[0179] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0180] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned singing synthesis model training method and singing synthesis method.
[0181] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0182] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0183] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0184] The above is a detailed introduction to a singing synthesis model training method, singing synthesis method, device and equipment provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A singing synthesis model training method, characterized in that: include: Acquire a sample song, wherein the sample song includes lyrics and music scores; Extracting semantic features of lyrics of the lyrics text through the singing voice synthesis model to be trained, and extracting music score features of the music score through the singing voice synthesis model to be trained; By using the singing synthesis model to be trained, attention features are extracted based on the semantic features of the lyrics and the features of the music score to obtain song expressiveness features; Obtaining a singing voice coding feature according to the song expressiveness feature and the music score feature; Decoding the singing voice encoding features by the singing voice synthesis model to be trained to obtain the synthesized singing voice features for the sample song; The singing synthesis model to be trained is updated based on the synthesized singing features and the original singing features of the sample song to obtain a target singing synthesis model. The target singing synthesis model is used to output target synthesized singing features according to the input target lyrics text and target music score. The target synthesized singing features are used to generate corresponding synthesized singing audio.
2. The singing synthesis model training method according to claim 1, characterized in that: The lyrics semantic features of the lyrics text are extracted by the singing synthesis model to be trained, including: Determining the initial lyrics semantic features of the lyrics text by the singing synthesis model to be trained; Determine whether there is a semantic feature belonging to the same word in the initial lyrics semantic features; When it exists, repeated processing is performed based on the semantic features belonging to the same word to obtain Repeating semantic features; The repeated semantic features are encoded to obtain the lyrics semantic features.
3. The singing synthesis model training method according to claim 1, characterized in that: Extracting the music score features of the music score through the singing synthesis model to be trained, including: Performing symbol recognition on the music score to determine note features; Determining a phoneme, a pitch, and a phoneme duration based on the note features; The phonemes and the pitches are subjected to text encoding processing by the singing synthesis model to be trained to obtain text encoding data, and the text encoding data and the phoneme duration are subjected to phoneme expansion to obtain the score features.
4. The singing synthesis model training method according to claim 1, characterized in that: By using the singing synthesis model to be trained, attention features are extracted based on the semantic features of the lyrics and the features of the music score to obtain song expressive features, including: The dimension of the lyrics semantic features is transformed using a linear layer or a convolutional layer to obtain The semantic features of lyrics in preset dimensions; The semantic features of the lyrics of the preset dimension and the features of the music score are correlated based on the self-attention mechanism to obtain the expressive features of the song.
5. The singing synthesis model training method according to claim 1, characterized in that: The method includes updating the singing voice synthesis model to be trained based on the synthesized singing voice feature and the original singing voice feature of the sample song to obtain a target singing voice synthesis model, including: Determining a Mel-spectrogram loss function based on the synthesized singing voice features and the original singing voice features of the sample song; The singing voice synthesis model to be trained is trained by using the Mel spectrum loss function until the Mel spectrum loss function converges, thereby obtaining the target singing voice synthesis model.
6. The singing synthesis model training method according to claim 5, characterized in that: The singing voice synthesis model to be trained is trained by using the Mel spectrum loss function until the Mel spectrum loss function converges to obtain the target singing voice synthesis model, including: Determine the predicted fundamental frequency of the synthesized singing voice feature using a fundamental frequency predictor, and determine a fundamental frequency loss function based on the predicted fundamental frequency and the original fundamental frequency of the sample song; After updating the singing voice synthesis model to be trained based on the fundamental frequency loss function and the Mel-spectrogram loss function, the training is continued until the training is completed to obtain the target singing voice synthesis model.
7. A singing voice synthesis method, characterized in that: include: Obtain target lyrics text and target music score; The target lyrics text and the target music score are processed by using the target singing voice synthesis model to obtain target synthesized singing voice features; wherein the target singing voice synthesis model is the target singing voice synthesis model according to any one of claims 1 to 6; The target synthesized singing voice features are processed by using a vocoder to obtain the target synthesized singing voice.
8. A singing synthesis model training device, characterized in that: include: A sample song acquisition module is used to acquire sample songs, wherein the sample songs include lyrics and music scores; A feature extraction module, used to extract the semantic features of the lyrics of the lyrics text through the singing synthesis model to be trained, and to extract the music score features of the music score through the singing synthesis model to be trained; An attention feature extraction module, used to extract attention features based on the lyrics semantic features and the music score features through the singing synthesis model to be trained, so as to obtain song expressiveness features; A singing coding feature determination module, used to obtain singing coding features according to the song expressiveness features and the music score features; A decoding module, used for decoding based on the singing coding features by the singing synthesis model to be trained to obtain the synthesized singing features for the sample song; A model training module is used to update the singing synthesis model to be trained based on the synthesized singing features and the original singing features of the sample song to obtain a target singing synthesis model. The target singing synthesis model is used to output target synthesized singing features according to the input target lyrics text and target music score. The target synthesized singing features are used to generate corresponding synthesized singing audio.
9. A singing voice synthesis device, characterized in that: include: An input information acquisition module is used to acquire target lyrics text and target music score; A synthetic singing voice feature determination module, used to process the target lyrics text and the target music score using the target singing voice synthesis model to obtain a target synthetic singing voice feature; wherein the target singing voice synthesis model is the target singing voice synthesis model according to any one of claims 1 to 6; The singing voice synthesis module is used to process the target synthesized singing voice features by using a vocoder to obtain the target synthesized singing voice.
10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 7 when executing the computer program.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
