Method and apparatus for singing voice synthesis

Abstract

The application provides a song synthesis method and device, which comprises the following steps: obtaining the spectrum data of a target song; extracting phonemes, pitches and audio from the spectrum data; applying an encoder to process the phonemes to obtain an implicit spectrum representation vector; applying a predictor to predict acoustic parameters based on the implicit spectrum representation vector and the pitches; extracting the pitch trajectory in the audio and decomposing the pitch trajectory to obtain the vibrato features corresponding to each syllable in the target song; predicting the probability of each syllable appearing with vibrato based on the vibrato features; when the probability of the syllable appearing with vibrato meets a preset threshold, synthesizing the simulated vibrato corresponding to the syllable and marking the syllable; inputting the spectrum data, the acoustic parameters and the simulated vibrato corresponding to the marked syllable into a preset sound model to generate the synthesized song corresponding to the target song. By using the method provided by the application, the implicit spectrum representation vector is used to improve the prosody of the song, and the vibrato is added to improve the expressiveness of the song, so that the synthesized song is more natural.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a singing synthesis method and device. Background Art

[0002] Singing synthesis is a technology that creates virtual singing voices based on a spectrum of information, including lyrics, rhythm, and pitch. Existing singing synthesis technologies, such as XiaoiceSing, ByteSing, and DurIAN-SC, typically directly use information from the sheet music, such as notes, phonemes, note duration, and pitch, to predict acoustic features. Furthermore, existing technologies also use singing models to predict the energy characteristics of songs and incorporate these characteristics into the synthesized vocals to enhance the overall rhythm of the song.

[0003] When singing, singers usually use vibrato techniques to improve their singing expressiveness. However, the energy features added in the existing technology cannot simulate the effect of a real singer singing vibrato. Moreover, the energy features of the existing technology are obtained by calculating the high-dimensional amplitude spectrum features. Compressing the high-dimensional energy features to one-dimensional energy features will lose a lot of information. The synthesized singing voice has insufficient rhythmic expression and poor expressiveness. Summary of the Invention

[0004] In view of this, the present invention provides a singing synthesis method, through which the singing rhythm is enhanced by implicit spectral representation vectors, and vibrato is added to improve the expressiveness of the singing, making the synthesized singing more natural.

[0005] The present invention also provides a singing voice synthesis device to ensure the realization and application of the above method in practice.

[0006] A singing voice synthesis method comprising:

[0007] Obtain the spectrogram data of the target song;

[0008] Extracting the phonemes, pitches, and audio corresponding to the target song from the spectrum data;

[0009] Applying a preset encoder to process the phonemes to obtain an implicit spectrum representation vector corresponding to the target song, wherein the implicit spectrum representation vector is used to represent energy information of the target song;

[0010] Applying a preset predictor to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch;

[0011] Extracting a pitch trajectory from the audio, and decomposing the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song;

[0012] Predicting the probability of vibrato occurring in each syllable based on the vibrato feature corresponding to each syllable;

[0013] When the probability of vibrato occurring in any syllable meets a preset threshold, synthesizing a simulated vibrato corresponding to the syllable and marking the syllable;

[0014] The spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable are input into a preset sound model to generate a synthetic singing voice corresponding to the target song.

[0015] In the above method, optionally, the step of applying a preset encoder to process the phonemes to obtain an implicit spectral representation vector corresponding to the target song includes:

[0016] Input the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, wherein the initial representation vector is a representation sequence at the phoneme level;

[0017] The initial representation vector is level-expanded to obtain an implicit spectrum representation vector corresponding to the initial representation vector, where the implicit spectrum representation vector is a representation sequence at the frame level.

[0018] In the above method, optionally, applying a preset predictor based on the implicit spectral representation vector and the pitch to predict the acoustic parameters corresponding to the target song includes:

[0019] Inputting the implicit spectral representation vector into a first test layer of the predictor to obtain a spectral representation vector output by the first test layer, where the first test layer is a one-dimensional convolutional layer;

[0020] splicing the spectral representation vector, the implicit spectral representation vector and the pitch to obtain spliced ​​data;

[0021] The spliced ​​data is input into the second test layer of the tester to obtain the acoustic parameters output by the second test layer, where the second test layer is a one-dimensional convolution layer.

[0022] In the above method, optionally, decomposing the pitch trajectory to obtain the vibrato feature corresponding to each syllable in the target song includes:

[0023] Applying a preset triangular filter to perform a convolution operation on the pitch trajectory to obtain a smooth pitch trajectory;

[0024] Decomposing the smooth pitch trajectory to obtain a vibrato corresponding to each syllable;

[0025] A preset Hilbert transform method is applied to perform a transform operation on the vibrato corresponding to each syllable to obtain a vibrato feature corresponding to each syllable.

[0026] In the above method, optionally, the synthesizing of the simulated vibrato corresponding to the syllable includes:

[0027] Selecting the amplitude, frequency and phase of the vibrato feature corresponding to the syllable;

[0028] A preset vibrato simulation algorithm is applied to calculate the amplitude, frequency, phase, and probability of vibrato occurring in the syllable, and to generate a simulated vibrato corresponding to the syllable.

[0029] A singing voice synthesis device, comprising:

[0030] A first acquiring unit, configured to acquire spectrum data of a target song;

[0031] an extraction unit, configured to extract phonemes, pitches, and audio corresponding to the target song from the spectrum data;

[0032] a processing unit, configured to process the phonemes using a preset encoder to obtain an implicit spectrum representation vector corresponding to the target song, wherein the implicit spectrum representation vector is used to represent energy information of the target song;

[0033] A first prediction unit, configured to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch using a preset predictor;

[0034] a decomposition unit, configured to extract a pitch trajectory from the audio, and decompose the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song;

[0035] a second prediction unit, configured to predict a probability of vibrato occurring in each of the syllables based on a vibrato feature corresponding to each of the syllables;

[0036] a first synthesis unit, configured to synthesize a simulated vibrato corresponding to any syllable and mark the syllable when the probability of vibrato occurring in the syllable meets a preset threshold;

[0037] The second synthesis unit is used to input the spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthesized singing voice corresponding to the target song.

[0038] In the above device, optionally, the processing unit includes:

[0039] A first input subunit is configured to input the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, wherein the initial representation vector is a representation sequence at the phoneme level;

[0040] The expansion subunit is used to perform level expansion on the initial representation vector to obtain an implicit spectrum representation vector corresponding to the initial representation vector, where the implicit spectrum representation vector is a frame-level representation sequence.

[0041] In the above device, optionally, the first prediction unit includes:

[0042] a second input subunit, configured to input the implicit spectral representation vector into a first test layer of the predictor to obtain a spectral representation vector output by the first test layer, where the first test layer is a one-dimensional convolutional layer;

[0043] a splicing subunit, configured to splice the spectral representation vector, the implicit spectral representation vector, and the pitch to obtain spliced ​​data;

[0044] The third input subunit is used to input the spliced ​​data into the second test layer of the tester to obtain the acoustic parameters output by the second test layer, where the second test layer is a one-dimensional convolutional layer.

[0045] In the above device, optionally, the decomposition unit includes:

[0046] an operation subunit, configured to apply a preset triangular filter to perform a convolution operation on the pitch trajectory to obtain a smooth pitch trajectory;

[0047] a decomposition subunit, configured to decompose the smooth pitch trajectory to obtain a vibrato corresponding to each syllable;

[0048] The transformation subunit is used to apply a preset Hilbert transformation method to perform a transformation operation on the vibrato corresponding to each syllable to obtain a vibrato feature corresponding to each syllable.

[0049] In the above device, optionally, the first synthesis unit includes:

[0050] A selection subunit, configured to select the amplitude, frequency, and phase of the vibrato feature corresponding to the syllable;

[0051] The synthesis subunit is used to apply a preset vibrato simulation algorithm to calculate the amplitude, frequency, phase and probability of vibrato occurring in the syllable, and generate a simulated vibrato corresponding to the syllable.

[0052] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned singing synthesis method.

[0053] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to execute the above-mentioned singing synthesis method by one or more processors.

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] The present invention provides a singing synthesis method, comprising: obtaining spectrum data of a target song; extracting phonemes, pitches and audio corresponding to the target song in the spectrum data; applying a preset encoder to process the phonemes to obtain an implicit spectrum representation vector corresponding to the target song, wherein the implicit spectrum representation vector is used to represent energy information of the target song; applying a preset predictor to predict acoustic parameters corresponding to the target song based on the implicit spectrum representation vector and the pitch; extracting the pitch trajectory in the audio and decomposing the pitch trajectory to obtain vibrato features corresponding to each syllable in the target song; predicting the probability of vibrato occurring in each syllable based on the vibrato features corresponding to each syllable; when the probability of vibrato occurring in any syllable meets a preset threshold, synthesizing a simulated vibrato corresponding to the syllable and marking the syllable; inputting the spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthesized singing voice corresponding to the target song. By applying the method provided by the present invention, the rhythm of the singing voice is enhanced by using the implicit spectral representation vector, and vibrato is added to improve the expressiveness of the singing voice, making the synthesized singing voice more natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0057] Figure 1 A method flow chart of a singing voice synthesis method provided by an embodiment of the present invention;

[0058] Figure 2 An overall network structure diagram of a singing model provided by an embodiment of the present invention;

[0059] Figure 3 Another method flow chart of a singing voice synthesis method provided by an embodiment of the present invention;

[0060] Figure 4 A structural diagram of a singing voice synthesis device provided by an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0063] In this application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0064] The present invention can be used in numerous general-purpose or special-purpose computing device environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, and so on.

[0065] An embodiment of the present invention provides a singing synthesis method, which can be applied to a variety of system platforms, and its execution subject can be a processor of a computer terminal or various mobile devices. The method flow chart of this method is as Figure 1 shown, specifically including:

[0066] S101: Obtain the spectrogram data of the target song.

[0067] In the embodiment of the present invention, the spectrogram data includes information such as phonemes, pitches, and audio.

[0068] S102: Extract the phonemes, pitches, and audio corresponding to the target song from the said spectrogram data.

[0069] In the embodiment of the present invention, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzed according to the pronunciation actions in a syllable, one action constitutes one phoneme. For example: the Chinese characters "开(kai)" and "心(xin)" each have three phonemes. Pitch refers to sounds with different tones.

[0070] S103: Apply a preset encoder to process the phonemes to obtain an implicit spectrum representation vector corresponding to the target song, where the implicit spectrum representation vector is used to represent energy information of the target song.

[0071] In the embodiment of the present invention, the encoder is a part of the acoustic model. The phoneme is input into the encoder of the acoustic model, and the encoder obtains the representation vector of the phoneme, that is, the implicit spectral representation vector.

[0072] S104: Applying a preset predictor to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch.

[0073] In this embodiment of the present invention, the predictor is part of the acoustic model. It processes the implicit spectral representation vector and combines it with the pitch input into the predictor, which then outputs acoustic parameters. These acoustic parameters include MGC (Mel-Generalized Cepstral), BAP (Band Aperiodicity), and pitch. Band Aperiodicity refers to the power ratio between the speech signal and the signal's aperiodic components.

[0074] S105: Extracting a pitch trajectory from the audio, and decomposing the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song.

[0075] In an embodiment of the present invention, after decomposing the pitch trajectory, the pitch in addition to the vibrato is also included. The vibrato is obtained from the high-pitched trajectory, and the vibrato features corresponding to the syllables in the vibrato are obtained, including amplitude extent, frequency rate and phase.

[0076] Furthermore, the pitch trajectory is decomposed to obtain the vibrato features corresponding to each syllable in the target song, including:

[0077] Applying a preset triangular filter to perform a convolution operation on the pitch trajectory to obtain a smooth pitch trajectory;

[0078] Decomposing the smooth pitch trajectory to obtain a vibrato corresponding to each syllable;

[0079] A preset Hilbert transform method is applied to perform a transform operation on the vibrato corresponding to each syllable to obtain a vibrato feature corresponding to each syllable, wherein the vibrato feature includes amplitude, frequency and phase.

[0080] It's important to note that the actual high-pitched pitch trajectory is extracted from the audio data. This is then convolved with a triangular filter to produce a smoothed high-pitched trajectory. This trajectory is then decomposed into pitch intonation and vibrato. Pitch intonation is the smoothed pitch contour, while vibrato demonstrates the vocalist's expressive technique. Vibrato is further decomposed into three features: amplitude extent, frequency rate, and phase using a Hilbert transform.

[0081] S106: Predicting the probability of vibrato occurring in each syllable based on the vibrato feature corresponding to each syllable.

[0082] In this embodiment of the present invention, the probability of trill occurring in each syllable is the average probability, that is, the average likelihood of trill occurring in each syllable. Trills are essentially absent in shorter syllables and only occur in longer syllables. Therefore, the longer the syllable, the higher the probability of trill occurring.

[0083] Optionally, each vibrato feature can be input into a vibrato similarity annotation network to predict the probability of vibrato occurring in each syllable.

[0084] S107: When the probability of vibrato occurring in any syllable meets a preset threshold, synthesize a simulated vibrato corresponding to the syllable and mark the syllable.

[0085] Furthermore, synthesizing the simulated vibrato corresponding to the syllable may specifically include:

[0086] Selecting the amplitude, frequency and phase of the vibrato feature corresponding to the syllable;

[0087] A preset vibrato simulation algorithm is applied to calculate the amplitude, frequency, phase, and probability of vibrato occurring in the syllable, and to generate a simulated vibrato corresponding to the syllable.

[0088] The calculation formula for synthesizing the simulated vibrato corresponding to each byte is as follows:

[0089]

[0090] Among them, p mean is the average probability of vibrato occurring calculated on the syllable, They are the extent, rate, phase and likeliness of the predicted vibrato respectively.

[0091] It should be noted that, because it is difficult to directly extract whether there is vibrato in the pitch trajectory through an algorithm as a label for the possibility of vibrato in singing data, the present invention proposes a method for constructing simulated vibrato data and labels to obtain vibrato labels and corresponding pitch trajectories.

[0092] First, we extract the true pitchtrajectory from the audio data. We then convolve the extracted pitchtrajectory with a triangular filter to obtain a smoothed pitchtrajectory. We then construct vibrato data using a vibrato simulation algorithm (for example, randomly selecting the vibrato extent and multiplying it by a 6Hz sine wave noise). We then randomly select a segment of the smoothed pitchtrajectory and add vibrato to it, thereby obtaining a pitchtrajectory containing vibrato. We can also determine the position of the randomly selected segment, thus knowing the vibrato label. The randomly selected segment contains vibrato, so the vibrato label for the frame corresponding to this segment is 1, indicating vibrato. Frames outside this segment do not contain vibrato and have a vibrato label of 0.

[0093] With the simulated pitch trajectory containing vibrato and the corresponding label of whether vibrato occurs, we can use this fake data to train a neural network to predict the possibility of vibrato, that is, a neural network that predicts the probability of vibrato occurrence. After the network training is completed, it can be used to label the training data of the pitch model. That is, we can extract the true pitch trajectory of the pitch model training data, and then predict whether vibrato exists at each frame position in the pitch trajectory. In this way, we get the label of the vibrato possibility of the training data for training the pitch model.

[0094] S108: Inputting the spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthetic singing voice corresponding to the target song.

[0095] In the embodiment of the present invention, vibrato and acoustic parameters are added to the singing voice synthesis process to improve the expressiveness of the synthesized singing voice and make the synthesized singing voice more natural.

[0096] It should be noted that the sound model is a neural network model for synthesizing singing voices. After multiple machine learning steps, the model simulates the human voice to synthesize the singing voice corresponding to the target song based on the various input parameters (such as spectral data, acoustic parameters and simulated vibrato).

[0097] In the method provided by an embodiment of the present invention, target spectral data is obtained, and the phonemes, pitch, and audio therein are extracted. The phonemes are processed by an encoder to obtain an implicit spectral representation vector. The implicit spectral representation vector and pitch are further processed by a predictor to obtain one-dimensional acoustic parameters. These acoustic parameters can be used to increase the rhythm of the song during the synthesis of the singing voice. After obtaining the acoustic parameters, the vibrato feature corresponding to each syllable is obtained through the audio, and the vibrato feature is used to predict the possibility of vibrato in each syllable, that is, the probability of vibrato occurring. When vibrato occurs in any syllable, that is, the probability of vibrato occurring is greater than a threshold, a simulated vibrato is synthesized and marked, and a label indicating the likelihood of vibrato occurring in the syllable is set using the mark. Finally, the spectral data, acoustic parameters, and simulated vibrato are input into the sound model to synthesize the singing voice of the target song.

[0098] By applying the method provided in the embodiment of the present invention, the rhythm of the singing voice is enhanced by using the implicit spectral representation vector, and vibrato is added to improve the expressiveness of the singing voice, making the synthesized singing voice more natural.

[0099] refer to Figure 2 , Figure 2 The overall network structure of the singing model consists of three parts. The first part (a) is the acoustic model. The input of the acoustic model includes phonemes and pitch. The acoustic model then predicts the implicit spectral representation vector from the output of the encoder. The predicted implicit spectral representation vector and the representation obtained by the encoder are concatenated and sent to the decoder to predict acoustic features (MGC, BAP, Pitch). Figure 1 (b) is the pitchmodel. The input of the pitchmodel is the phoneme and syllable as well as the corresponding singer ID. In the pitchmodel, pitch is decomposed into information such as intonation, phase, extent, rate, etc., and we also model the possibility of vibrato, which is the likeliness in the figure. Figure 1 (c) is the vibrato possibility labeling network, which is used to label the vibrato possibility labels for the training data. Figure 2 In the figure, “c” stands for concatenation, “+” stands for addition, “f” is the formula for generating simulated vibrato, “FC” is the fully connected layer, “LR” is the length regulator, “Rep” is the abbreviation of representation, and Vibrato likeliness is the possibility of vibrato appearing, that is, the probability of vibrato appearing.

[0100] In the method provided in the embodiment of the present invention, a preset encoder is applied to process the phonemes to obtain an implicit spectral representation vector corresponding to the target song, including:

[0101] Input the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, wherein the initial representation vector is a representation sequence at the phoneme level;

[0102] The initial representation vector is level-expanded to obtain an implicit spectrum representation vector corresponding to the initial representation vector, where the implicit spectrum representation vector is a representation sequence at the frame level.

[0103] It should be noted that if Figure 2 As shown, the process of obtaining the implicit spectral representation vector can be: the input of the encoder of the acoustic model is the phoneme, after the representation vector of the phoneme is obtained through the encoder, the output of the encoder is expanded to the frame level according to the phoneme duration information in the song score, that is, the representation sequence at the phoneme level is expanded to the representation sequence at the frame level.

[0104] Further, refer to Figure 3 , applying a preset predictor based on the implicit spectrum representation vector and the spectrum data to predict the acoustic parameters corresponding to the target song, including:

[0105] S301: Input the implicit spectral representation vector into the first test layer of the predictor to obtain the spectral representation vector output by the first test layer.

[0106] The first test layer is a one-dimensional convolutional layer.

[0107] It should be noted that the predictor contains two layers of one-dimensional convolution, and the output layer is a linear transformation, which predicts a spectral representation vector with a dimension of 256 per frame.

[0108] S302: Concatenate the spectral representation vector, the implicit spectral representation vector, and the pitch to obtain concatenated data.

[0109] S303: Input the spliced ​​data into the second test layer of the tester to obtain the acoustic parameters output by the second test layer.

[0110] Among them, the second test layer is a one-dimensional convolutional layer.

[0111] It should be noted that the output of the encoder that predicts the spectral representation and acoustic model, as well as the pitch chembedding, are concatenated and fed into the acoustic model tester for acoustic parameter prediction. The acoustic parameters used in this invention are MGC (Mel-Generalized Cepstral), BAP (Band Aperiodicity, with aperiodicity), and Pitch. The encoder output is an implicit spectral representation vector. As shown in Table 1, Table 1 shows the dimensions of different features of acoustic parameters in the acoustic model:

[0112]

[0113] Table 1

[0114] It should also be noted that in order to make the spectral representation vector represent the energy information very well, the present invention adds a decoder to the acoustic model, so that the predicted spectral representation vector can reply the energy spectrum of the audio corresponding to the input text through the decoder. Because the singing data appear in pairs, that is, one text input corresponds to one audio output, the spectral representation obtained by the spectral representation predictor can be replied to the corresponding energy spectrum through a decoder. This decoder is Figure 2 The part in the dotted box in part (a) is only needed during training and is not needed during inference. During inference, the spectral representation predicted by the spectral representation predictor can be used to predict acoustic parameters.

[0115] Based on the method provided in the above embodiment, the present invention uses a vibrato model to display the Phase, Extent, Rate, Likeliness and other features of the predicted vibrato, and then uses these features to improve the expressiveness of the singing model; a method for improving the rhythmic expressiveness of singing by introducing a compressed, highly expressive latent spectrogram representation (i.e., energy bottle neck feature) into the singing model.

[0116] The specific implementation processes and derivative methods of the above embodiments are all within the protection scope of the present invention.

[0117] and Figure 1 Corresponding to the method described above, the embodiment of the present invention further provides a singing synthesis device for Figure 1 The specific implementation of the method, the singing voice synthesis device provided by the embodiment of the present invention can be applied to a computer terminal or various mobile devices, and its structural diagram is as shown in FIG. Figure 4 As shown, specifically including:

[0118] The first acquisition unit 401 is used to acquire the spectrum data of the target song;

[0119] An extraction unit 402 is configured to extract phonemes, pitches, and audio corresponding to the target song from the spectrum data;

[0120] The processing unit 403 is configured to process the phonemes using a preset encoder to obtain an implicit spectrum representation vector corresponding to the target song, where the implicit spectrum representation vector is used to represent energy information of the target song.

[0121] A first prediction unit 404 is configured to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch using a preset predictor;

[0122] a decomposition unit 405 for extracting a pitch trajectory from the audio, and decomposing the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song;

[0123] A second prediction unit 406 is configured to predict a probability of vibrato occurring in each syllable based on a vibrato feature corresponding to each syllable;

[0124] A first synthesis unit 407 is configured to synthesize a simulated vibrato corresponding to any syllable and mark the syllable when the probability of vibrato occurring in the syllable meets a preset threshold;

[0125] The second synthesis unit 408 is used to input the spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthesized singing voice corresponding to the target song.

[0126] In the device provided by an embodiment of the present invention, target spectral data is acquired and the phonemes, pitch, and audio are extracted. The phonemes are processed by an encoder to obtain an implicit spectral representation vector. The implicit spectral representation vector and pitch are further processed by a predictor to obtain one-dimensional acoustic parameters. These acoustic parameters can be used to enhance the rhythm of the song during vocal synthesis. After obtaining the acoustic parameters, the vibrato feature corresponding to each syllable is obtained from the audio. The vibrato feature is used to predict the likelihood of vibrato occurring in each syllable, i.e., the probability of vibrato occurring. When vibrato occurs in any syllable, i.e., the probability of vibrato occurring is greater than a threshold, a simulated vibrato is synthesized and marked. The mark is used to set a label indicating the likelihood of vibrato occurring in the syllable. Finally, the spectral data, acoustic parameters, and simulated vibrato are input into a sound model to synthesize the vocals of the target song.

[0127] By using the device provided by the embodiment of the present invention, the rhythm of the singing voice is improved through the implicit spectral representation vector, and vibrato is added to improve the expressiveness of the singing voice, making the synthesized singing voice more natural.

[0128] In the apparatus provided by the embodiment of the present invention, the processing unit 403 includes:

[0129] A first input subunit is configured to input the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, wherein the initial representation vector is a representation sequence at the phoneme level;

[0130] The expansion subunit is used to perform level expansion on the initial representation vector to obtain an implicit spectrum representation vector corresponding to the initial representation vector, where the implicit spectrum representation vector is a frame-level representation sequence.

[0131] In the apparatus provided by the embodiment of the present invention, the first prediction unit 404 includes:

[0132] a second input subunit, configured to input the implicit spectral representation vector into a first test layer of the predictor to obtain a spectral representation vector output by the first test layer, where the first test layer is a one-dimensional convolutional layer;

[0133] a splicing subunit, configured to splice the spectral representation vector, the implicit spectral representation vector, and the pitch to obtain spliced ​​data;

[0134] The third input subunit is used to input the spliced ​​data into the second test layer of the tester to obtain the acoustic parameters output by the second test layer, where the second test layer is a one-dimensional convolutional layer.

[0135] In the device provided by the embodiment of the present invention, the decomposition unit 405 includes:

[0136] an operation subunit, configured to apply a preset triangular filter to perform a convolution operation on the pitch trajectory to obtain a smooth pitch trajectory;

[0137] a decomposition subunit, configured to decompose the smooth pitch trajectory to obtain a vibrato corresponding to each syllable;

[0138] The transformation subunit is used to apply a preset Hilbert transformation method to perform a transformation operation on the vibrato corresponding to each syllable to obtain a vibrato feature corresponding to each syllable.

[0139] In the apparatus provided in the embodiment of the present invention, the first synthesis unit 407 includes:

[0140] A selection subunit, configured to select the amplitude, frequency, and phase of the vibrato feature corresponding to the syllable;

[0141] The synthesis subunit is used to apply a preset vibrato simulation algorithm to calculate the amplitude, frequency, phase and probability of vibrato occurring in the syllable, and generate a simulated vibrato corresponding to the syllable.

[0142] The specific working process of each unit and sub-unit in the singing voice synthesis device disclosed in the above embodiment of the present invention can be found in the corresponding content in the singing voice synthesis method disclosed in the above embodiment of the present invention, and will not be repeated here.

[0143] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned singing synthesis method.

[0144] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 5As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to perform the following operations:

[0145] Obtain the spectrogram data of the target song;

[0146] Extracting the phonemes, pitches, and audio corresponding to the target song from the spectrum data;

[0147] Applying a preset encoder to process the phonemes to obtain an implicit spectrum representation vector corresponding to the target song, wherein the implicit spectrum representation vector is used to represent energy information of the target song;

[0148] Applying a preset predictor to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch;

[0149] Extracting a pitch trajectory from the audio, and decomposing the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song;

[0150] Predicting the probability of vibrato occurring in each syllable based on the vibrato feature corresponding to each syllable;

[0151] When the probability of vibrato occurring in any syllable meets a preset threshold, synthesizing a simulated vibrato corresponding to the syllable and marking the syllable;

[0152] The spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable are input into a preset sound model to generate a synthetic singing voice corresponding to the target song.

[0153] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0154] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0155] To clearly illustrate the interchangeability of hardware and software, the above descriptions have generally described the components and steps of each example by function. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the present invention.

[0156] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A singing voice synthesis method, characterized in that: include: Obtain the spectrogram data of the target song; Extracting the phonemes, pitches, and audio corresponding to the target song from the spectrum data; Applying a preset encoder to process the phonemes to obtain an implicit spectrum representation vector corresponding to the target song, wherein the implicit spectrum representation vector is used to represent energy information of the target song; Applying a preset predictor to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch; the acoustic parameters include a Mel-generalized cepstrum, a non-periodic signal, and pitch; the non-periodic signal refers to the power ratio between the speech signal and the non-periodic component of the signal; Extracting a pitch trajectory from the audio, and decomposing the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song; Obtaining a probability of vibrato occurring in each of the syllables predicted by a vibrato similarity annotation network based on the vibrato features corresponding to each of the syllables; When the probability of vibrato occurring in any syllable meets a preset threshold, synthesizing a simulated vibrato corresponding to the syllable and marking the syllable; Inputting the spectrum data, the acoustic parameters, and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthetic singing voice corresponding to the target song; The synthesizing of the simulated vibrato corresponding to the syllable includes: Selecting the amplitude, frequency and phase of the vibrato feature corresponding to the syllable; A preset vibrato simulation algorithm is applied to calculate the amplitude, frequency, phase, and probability of vibrato occurring in the syllable, and to generate a simulated vibrato corresponding to the syllable.

2. The method according to claim 1, characterized in that The applying a preset encoder to process the phoneme to obtain an implicit spectrum representation vector corresponding to the target song includes: Input the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, wherein the initial representation vector is a representation sequence at the phoneme level; The initial representation vector is level-expanded to obtain an implicit spectrum representation vector corresponding to the initial representation vector, where the implicit spectrum representation vector is a representation sequence at the frame level.

3. The method according to claim 1 or 2, characterized in that The application of a preset predictor to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch includes: Inputting the implicit spectral representation vector into a first test layer of the predictor to obtain a spectral representation vector output by the first test layer, where the first test layer is a one-dimensional convolutional layer; splicing the spectral representation vector, the implicit spectral representation vector and the pitch to obtain spliced ​​data; The spliced ​​data is input into the second test layer of the tester to obtain the acoustic parameters output by the second test layer, where the second test layer is a one-dimensional convolution layer.

4. The method according to claim 1, wherein Decomposing the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song includes: Applying a preset triangular filter to perform a convolution operation on the pitch trajectory to obtain a smooth pitch trajectory; Decomposing the smooth pitch trajectory to obtain a vibrato corresponding to each syllable; A preset Hilbert transform method is applied to perform a transform operation on the vibrato corresponding to each syllable to obtain a vibrato feature corresponding to each syllable.

5. A singing voice synthesis device, characterized in that: include: A first acquiring unit, configured to acquire spectrum data of a target song; an extraction unit, configured to extract phonemes, pitches, and audio corresponding to the target song from the spectrum data; a processing unit, configured to process the phonemes using a preset encoder to obtain an implicit spectrum representation vector corresponding to the target song, wherein the implicit spectrum representation vector is used to represent energy information of the target song; A first prediction unit is configured to apply a preset predictor to predict acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch; the acoustic parameters include a Mel-valued generalized cepstrum, a non-periodic signal, and pitch; the non-periodic signal refers to a power ratio between a speech signal and a non-periodic component of the signal; a decomposition unit, configured to extract a pitch trajectory from the audio, and decompose the pitch trajectory to obtain a vibrato feature corresponding to each syllable in the target song; a second prediction unit, configured to obtain a probability of vibrato occurring in each syllable, predicted by a vibrato similarity annotation network based on the vibrato feature corresponding to each syllable; a first synthesis unit, configured to synthesize a simulated vibrato corresponding to any syllable and mark the syllable when the probability of vibrato occurring in the syllable meets a preset threshold; a second synthesis unit, configured to input the spectrum data, the acoustic parameters, and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthesized singing voice corresponding to the target song; Wherein, the first synthesis unit comprises: A selection subunit, configured to select the amplitude, frequency, and phase of the vibrato feature corresponding to the syllable; The synthesis subunit is used to apply a preset vibrato simulation algorithm to calculate the amplitude, frequency, phase and probability of vibrato occurring in the syllable, and generate a simulated vibrato corresponding to the syllable.

6. The device according to claim 5, characterized in that The processing unit includes: A first input subunit is configured to input the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, wherein the initial representation vector is a representation sequence at the phoneme level; The expansion subunit is used to perform level expansion on the initial representation vector to obtain an implicit spectrum representation vector corresponding to the initial representation vector, where the implicit spectrum representation vector is a frame-level representation sequence.

7. The device according to claim 5 or 6, characterized in that The first prediction unit includes: a second input subunit, configured to input the implicit spectral representation vector into a first test layer of the predictor to obtain a spectral representation vector output by the first test layer, where the first test layer is a one-dimensional convolutional layer; a splicing subunit, configured to splice the spectral representation vector, the implicit spectral representation vector, and the pitch to obtain spliced ​​data; The third input subunit is used to input the spliced ​​data into the second test layer of the tester to obtain the acoustic parameters output by the second test layer, where the second test layer is a one-dimensional convolutional layer.

8. The device according to claim 5, characterized in that The decomposition unit comprises: an operation subunit, configured to apply a preset triangular filter to perform a convolution operation on the pitch trajectory to obtain a smooth pitch trajectory; a decomposition subunit, configured to decompose the smooth pitch trajectory to obtain a vibrato corresponding to each syllable; The transformation subunit is used to apply a preset Hilbert transformation method to perform a transformation operation on the vibrato corresponding to each syllable to obtain a vibrato feature corresponding to each syllable.

Images