Model training methods and components, phoneme pronunciation duration annotation methods and components

Abstract

The present application discloses a model training method and component and a phoneme pronunciation duration labeling method and component. The scheme uses syllables in a song syllable set as training samples, and can train a phoneme pronunciation duration recognition model that can constrain the phoneme pronunciation duration. The model can determine, for any syllable, how long the vowel is pronounced and how long the consonant is pronounced when the syllable is pronounced in a song. Therefore, the model can be used to label the duration of the pronunciation phonemes of the syllable in the song, so that it can be used to synthesize singing based on the labeled information.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method and component, and a phoneme pronunciation duration labeling method and component. Background Art

[0002] Currently, the existing technology predicts the pronunciation duration of phonemes based on text, but text has no constraints on pronunciation duration. This method without duration constraints is only applicable to speech synthesis scenarios, that is, the pronunciation duration of each phoneme is not restricted and has no strict standards.

[0003] The accuracy of pronunciation duration prediction only affects the naturalness of speech synthesis and has little impact on actual use. However, for singing synthesis, the accuracy of pronunciation duration prediction directly affects the quality of song synthesis. Because the beat information of the song limits the pronunciation duration of each word, if the pronunciation duration of the phoneme is not constrained, the synthesized song pronunciation will not match the song melody. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a model training method and component, a phoneme pronunciation duration labeling method and component, so as to obtain a phoneme pronunciation duration recognition model suitable for singing and chorus scenes. The specific scheme is as follows:

[0005] To achieve the above objectives, on the one hand, the present application provides a model training method, comprising:

[0006] Acquire a target syllable and a label of the target syllable from a song syllable set; the label includes: a consonant pronunciation frame number N0 of a consonant phoneme in the target syllable, and a vowel pronunciation frame number N1 of a vowel phoneme in the target syllable;

[0007] According to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, constructing a target sequence including N0 consonant tokens and N1 vowel tokens;

[0008] Repeat the target syllable N times to obtain a syllable group; wherein N = N0 + N1;

[0009] Inputting the syllable group into a neural network model so that the neural network model outputs a prediction sequence; the prediction sequence includes N3 consonant tokens and N4 vowel tokens; wherein N3+N4=N;

[0010] Calculating a loss value between the predicted sequence and the target sequence;

[0011] Adjusting the parameters of the neural network model based on the loss value to obtain an updated model;

[0012] If the updated model converges, the updated model is used as the phoneme pronunciation duration recognition model.

[0013] Preferably, it also includes:

[0014] If the updated model does not converge, another syllable and the label of the syllable are reacquired from the song syllable set to iteratively train the updated model until the model converges, and the updated model is used as the phoneme pronunciation duration recognition model.

[0015] Preferably, inputting the syllable group into a neural network model so that the neural network model outputs a prediction sequence comprises:

[0016] The neural network model determines the total number of pronunciation frames N of the target syllable based on the number of repetitions of the target syllable in the syllable group, and predicts the phoneme type of each frame of pronunciation in the target syllable;

[0017] If the phoneme type of any frame pronunciation is a consonant, a consonant mark is recorded; if the phoneme type of any frame pronunciation is a vowel, a vowel mark is recorded to obtain the predicted sequence.

[0018] Preferably, the predicting the phoneme type of each frame of pronunciation in the target syllable comprises:

[0019] The phoneme type of each frame of pronunciation in the target syllable is predicted by consulting a phoneme dictionary.

[0020] Preferably, the song syllable set is obtained by converting the Chinese lyrics text according to the corresponding song audio.

[0021] Preferably, if the target syllable is a monophonetic vowel syllable, after constructing a target sequence including N0 consonant markers and N1 vowel markers according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, the method further includes:

[0022] Fill a consonant marker at the first position of the target sequence;

[0023] Accordingly, before calculating the loss value between the predicted sequence and the target sequence, the method further includes:

[0024] A consonant token is filled at the first position of the predicted sequence.

[0025] Preferably, before calculating the loss value between the predicted sequence and the target sequence, the method further includes:

[0026] Noise points in the predicted sequence are detected and removed.

[0027] Preferably, the calculating the loss value between the predicted sequence and the target sequence includes:

[0028] Compare the predicted sequence with the target sequence to obtain N comparison results;

[0029] The loss value is determined based on the N comparison results.

[0030] On the other hand, the present application also provides a method for marking the pronunciation duration of a phoneme, comprising:

[0031] Get the text to be annotated;

[0032] Convert the text to be annotated into syllables;

[0033] Inputting the syllable into the above-mentioned phoneme pronunciation duration recognition model so that the phoneme pronunciation duration recognition model outputs a marked sequence;

[0034] Determine the number of vowel pronunciation frames of the vowel phonemes and the number of consonant pronunciation frames of the consonant phonemes in the syllable based on the marked sequence;

[0035] The product of the number of vowel pronunciation frames and the unit frame length is recorded as the vowel pronunciation duration, and the product of the number of consonant pronunciation frames and the unit frame length is recorded as the consonant pronunciation duration;

[0036] The vowel pronunciation duration and the consonant pronunciation duration are marked on the syllable.

[0037] Preferably, it also includes:

[0038] A set annotated with the vowel pronunciation duration and the consonant pronunciation duration is constructed to train a singing synthesis model using the song syllable set.

[0039] On the other hand, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned model training method.

[0040] On the other hand, the present application also provides a storage medium, which stores computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, the aforementioned model training method is implemented.

[0041] The present application uses syllables in a song syllable set as training samples, and can obtain a target syllable, the number of consonant pronunciation frames N0 of the consonant phonemes in the target syllable, and the number of vowel pronunciation frames N1 of the vowel phonemes in the target syllable; then, according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, a target sequence including N0 consonant markers and N1 vowel markers is constructed; the target syllable is repeated N times to obtain a syllable group, and the syllable group is input into a neural network model so that the neural network model knows that the target syllable needs N frame times to be pronounced, and thus outputs a predicted sequence including N3 consonant markers and N4 vowel markers; the loss value between the predicted sequence and the target sequence is calculated; the parameters of the neural network model are adjusted based on the loss value to obtain an updated model; if the updated model converges, the updated model is used as a phoneme pronunciation duration recognition model. It can be seen that this scheme uses syllables in the song syllable set as training samples, and can train a phoneme pronunciation duration recognition model that can constrain the pronunciation duration of phonemes. For any syllable, this model can determine how long the vowel is pronounced and how long the consonant is pronounced when the syllable is pronounced in a song. Therefore, this model can be used to annotate the duration of the pronunciation phonemes of the syllable in the song, and then it can be used to synthesize singing based on the labeled information.

[0042] Correspondingly, a model training component and a phoneme pronunciation duration labeling method and component (components include: devices, equipment and storage media) provided in this application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of a physical architecture applicable to the present application provided for the present application;

[0045] Figure 2 A flow chart of a model training method provided for this application;

[0046] Figure 3 A syllable decomposition diagram provided for this application;

[0047] Figure 4 A flow chart of a method for marking the pronunciation duration of phonemes provided in this application;

[0048] Figure 5 A flow chart of another method for marking the pronunciation duration of phonemes provided in this application;

[0049] Figure 6 A schematic diagram of a phoneme pronunciation duration recognition model provided for this application;

[0050] Figure 7 A schematic diagram of a model training device provided in this application;

[0051] Figure 8 A schematic diagram of a phoneme pronunciation duration marking device provided in this application;

[0052] Fig. 9 A server structure diagram provided for this application;

[0053] Fig.10 A terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0054] The existing technology uses text as the object to predict the pronunciation duration of phonemes, but text has no pronunciation duration constraints. This method without duration constraints is only applicable to speech synthesis scenarios, not to singing synthesis scenarios. Because the beat information of the song limits the pronunciation duration of each word, if the phoneme pronunciation duration is not constrained, the synthesized song pronunciation will not match the song melody.

[0055] In view of the above-mentioned problems that currently exist, the present application proposes a model training scheme, which uses syllables in a song syllable set as training samples, and can train a phoneme pronunciation duration recognition model that can constrain the pronunciation duration of phonemes. For any syllable, the model can determine how long the vowel is pronounced and how long the consonant is pronounced when the syllable is pronounced in a song. Therefore, the model can be used to annotate the duration of the pronunciation phonemes of the syllable in the song, and thus can be used to synthesize singing based on the annotated information.

[0056] The professional terms involved in this application:

[0057] Phoneme: The smallest unit of speech divided according to the natural properties of speech. From the perspective of acoustic properties, phonemes are the smallest unit of speech divided from the perspective of sound quality. From the perspective of physiological properties, one pronunciation action forms a phoneme. For example, 〔ma〕 contains two pronunciation actions 〔m〕〔a〕, which are two phonemes. The sound produced by the same pronunciation action is the same phoneme, and the sounds produced by different pronunciation actions are different phonemes. For example, in 〔ma-mi〕, the two 〔m〕 have the same pronunciation action, which is the same phoneme, while the pronunciation actions of 〔a〕〔i〕 are different, which are different phonemes. The analysis of phonemes is generally described based on pronunciation actions.

[0058] Syllable: The most natural structural unit of speech. A syllable is the smallest unit of speech structure composed of a combination of phonemes. It consists of three parts: the head, the ventral part, and the tail part, so there are clear and perceptible boundaries between syllables. In Chinese, the pronunciation of a Chinese character is generally a syllable. There are 400 basic untoned syllables and more than 1,300 toned syllables (excluding light tones) in Mandarin.

[0059] For ease of understanding, the physical framework to which this application applies is first introduced.

[0060] It should be understood that the model training method provided in the present application can be applied to a system or program with a model training function. Of course, the phoneme pronunciation duration annotation method provided in the present application can also be applied to a system or program with a phoneme pronunciation duration annotation function. Specifically, a system or program with a model training function, and a system or program with a phoneme pronunciation duration annotation function can be run in a server, a personal computer, or other device.

[0061] like Figure 1 As shown, Figure 1 This is a schematic diagram of the physical architecture applicable to this application. Figure 1 In the invention, a system or program with a model training function can be run on a server, which obtains a song syllable set from other terminal devices through a network, and obtains a target syllable and a label of the target syllable from the song syllable set; the label includes: the number of consonant pronunciation frames N0 of the consonant phoneme in the target syllable, and the number of vowel pronunciation frames N1 of the vowel phoneme in the target syllable; according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, a target sequence including N0 consonant markers and N1 vowel markers is constructed; the target syllable is repeated N times to obtain a syllable group; the syllable group is input into a neural network model so that the neural network model outputs a predicted sequence including N3 consonant markers and N4 vowel markers; the loss value between the predicted sequence and the target sequence is calculated; the parameters of the neural network model are adjusted based on the loss value to obtain an updated model; if the updated model converges, the updated model is used as a phoneme pronunciation duration recognition model.

[0062] exist Figure 1 In the invention, a system or program with a phoneme pronunciation duration annotation function can also be run on a server, which obtains the text to be annotated from other terminal devices through a network; converts the text to be annotated into syllables; inputs the syllables into the phoneme pronunciation duration recognition model described above, so that the phoneme pronunciation duration recognition model outputs an annotation sequence; determines the number of vowel pronunciation frames of the vowel phonemes and the number of consonant pronunciation frames of the consonant phonemes in the syllable based on the annotation sequence; records the product of the number of vowel pronunciation frames and the unit frame length as the vowel pronunciation duration, and records the product of the number of consonant pronunciation frames and the unit frame length as the consonant pronunciation duration; annotates the vowel pronunciation duration and the consonant pronunciation duration on the syllable.

[0063] As shown in the figure, the server can establish communication connections with multiple devices, and the server obtains syllables that meet the training conditions or texts that need to be annotated with phoneme pronunciation duration from these devices. The server collects syllables uploaded by these devices and trains the neural network model to obtain a phoneme pronunciation duration recognition model.

[0064] Figure 1 A variety of terminal devices are shown in FIG. In actual scenarios, more or fewer types of terminal devices may participate in the model training or phoneme pronunciation duration labeling process. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in an actual scenario, multiple servers may be involved, and the specific number of servers depends on the actual scenario.

[0065] It should be noted that the model training method provided in this embodiment can be performed offline, that is, the server locally stores a set of song syllables, which can directly use the solution provided in this application to train the desired model. Of course, the phoneme pronunciation duration annotation method provided in this embodiment can be performed offline, that is, the server locally stores a text to be annotated, which can directly use the solution provided in this application to evaluate the syllables corresponding to the text.

[0066] It is understandable that the above-mentioned systems and programs with model training functions, or systems and programs with phoneme pronunciation duration marking functions, can also be run on personal mobile terminals or as a cloud service program. The specific operating mode depends on the actual scenario and is not limited here.

[0067] Specifically, after the model training is completed, the obtained model can perform syllable annotation on a large amount of text, thereby obtaining a syllable set with annotation information, and the syllable set can be used to train a singing synthesis model.

[0068] Combined with the above commonalities, please see Figure 2 , Figure 2 A flow chart of a model training method provided in an embodiment of the present application. Figure 2 As shown, the model training method may include the following steps:

[0069] S201. Acquire a target syllable, a consonant pronunciation frame number N0 of a consonant phoneme in the target syllable, and a vowel pronunciation frame number N1 of a vowel phoneme in the target syllable from a song syllable set.

[0070] In this embodiment, the song syllable set is obtained by converting the lyric text according to the corresponding song audio. That is: the total number of pronunciation frames of each syllable in the song syllable set, the number of consonant pronunciation frames of the consonant phoneme in the syllable, and the number of vowel pronunciation frames of the vowel phoneme in the syllable are all determined based on the pronunciation frames of the syllable in the corresponding song. Therefore, the pronunciation duration of each syllable in the song syllable set is known, the pronunciation duration of the consonant phoneme in the syllable is known, and the pronunciation duration of the vowel phoneme in the syllable is also known.

[0071] Among them, Chinese characters generally consist of two phonemes, namely a consonant phoneme and a vowel phoneme. Therefore, the total number of frames N of the target syllable = N0 + N1.

[0072] For example: the pronunciation syllable of "我" is 〔wo〕, which can be split into two phonemes 〔_u〕 and 〔uo〕. Suppose "我" is pronounced for 203 milliseconds in a certain song, and the preset unit frame length is 10 milliseconds (that is, the time length corresponding to one frame of audio). Then 203 / 10 = 20.3 ≈ 21 frames (rounded by the rounding method). At this time, it can be considered that the total number of pronunciation frames of "我" in a certain song is 21. If it is detected that 〔_u〕 accounts for 112 milliseconds, then it can be considered that 〔_u〕 is pronounced for 11 frames. If it is detected that 〔uo〕 accounts for 96 milliseconds, then it can be considered that 〔uo〕 is pronounced for 10 frames.

[0073] Among them, any Chinese character corresponding syllable can be decomposed by a speech annotation tool to obtain information such as the pronunciation duration of different phonemes in the syllable and the total pronunciation duration of the syllable in a certain song. However, these tools often decompose inaccurately, and the decomposition results can be manually adjusted when necessary. The speech annotation tool supports manual adjustment.

[0074] As Figure 3 shown, the speech annotation tool can visualize the song audio, and then decompose each syllable to form a visualization graph as Figure 3 shown. Figure 3 The "丨" - shaped black line in it supports mouse dragging, so the decomposition result can be adjusted manually. Accordingly, some songs can be decomposed to obtain the song syllable set.

[0075] It should be noted that since the time length is not easy to annotate, in this embodiment, the pronunciation frames are used as the labels of the syllables. Because the unit frame length is always constant, only by determining the pronunciation frames of the phonemes, it is equivalent to determining the pronunciation duration of the phonemes.

[0076] Of course, for songs and lyrics in other languages (such as English, etc.), they can also be converted according to the above to obtain the corresponding song syllable set.

[0077] S202 . Construct a target sequence including N0 consonant tokens and N1 vowel tokens according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1 .

[0078] Taking the word "I" as an example, assuming that "0" is used as a consonant marker and "1" is used as a vowel marker, the target sequence is: [000000000001111111111], which consists of 11 "0"s and 10 "1"s. At this time, one "0" or one "1" in the target sequence corresponds to one frame, so it can be considered that the target sequence corresponds to 21 frames.

[0079] S203, repeat the target syllable N times to obtain a syllable group.

[0080] It should be noted that in order to let the model know how long it takes to pronounce the currently input syllable (i.e., how many pronunciation frames it takes), this embodiment repeats the syllable for the number of pronunciation frames, thereby obtaining a syllable group including multiple identical syllables. In other words, a syllable is repeated as many times as it is pronounced in the song.

[0081] S204: Input the syllable group into the neural network model so that the neural network model outputs a prediction sequence.

[0082] The predicted sequence includes N3 consonant tags and N4 vowel tags, that is, the total number of consonant tags and vowel tags included in the predicted sequence is also N. Of course, ideally, N3=N0, N4=N1, that is, the predicted sequence also includes N0 consonant tags and N1 vowel tags.

[0083] Taking the word "I" as an example, ideally, the predicted sequence and the target sequence [000000000001111111111] are exactly the same, and the predicted sequence output by the model will always be biased, but the total number of tokens in the predicted sequence must be N. As for how many consonant tokens and vowel tokens there are in the predicted sequence, it depends on the type of phoneme prediction of each frame of pronunciation by the neural network model.

[0084] Among them, the neural network model can predict whether each frame of the target syllable is a vowel or a consonant, and record it with corresponding tags, so as to obtain a predicted sequence.

[0085] S205: Calculate the loss value between the predicted sequence and the target sequence.

[0086] In a specific implementation, calculating the loss value between the predicted sequence and the target sequence includes: comparing the predicted sequence with the target sequence to obtain N comparison results; and determining the loss value based on the N comparison results.

[0087] An example is as follows: The target sequence corresponding to "〔wo〕" is: [0000000000001111111111], assuming that the predicted sequence output by the model is [000010000001111111111], then compare the two "0"s in the first position of the two sequences, and if they are found to be consistent, then record a comparison result; then compare the elements in the second position, third position... until the 21st position of the two sequences respectively, and finally get 21 comparison results, based on which the loss value can be calculated. For example: the ratio of the number of abnormal comparison results (i.e., the comparison results at the fifth position) to the total number of comparison results is determined as the loss value, then the loss value is: 1 / 21. Of course, this method is only an exemplary calculation and does not constitute a limitation to the present application.

[0088] In fact, based on the above-listed loss value calculation ideas, any loss function can be used to calculate the loss value, such as the cross entropy loss function.

[0089] S206. Adjust the parameters of the neural network model based on the loss value to obtain an updated model.

[0090] S207, determine whether the updated model converges; if so, execute S208; if not, execute S209.

[0091] Among them, judging whether the updated model converges includes: judging whether the loss value is small enough; or judging whether the loss value remains unchanged or changes very little during several consecutive iterations.

[0092] S208. Use the updated model as a phoneme pronunciation duration recognition model.

[0093] S209, replacing the neural network model with the updated model, and executing S201 to reacquire another syllable and the label of the syllable from the song syllable set, and iteratively training the updated model until the model converges.

[0094] In this embodiment, the neural network model can be of any structure.

[0095] It can be seen that in this embodiment, the syllables in the song syllable set are used as training samples, and the target syllable, the number of consonant pronunciation frames N0 of the consonant phoneme in the target syllable, and the number of vowel pronunciation frames N1 of the vowel phoneme in the target syllable can be obtained; then, according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, a target sequence including N0 consonant markers and N1 vowel markers is constructed; the target syllable is repeated N times to obtain a syllable group, and the syllable group is input into the neural network model so that the neural network model knows that the target syllable needs to be pronounced for N frame times, and thus a prediction sequence including N markers is output; the loss value between the prediction sequence and the target sequence is calculated; the parameters of the neural network model are adjusted based on the loss value to obtain an updated model; if the updated model converges, the updated model is used as the phoneme pronunciation duration recognition model. It can be seen that this solution uses the syllables in the song syllable set as training samples, and can train a phoneme pronunciation duration recognition model that can constrain the phoneme pronunciation duration. For any syllable, this model can determine how long the vowel is pronounced and how long the consonant is pronounced when the syllable is pronounced in the song. Therefore, this model can be used to label the duration of the phonemes of the syllables in the song, and based on this annotation information, it can be used for singing synthesis.

[0096] Based on the above embodiment, it should be noted that inputting the syllable group into the neural network model so that the neural network model outputs a prediction sequence includes: the neural network model determines the total number of pronunciation frames N of the target syllable based on the repetition times of the target syllable in the syllable group, and predicts the phoneme type of each frame of pronunciation in the target syllable; if the phoneme type of any frame of pronunciation is a consonant, a consonant marker is recorded, and if the phoneme type of any frame of pronunciation is a vowel, a vowel marker is recorded to obtain a prediction sequence.

[0097] Among them, predicting the phoneme type of each frame of pronunciation in the target syllable includes: predicting the phoneme type of each frame of pronunciation in the target syllable by querying the phoneme dictionary. The phoneme dictionary records various vowel phonemes and consonant phonemes in Chinese.

[0098] Among them, predicting the phoneme type of each frame of pronunciation in the target syllable, that is: predicting whether each frame of the target syllable pronounces a vowel or a consonant.

[0099] Based on the above embodiment, it should be noted that there are also single-phoneme syllables in Chinese characters, such as: Wu, Ah, E, etc. Some of these single-phoneme syllables only pronounce vowels, and some only pronounce consonants.

[0100] In a specific implementation manner, if the target syllable is a single-phoneme vowel syllable, after constructing a target sequence including N0 consonant markers and N1 vowel markers according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, it further includes: filling a consonant marker at the beginning of the target sequence; correspondingly, before calculating the loss value between the prediction sequence and the target sequence, it further includes: filling a consonant marker at the beginning of the prediction sequence.

[0101] In the above way, monophonetic vowel syllables can be distinguished, which helps the model output more accurate prediction sequences.

[0102] Of course, in order to make the prediction sequence more accurate, the prediction sequence can be noise-eliminated. For example, the target sequence corresponding to "〔wo〕" is: [0000000000001111111111], assuming that the prediction sequence output by the model is [000010000001111111111], it is obvious that the model has made an incorrect prediction of the phoneme type of the pronunciation of the 5th frame, and it is considered that noise has appeared in the prediction sequence. If the noise can be detected, the "1" in the 5th position of the prediction sequence can be modified to "0" to eliminate the noise. Of course, because the noise is random and not easy to be detected, it is possible to skip noise detection and elimination, and let the model iterate for several times to improve the accuracy of the prediction sequence.

[0103] The noise point detection method may be: if a small number of (such as 1 or 2 or other) vowel marks appear accidentally in the consecutively arranged consonant marks in the prediction sequence, these small number of vowel marks may be considered as noise points. Conversely, if a small number of (such as 1 or 2 or other) consonant marks appear accidentally in the consecutively arranged vowel marks in the prediction sequence, these small number of consonant marks may be considered as noise points.

[0104] See also Figure 4 , Figure 4 A flow chart of a method for marking the pronunciation duration of a phoneme provided in an embodiment of the present application. Figure 4 As shown, the marking method may include the following steps:

[0105] S401: Obtain the text to be annotated.

[0106] S402: Convert the text to be annotated into syllables.

[0107] S403: Input the syllable into the phoneme pronunciation duration recognition model described above, so that the phoneme pronunciation duration recognition model outputs a marked sequence.

[0108] S404: Determine the number of vowel pronunciation frames of the vowel phonemes and the number of consonant pronunciation frames of the consonant phonemes in the syllable based on the marked sequence.

[0109] S405. Record the product of the number of vowel pronunciation frames and the unit frame length as the vowel pronunciation duration, and record the product of the number of consonant pronunciation frames and the unit frame length as the consonant pronunciation duration.

[0110] S406. Mark the vowel pronunciation duration and the consonant pronunciation duration on the syllable.

[0111] In a specific implementation, the method further includes: constructing a set labeled with vowel pronunciation durations and consonant pronunciation durations to train a singing synthesis model using a song syllable set.

[0112] The phoneme pronunciation duration recognition model in this embodiment can determine how many frames a syllable has for vowels and how many frames a syllable has for consonants, and then determine the vowel pronunciation duration and consonant pronunciation duration based on the number of vowel pronunciation frames and the number of consonant pronunciation frames, and mark the corresponding syllables accordingly to obtain syllables with vowel pronunciation duration and consonant pronunciation duration. The set of these syllables can be used as training samples for the singing synthesis model to train the singing synthesis model.

[0113] The following describes the solution provided by the present application through a specific application scenario example, namely: a specific solution for labeling syllables using a phoneme pronunciation duration recognition model. The solution can convert any text into syllables with phoneme pronunciation duration.

[0114] Set up the phoneme pronunciation duration recognition model on the server. Figure 5 The specific process includes:

[0115] S501, terminal requests server;

[0116] S502, the server feeds back a response message to the terminal;

[0117] S503, after receiving the response message, the terminal transmits the text to the server;

[0118] S504, the server converts the characters in the text into syllables;

[0119] S505, the server inputs each syllable into a phoneme pronunciation duration recognition model to obtain each annotation sequence;

[0120] S506: The server processes each marked sequence to determine the number of vowel pronunciation frames of the vowel phoneme and the number of consonant pronunciation frames of the consonant phoneme in each syllable;

[0121] S507: The server calculates the vowel pronunciation duration and the consonant pronunciation duration in each syllable.

[0122] S508, the server marks each syllable and forms a set;

[0123] S509: The server returns the collection and corresponding notification message to the terminal.

[0124] The terminal may be a smart phone, tablet computer, laptop computer, desktop computer or other device.

[0125] The structure of the phoneme pronunciation duration recognition model can be found in Figure 6 .exist Figure 6 In the model, the phoneme pronunciation duration recognition model can be regarded as a phoneme classifier, which can determine whether each frame in a syllable is a consonant or a vowel. Generally, 0.5 is used as the threshold to determine whether it is a consonant or a vowel.

[0126] After a syllable is repeated, enter Figure 6 In the model shown, the syllable will go through: phonemeembedding(390*64)→[conv(3,3,64)→ReLU→pooling(2,2)]*5→GRU→[conv(3,3,64)→ReLU→upsample(2,2)]*5→sigmoid, and finally output a sequence of 0 and 1.

[0127] See also Figure 6 From left to right, the first part is phoneme embedding, the second to sixth parts represent conv, relu, pooling, GUR (Gated Recurrent Unit) is the gated recurrent unit, the eighth to twelfth parts represent conv, relu, upsample, and the thirteenth part is sigmoid. The part before GUR is the encoder, and the part after GUR is the decoder.

[0128] Among them, phoneme embedding can embed syllables into a high-dimensional space. There are 390 syllables in Chinese, and the embedding dimension is 64. conv(3,3,64) means that the width of the convolution kernel is 3, the height is 3, and the number of channels is 128; pooling(2,2) means that the size of the pooling layer is 2, and the stride is 2; upsample uses the bilinear interpolation method. Except for the last layer that uses the sigmoid activation function, the other convolution layers use the ReLU activation function. Figure 6 As shown, the combination of conv and pooling is repeated 5 times, and the combination of conv and upsample is also repeated 5 times. The residual connection method of conv(3,3,64) is used between the encoder and decoder, and the GRU output unit size is 128.

[0129] It should be noted that the parameters such as the convolution kernel size and the number of convolution layers are the experimental preferred values ​​here. Of course, if necessary, these parameters can also be set to other values.

[0130] Figure 6The multi-scale skip convolution used in the algorithm can obtain more and larger receptive fields, that is, it can extract more features. GRU can integrate temporal information (i.e., multi-scale features). The use of these two allows the model to better consider contextual information, improve the accuracy of phoneme duration estimation, and thus improve the expressiveness and naturalness of singing synthesis and cover synthesis.

[0131] Among them, the training set of the above model can be constructed as follows.

[0132] Taking Chinese lyrics as an example, we first extract each Chinese character and the duration of its pronunciation from the lyrics file. For example, "wo" is 208 milliseconds, and "ting" is 416 milliseconds. Then convert the text into syllables (the commonly used syllable representation of Chinese characters is pinyin), so "wo" is [wo] and "ting" is [ting]. According to the duration information, the syllables are rounded to 10 milliseconds per frame, and the syllables are expanded at the frame level, so 208 milliseconds of [wo] is rounded to 21 frames, and 416 milliseconds of [ting] is rounded to 42 frames.

[0133] Use speech annotation tools, such as Figure 3 The praat shown in the figure further decomposes the syllable into multiple phonemes and marks the duration of each phoneme. For example, 〔wo〕 can be divided into 〔_u〕 and 〔uo〕, and 〔ting〕 can be divided into 〔t〕 and 〔ing〕. 〔_u〕 is 112 milliseconds corresponding to 11 frames, and 〔uo〕 is 96 milliseconds corresponding to 10 frames. After the above two steps, syllables and phonemes with duration information can be obtained.

[0134] By decomposing each Chinese character according to the above steps, a song syllable set, namely, a training set, can be obtained. It should be noted that a Chinese character can generally be decomposed into two phonemes, the first phoneme is generally a consonant, and the second phoneme is generally a vowel.

[0135] Assume the input to the model is the three syllables "wo ting dao". After each of these three syllables is repeated according to its total number of pronunciation frames, the following sequence can be obtained: [wo,wo,wo,…,wo,ting,ting,…ting,dao,dao,…,dao]. Among them, the repetition times of these three syllables are 21, 42, and 42 respectively, because the total number of frames of "wo" is 21, the total number of frames of "ting" is 42, and the total number of frames of "dao" is 42. Accordingly, the predicted sequence output by the model can be [0,0,…,0,1,1,…,1,0,0,…,0,1,1,…,1,0,0,…,0,1,1,…,1,]. The first 21 bits in this sequence (i.e., 11 "0"s and 10 "1"s) correspond to the character "wo", the middle 42 bits correspond to the character "ting", and the last 42 bits correspond to the character "dao". In this sequence, there are: 11 "0"s + 10 "1"s + 9 "0"s + 33 "1"s + 9 "0"s + 33 "1"s.

[0136] It can be seen that during the model training or usage process, multiple syllables can be input at one time for the model to process. The purpose of repeating the syllables is to inform the model of the total number of frames of the syllable by the repetition times. The Adam algorithm of the deep neural network can be used in the training process, and the loss function can be the cross-entropy loss function commonly used in the binary classification algorithm.

[0137] This embodiment can parse information such as the total number of frames of syllables and the number of frames of phonemes in syllables from the lyric text. Accordingly, the pronunciation frames occupied by different phonemes in each syllable can be marked, with consonants marked as 0 and vowels marked as 1. Accordingly, a label sequence of a syllable can be obtained. The above-mentioned convolutional neural network (CNN, convolution neural network) takes the frame-level syllables as the input of the model and the frame-level phoneme types as the output for modeling, which can enable the model to generate a sequence composed of 0s and 1s and can play an important role in scenarios such as singing synthesis and cover synthesis.

[0138] It can be seen that the phoneme pronunciation duration recognition model provided by this embodiment can accurately analyze the pronunciation frames of consonants and vowels in each syllable. Accordingly, the corresponding pronunciation duration can be predicted, which significantly improves the naturalness and pronunciation accuracy of singing synthesis and has better advantages in the accuracy and automation of phoneme duration estimation.

[0139] Please refer to Figure 7 , Figure 7 which is a schematic diagram of a model training device provided by an embodiment of the present application, including:

[0140] The syllable acquisition module 701 is used to acquire the target syllable and the label of the target syllable from the song syllable set; the label includes: the consonant pronunciation frame number N0 of the consonant phoneme in the target syllable, and the vowel pronunciation frame number N1 of the vowel phoneme in the target syllable;

[0141] A construction module 702 is used to construct a target sequence including N0 consonant tokens and N1 vowel tokens according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1;

[0142] A repeating module 703 is used to repeat the target syllable N times to obtain a syllable group; N = N0 + N1;

[0143] Prediction module 704, used for inputting the syllable group into the neural network model so that the neural network model outputs a prediction sequence; the prediction sequence includes N3 consonant tokens and N4 vowel tokens; N3+N4=N;

[0144] A calculation module 705 is used to calculate the loss value between the predicted sequence and the target sequence;

[0145] An updating module 706, used to adjust the parameters of the neural network model based on the loss value to obtain an updated model;

[0146] The output module 707 is used to use the updated model as a phoneme pronunciation duration recognition model if the updated model converges.

[0147] In a specific embodiment, it also includes:

[0148] The iteration module is used to reacquire another syllable and the label of the syllable from the song syllable set if the updated model has not converged, so as to iteratively train the updated model until the model converges, and then use the updated model as the phoneme pronunciation duration recognition model.

[0149] In a specific embodiment, the prediction module includes:

[0150] The neural network model determines the total number of pronunciation frames N of the target syllable based on the number of repetitions of the target syllable in the syllable group, and predicts the phoneme type of each pronunciation frame in the target syllable;

[0151] If the phoneme type of any frame pronunciation is a consonant, a consonant mark is recorded; if the phoneme type of any frame pronunciation is a vowel, a vowel mark is recorded to obtain a predicted sequence.

[0152] In a specific implementation, the neural network model predicts the phoneme type of each frame of pronunciation in the target syllable by querying the phoneme dictionary.

[0153] In a specific implementation, the song syllable set is obtained by converting the Chinese lyrics text according to the corresponding song audio.

[0154] In a specific embodiment, it also includes:

[0155] A filling module is used for filling a consonant marker at the first position of the target sequence after constructing a target sequence including N0 consonant markers and N1 vowel markers according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1 if the target syllable is a monophonetic vowel syllable; accordingly, before calculating the loss value between the predicted sequence and the target sequence, it also includes: filling a consonant marker at the first position of the predicted sequence.

[0156] In a specific embodiment, it also includes:

[0157] The optimization module is used to detect and eliminate noise points in the prediction sequence.

[0158] In a specific implementation, the calculation module is specifically used for:

[0159] The predicted sequence is compared with the target sequence to obtain N comparison results; and the loss value is determined based on the N comparison results.

[0160] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0161] It can be seen that the present embodiment provides a model training device, which uses syllables in a song syllable set as training samples, and can train a phoneme pronunciation duration recognition model that can constrain the pronunciation duration of phonemes. For any syllable, the model can determine how long the vowel is pronounced and how long the consonant is pronounced when the syllable is pronounced in a song. Therefore, the model can be used to annotate the duration of the pronunciation phonemes of the syllable in the song, and can be used to synthesize singing based on the annotated information.

[0162] See also Figure 8 , Figure 8 A schematic diagram of a phoneme pronunciation duration marking device provided in an embodiment of the present application includes:

[0163] The text acquisition module 801 is used to acquire the text to be annotated;

[0164] A conversion module 802, used to convert the text to be annotated into syllables;

[0165] Processing module 803, used for inputting syllables into the above-mentioned phoneme pronunciation duration recognition model, so that the phoneme pronunciation duration recognition model outputs a label sequence;

[0166] A determination module 804, configured to determine the number of vowel pronunciation frames of the vowel phonemes and the number of consonant pronunciation frames of the consonant phonemes in the syllable based on the marked sequence;

[0167] The calculation module 805 is used to record the product of the number of vowel pronunciation frames and the unit frame length as the vowel pronunciation duration, and record the product of the number of consonant pronunciation frames and the unit frame length as the consonant pronunciation duration;

[0168] The marking module 806 is used to mark the vowel pronunciation duration and the consonant pronunciation duration on the syllable.

[0169] In a specific embodiment, it also includes:

[0170] The singing synthesis model training module is used to construct a set labeled with vowel pronunciation duration and consonant pronunciation duration, so as to train the singing synthesis model using the song syllable set.

[0171] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0172] It can be seen that the present embodiment provides a device for marking the pronunciation duration of phonemes. The phoneme pronunciation duration recognition model in the device can determine, for any syllable, how long the vowel is pronounced and how long the consonant is pronounced when the syllable is pronounced in a song. Therefore, the model can be used to mark the duration of the pronunciation phonemes of the syllable in the song, and the marked information can be used to synthesize singing.

[0173] Furthermore, the present application also provides an electronic device. The electronic device can be Fig. 9 The server 50 shown may also be Fig.10 Terminal 60 is shown. Fig. 9 and Fig.10 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0174] Fig. 9 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 is used to store a computer program, which is loaded and executed by the processor 51 to implement the relevant steps in the model training method and the phoneme pronunciation duration labeling method disclosed in any of the aforementioned embodiments.

[0175] In this embodiment, the power supply 53 is used to provide working voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 55 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0176] In addition, the memory 52, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 521, a computer program 522 and data 523, etc. The storage method can be temporary storage or permanent storage.

[0177] Among them, the operating system 521 is used to manage and control the hardware devices and computer programs 522 on the server 50 to realize the operation and processing of the data 523 in the memory 52 by the processor 51, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the model training method and the phoneme pronunciation duration marking method disclosed in any of the aforementioned embodiments, the computer program 522 can further include computer programs that can be used to complete other specific tasks. In addition to including application update information, songs, texts, syllables and other data, the data 523 can also include application developer information and other data.

[0178] Fig.10 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 60 may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0179] Generally, the terminal 60 in this embodiment includes: a processor 61 and a memory 62 .

[0180] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0181] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 62 is at least used to store the following computer program 621, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps in the model training method and the phoneme pronunciation duration labeling method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 62 may also include an operating system 622 and data 623, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 622 may include Windows, Unix, Linux, etc. Data 623 may include, but is not limited to, update information of the application.

[0182] In some embodiments, the terminal 60 may further include a display screen 63 , an input / output interface 64 , a communication interface 65 , a sensor 66 , a power supply 67 , and a communication bus 68 .

[0183] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal 60, and the terminal 60 may include more or less components than those shown in the figure.

[0184] Furthermore, the embodiment of the present application also discloses a storage medium, in which computer executable instructions are stored, and when the computer executable instructions are loaded and executed by the processor, the model training method disclosed in any of the above embodiments is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0185] It should be pointed out that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0186] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0187] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: include: Obtaining a target syllable and a label of the target syllable from a song syllable set; The label includes: the number of consonant pronunciation frames N0 of the consonant phonemes in the target syllable, the number of vowel pronunciation frames N1 of the vowel phonemes in the target syllable; According to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, constructing a target sequence including N0 consonant tokens and N1 vowel tokens; Repeat the target syllable N times to obtain a syllable group; wherein N = N0 + N1; Inputting the syllable group into a neural network model so that the neural network model outputs a prediction sequence; the prediction sequence includes N3 consonant tokens and N4 vowel tokens; wherein N3+N4=N; Calculating a loss value between the predicted sequence and the target sequence; Adjusting the parameters of the neural network model based on the loss value to obtain an updated model; If the updated model converges, the updated model is used as the phoneme pronunciation duration recognition model.

2. The method according to claim 1, characterized in that Also includes: If the updated model does not converge, another syllable and the label of the syllable are reacquired from the song syllable set to iteratively train the updated model until the model converges, and the updated model is used as the phoneme pronunciation duration recognition model.

3. The method according to claim 1, characterized in that The step of inputting the syllable group into a neural network model so that the neural network model outputs a prediction sequence comprises: The neural network model determines the total number of pronunciation frames N of the target syllable based on the number of repetitions of the target syllable in the syllable group, and predicts the phoneme type of each frame of pronunciation in the target syllable; If the phoneme type of any frame pronunciation is a consonant, a consonant mark is recorded; if the phoneme type of any frame pronunciation is a vowel, a vowel mark is recorded to obtain the predicted sequence.

4. The method according to claim 3, characterized in that The predicting the phoneme type of each frame of pronunciation in the target syllable comprises: The phoneme type of each frame of pronunciation in the target syllable is predicted by consulting a phoneme dictionary.

5. The method according to any one of claims 1 to 4, characterized in that: If the target syllable is a monophonetic vowel syllable, after constructing a target sequence including N0 consonant markers and N1 vowel markers according to the number of consonant pronunciation frames N0 and the number of vowel pronunciation frames N1, the method further includes: Fill a consonant marker at the first position of the target sequence; Accordingly, before calculating the loss value between the predicted sequence and the target sequence, the method further includes: A consonant token is filled at the first position of the predicted sequence.

6. The method according to any one of claims 1 to 4, characterized in that: Before calculating the loss value between the predicted sequence and the target sequence, the method further includes: Noise points in the predicted sequence are detected and removed.

7. The method according to any one of claims 1 to 4, characterized in that: The calculating the loss value between the predicted sequence and the target sequence includes: Compare the predicted sequence with the target sequence to obtain N comparison results; The loss value is determined based on the N comparison results.

8. A method for marking the pronunciation duration of a phoneme, characterized in that: include: Get the text to be annotated; Convert the text to be annotated into syllables; Inputting the syllable into the phoneme pronunciation duration recognition model according to any one of claims 1 to 7, so that the phoneme pronunciation duration recognition model outputs a labeling sequence; Determine the number of vowel pronunciation frames of the vowel phonemes and the number of consonant pronunciation frames of the consonant phonemes in the syllable based on the marked sequence; The product of the number of vowel pronunciation frames and the unit frame length is recorded as the vowel pronunciation duration, and the product of the number of consonant pronunciation frames and the unit frame length is recorded as the consonant pronunciation duration; The vowel pronunciation duration and the consonant pronunciation duration are marked on the syllable.

9. The method according to claim 8, characterized in that Also includes: A set annotated with the vowel pronunciation duration and the consonant pronunciation duration is constructed to train a singing synthesis model using the song syllable set.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

11. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 9 is implemented.

Images