Model training method, speech synthesis method, device, and computer program product

Abstract

The application relates to the field of intelligent voice, and provides a model training method, a speech synthesis method, equipment and a computer program product. The application can improve the training efficiency of a character phoneme conversion model. The method comprises the following steps: obtaining a text sequence containing target characters with multiple pronunciations and audio materials corresponding to the pronunciations of the text sequence from a training database, extracting a phoneme sequence corresponding to the audio materials by a speech recognition model and taking the phoneme sequence as a phoneme sequence annotation result, inputting the text sequence into a character phoneme conversion model to be trained to obtain a phoneme sequence prediction result output by the character phoneme conversion model, comparing the phoneme sequence annotation result with the phoneme sequence prediction result to obtain a phoneme sequence prediction deviation, adjusting a parameter of the character phoneme conversion model to be trained according to the phoneme sequence prediction deviation, and obtaining the character phoneme conversion model when a training completion condition is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent speech, in particular to a model training method, a speech synthesis method, a device and a computer program product. BACKGROUND

[0002] Speech recognition and speech synthesis are widely used intelligent speech technologies. Speech recognition converts human speech content into text, and speech synthesis is the reverse process, i.e., generating corresponding speech audio from a piece of text.

[0003] For speech synthesis, when a piece of text is input, a speech synthesis system converts it into a phoneme sequence and generates audio based on the phoneme sequence. For multi-pronunciation words, although the text-to-phoneme conversion model in the speech synthesis system can refer to word segmentation results and context information to determine the pronunciation of the text and obtain the corresponding phonemes, it may still make conversion errors.

[0004] To find conversion errors of the text-to-phoneme conversion model and train the model to optimize it, the traditional technology mainly uses an artificial exhaustive method. First, corresponding text is artificially created according to the target text, and the text in the text is manually phoneme-labeled to obtain model training samples, so as to train the model using these samples. However, this method may cause the problem of low model training efficiency and affect the conversion accuracy of the model due to the scarcity of training samples with artificial phoneme labeling. SUMMARY

[0005] Therefore, it is necessary to provide a model training method, a speech synthesis method, a device and a computer program product to solve the above technical problems.

[0006] In a first aspect, the present application provides a model training method. The method comprises:

[0007] obtaining a text sequence containing a target text and audio material corresponding to the pronunciation of the text sequence from a training database; the target text is a text with multiple pronunciations;

[0008] extracting a phoneme sequence corresponding to the audio material by a speech recognition model;

[0009] inputting the text sequence into a text-to-phoneme conversion model to be trained to obtain a phoneme sequence prediction result of the text sequence output by the text-to-phoneme conversion model to be trained;

[0010] comparing the phoneme sequence as the phoneme sequence annotation result corresponding to the text sequence with the phoneme sequence prediction result to obtain a phoneme sequence prediction deviation;

[0011] Adjusting the to-be-trained character phoneme conversion model parameter according to the phoneme sequence prediction bias, and obtaining a character phoneme conversion model when a training completion condition is met.

[0012] In one embodiment, the method further comprises: obtaining, according to a pronunciation dictionary, a plurality of characters of interest having a plurality of pronunciations; for each character of interest, finding a corresponding text sequence of interest; the text sequence of interest contains the corresponding character of interest and is associated with an audio material of interest; the audio material of interest is an audio material corresponding to the pronunciation of the text sequence of interest; and constructing the training database based on the text sequence of interest corresponding to each character of interest and the audio material of interest associated with the text sequence of interest.

[0013] In one embodiment, the method further comprises: obtaining, according to a pronunciation dictionary, a plurality of characters of interest having a plurality of pronunciations; for each character of interest, finding a corresponding text sequence of interest; the text sequence of interest contains the corresponding character of interest and is associated with an audio material of interest; the audio material of interest is an audio material corresponding to the pronunciation of the text sequence of interest; and constructing the training database based on the text sequence of interest corresponding to each character of interest and the audio material of interest associated with the text sequence of interest.

[0014] In one embodiment, the method further comprises: obtaining, according to a pronunciation dictionary, a plurality of characters of interest having a plurality of pronunciations; for each character of interest, finding a corresponding text sequence of interest; the text sequence of interest contains the corresponding character of interest and is associated with an audio material of interest; the audio material of interest is an audio material corresponding to the pronunciation of the text sequence of interest; and constructing the training database based on the text sequence of interest corresponding to each character of interest and the audio material of interest associated with the text sequence of interest.

[0015] In one embodiment, the method further comprises: when the training completion condition is not met, obtaining another text sequence containing the target character and an audio material corresponding to the pronunciation of the another text sequence from the training database, and continuing to train the to-be-trained character phoneme conversion model based on the another text sequence and the audio material corresponding to the pronunciation of the another text sequence.

[0016] In one embodiment, the method further comprises: obtaining, according to a pronunciation dictionary, a plurality of characters of interest having a plurality of pronunciations; for each character of interest, finding a corresponding text sequence of interest; the text sequence of interest contains the corresponding character of interest and is associated with an audio material of interest; the audio material of interest is an audio material corresponding to the pronunciation of the text sequence of interest; and constructing the training database based on the text sequence of interest corresponding to each character of interest and the audio material of interest associated with the text sequence of interest.

[0017] In one embodiment, the phoneme position includes a target phoneme position corresponding to the target character; and the phoneme probability corresponding to the target phoneme position includes a phoneme probability corresponding to the target character under a plurality of pronunciations.

[0018] In a second aspect, the present application also provides a speech synthesis method. The method comprises:

[0019] obtaining a target text; inputting the target text into the phoneme conversion model trained according to the model training method to obtain a phoneme conversion result corresponding to the target text output by the phoneme conversion model; and generating corresponding speech according to the phoneme conversion result.

[0020] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0021] obtaining a text sequence containing a target character and audio material corresponding to pronunciation of the text sequence from a training database; the target character is a character having multiple pronunciations; extracting a phoneme sequence corresponding to the audio material through a speech recognition model; inputting the text sequence into a to-be-trained character phoneme conversion model to obtain a phoneme sequence prediction result corresponding to the text sequence output by the to-be-trained character phoneme conversion model; comparing the phoneme sequence as a phoneme sequence annotation result corresponding to the text sequence with the phoneme sequence prediction result to obtain a phoneme sequence prediction deviation; adjusting parameters of the to-be-trained character phoneme conversion model according to the phoneme sequence prediction deviation, and obtaining a character phoneme conversion model when a training completion condition is met.

[0022] In a fourth aspect, the present application also provides a computer program product. The computer program product comprises a computer program, and the computer program implements the following steps when executed by a processor:

[0023] obtaining a text sequence containing a target character and audio material corresponding to pronunciation of the text sequence from a training database; the target character is a character having multiple pronunciations; extracting a phoneme sequence corresponding to the audio material through a speech recognition model; inputting the text sequence into a to-be-trained character phoneme conversion model to obtain a phoneme sequence prediction result corresponding to the text sequence output by the to-be-trained character phoneme conversion model; comparing the phoneme sequence as a phoneme sequence annotation result corresponding to the text sequence with the phoneme sequence prediction result to obtain a phoneme sequence prediction deviation; adjusting parameters of the to-be-trained character phoneme conversion model according to the phoneme sequence prediction deviation, and obtaining a character phoneme conversion model when a training completion condition is met.

[0024] The aforementioned model training method, speech synthesis method, device, and computer program product obtain text sequences containing target characters with multiple pronunciations and audio materials corresponding to the pronunciations of the text sequences from a training database. The speech recognition model extracts the phoneme sequences corresponding to the audio materials and uses them as phoneme sequence annotation results. The text sequences are then input into the text-to-phoneme conversion model to be trained to obtain its output phoneme sequence prediction results. The phoneme sequence annotation results are then compared with the phoneme sequence prediction results to obtain the phoneme sequence prediction deviation. The parameters of the text-to-phoneme conversion model to be trained are adjusted according to the phoneme sequence prediction deviation. When the training completion conditions are met, the text-to-phoneme conversion model is obtained. This scheme leverages a rich training database of text sequences and their corresponding audio recordings. It efficiently converts text sequences into audio recordings and then into phoneme sequences using a speech recognition model, addressing the scarcity of text-phoneme sequence data for polyphonic characters. The text sequences are then used as input to the model to be trained, and the converted phoneme sequences are used as the phoneme sequence annotation results. Once the training completion conditions are met, a text-phoneme conversion model is obtained. This allows the model to be trained efficiently and fully utilizes the abundant text-phoneme sequences, improving training efficiency. The scheme also automates the entire error correction process, rapidly checking numerous test cases and significantly saving manual intervention and time. This ensures the improved accuracy of the text-phoneme conversion model for polyphonic characters during application. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a model training method in one embodiment;

[0026] Figure 2 This is a flowchart illustrating the steps involved in constructing a training database in one embodiment.

[0027] Figure 3 This is a flowchart illustrating the steps for extracting phoneme sequences in one embodiment;

[0028] Figure 4 This is a flowchart illustrating the process of processing audio materials in one embodiment;

[0029] Figure 5 This is a flowchart illustrating the model training method in an application example.

[0030] Figure 6 This is a flowchart illustrating a speech synthesis method in one embodiment;

[0031] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0032] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0033] The model training method and the speech synthesis method provided by the embodiments of the present application can be applied to computer devices such as terminals and servers. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0034] The model training method and the speech synthesis method provided by the present application will be described below in combination with various embodiments and corresponding drawings.

[0035] In one embodiment, as shown in FIG. 1, a model training method is provided, including the following steps: Figure 1

[0036] Step S101, obtaining a text sequence containing a target character and audio materials corresponding to the pronunciation of the text sequence from a training database.

[0037] ​In this step, the target character refers to a character with multiple pronunciations, and the type of the character can include Chinese characters, English characters, etc. Taking Chinese characters as an example, a multi-syllable character such as “chang” can have pronunciations including “chang (second tone)” and “zhang (third tone)”, and can be used as a target character. The text sequence is a text sequence containing the target character, for example, for the target character “chang”, the text sequence can be “changchun’s snow is very beautiful”. The audio material is an audio corresponding to the pronunciation of the text sequence, for example, for the text sequence “changchun’s snow is very beautiful”, the audio material is the audio corresponding to the pronunciation of “changchun’s snow is very beautiful”, which can be a real human pronunciation audio. The text sequence of the target character and the audio material corresponding to the pronunciation thereof are included in a pre-constructed training database, which can include a set of text sequences of interest and a set of audio materials of interest. The set of text sequences of interest includes a plurality of text sequences of interest, each of which can include at least one or more multi-syllable characters. The set of audio materials of interest includes a plurality of audio materials of interest, each of which corresponds to a text sequence of interest in the set of text sequences of interest, that is, for a text sequence of interest in the set of text sequences of interest, there is an audio material of interest corresponding to the pronunciation thereof in the set of audio materials of interest. Based on this, after determining the target character, the text sequence containing the target character can be obtained from the set of text sequences of interest in the training database, and the audio material corresponding to the pronunciation of the text sequence can be obtained from the set of audio materials of interest in the training database, so that the text sequence containing the target character and the audio material corresponding to the pronunciation thereof can be quickly extracted from a large amount of existing data in the training database.

[0038] In step S102, the phoneme sequence corresponding to the audio material is extracted by the speech recognition model.

[0039] This step is mainly to extract the phoneme sequence corresponding to the audio material obtained in step S101 by using an existing speech recognition model. The phoneme sequence is composed of phonemes recognized by the speech recognition model based on the audio material. For example, the audio material can be the audio corresponding to the text sequence “changchun’s snow is very beautiful”, and the phoneme sequence corresponding to the audio extracted by the existing speech recognition model can be represented as “chang chun de xue henmei”. In practical applications, the corresponding tones of the phonemes in the phoneme sequence can also be identified, such as identifying “chang” as the second tone.

[0040] In step S103, the text sequence is input into the to-be-trained character-phoneme conversion model to obtain a phoneme sequence prediction result corresponding to the text sequence output by the to-be-trained character-phoneme conversion model.

[0041] Step S104, comparing the phoneme sequence as the phoneme sequence annotation result corresponding to the text sequence with the phoneme sequence prediction result to obtain a phoneme sequence prediction deviation;

[0042] In the above steps S103 and S104, the text sequence and the phoneme sequence corresponding to the pronunciation thereof are jointly formed into a model training sample for training of a character phoneme conversion model. The character phoneme conversion model refers to a model for converting characters into corresponding phonemes, which can be a G2P model (grapheme to phoneme). For example, the text sequence is “chang chun de xue hen mei”, and the phoneme sequence corresponding to the pronunciation thereof is “chang chun de xue hen mei”. Thus, the model training sample can be represented as {“chang chun de xue hen mei”, “chang chun de xue hen mei”}. Here, “chang chun de xue hen mei” is the input of the character phoneme conversion model to be trained, and “chang chun de xue hen mei” is the corresponding phoneme sequence annotation result. The text sequence “chang chun de xue hen mei” is input into the character phoneme conversion model to be trained, and the character phoneme conversion model to be trained outputs a corresponding phoneme sequence prediction result. Then, the phoneme sequence prediction result output by the model is compared with the aforementioned phoneme sequence annotation result to obtain a phoneme sequence prediction deviation. The phoneme sequence prediction deviation represents the deviation of the phoneme sequence prediction result output by the character phoneme conversion model during the model training stage relative to the phoneme sequence annotation result.

[0043] Step S105, adjusting the parameters of the character phoneme conversion model to be trained according to the phoneme sequence prediction deviation, and obtaining the character phoneme conversion model when a training completion condition is met.

[0044] This step is mainly to adjust the network parameters of the character phoneme conversion model to be trained according to the deviation of the phoneme sequence prediction result relative to the phoneme sequence annotation result, so that when the character phoneme conversion model after parameter adjustment meets the training completion condition, a trained character phoneme conversion model is obtained. In practical applications, a loss function can be used to calculate the loss value of each training round according to the deviation of the phoneme sequence prediction result relative to the phoneme sequence annotation result, i.e., the phoneme sequence prediction deviation. Based on the loss value of each training round, the parameters of the character phoneme conversion model to be trained are adjusted, and then the next round of training is entered and the loss value of the next round is obtained. The training completion condition can be a loss threshold condition. When the loss value of the current round meets the loss threshold condition (e.g., the loss value of the current round is less than or equal to the loss threshold), it can be determined that the training completion condition is met, and the character phoneme conversion model obtained in the current round of training is taken as the trained character phoneme conversion model.

[0045] The above method, based on rich text sequences and audio materials corresponding to the pronunciation in the training database, efficiently completes the conversion from the text sequence-audio material in the training database to the text sequence-phoneme sequence by means of the speech recognition model, solves the problem of lack of text sequence-phoneme sequence data for multi-pronunciation words, then trains the to-be-trained model by taking the text sequence as the input of the to-be-trained model and taking the converted phoneme sequence as the phoneme sequence labeling result, obtains the character phoneme conversion model when the training completion condition is met, enables the to-be-trained character phoneme conversion model to be efficiently and sufficiently trained based on the rich text sequence-phoneme sequence obtained efficiently, improves the training efficiency of the character phoneme conversion model, can realize full automation of the whole error correction process of the character phoneme conversion model, rapidly checks a large number of test samples, greatly saves manual intervention and time resources, and thus guarantees the improvement of the conversion accuracy of the character phoneme conversion model for multi-pronunciation words in the application stage.

[0046] For the construction of the training database, in one embodiment, as shown in Figure 2 The above method further includes the following steps:

[0047] Step S201, according to the pronunciation dictionary, obtaining a plurality of interested characters with multiple pronunciations.

[0048] In this step, repeated items in the pronunciation dictionary are searched to obtain characters with multiple pronunciations, which are referred to as interested characters. The number of characters with multiple pronunciations searched in the pronunciation dictionary is multiple, thereby obtaining a plurality of interested characters with multiple pronunciations. For example, repeated items in the pronunciation dictionary can obtain a plurality of interested characters with multiple pronunciations such as “long” and “good”.

[0049] Step S202, for each interested character, searching for a corresponding interested text sequence.

[0050] The step S201 obtains each interested word. The step S202 finds the corresponding interested text sequence for each interested word. The interested text sequence has two characteristics. The first one is that it contains the corresponding interested word. For example, the text sequence "the snow in Changchun is beautiful" contains the interested word "Chang". The second one is that it is associated with the interested audio material, which refers to the audio material corresponding to the pronunciation of the interested text sequence. For example, the text sequence "the snow in Changchun is beautiful" is associated with the audio corresponding to the pronunciation of "the snow in Changchun is beautiful". For the acquisition of the interested text sequence, an example can be that the text sequence containing the interested word "Chang" is identified from the subtitles of the audio and video as the interested text sequence, and then the audio corresponding to the interested text sequence is extracted from the audio and video as the interested audio material. In actual application, for each interested word, the corresponding interested text sequence found can include multiple ones. For example, for the interested word "Chang", the corresponding interested text sequence found can include "the snow in Changchun is beautiful" and "Chang is beside the sentry post".

[0051] The step S203 constructs a training database based on the interested text sequence corresponding to each interested word and the interested audio material associated with the interested text sequence.

[0052] Based on the steps S201 and S202, the interested text sequence corresponding to each interested word and the interested audio material associated with the interested text sequence can be obtained. Then, the interested text sequence corresponding to each interested word can be sorted into a set of interested text sequences, and the interested audio material associated with each interested text sequence can be sorted into a set of interested audio materials. Thus, the training database can be constructed based on the set of interested text sequences and the set of interested audio materials and the association between the two sets of text sequences and audio materials. Therefore, the training database constructed by the embodiment can be used to quickly and accurately obtain the required text sequence and the audio material corresponding to the pronunciation of the text sequence after the target word is determined.

[0053] In one embodiment, the step S202 of finding the corresponding interested text sequence for each interested word further includes the following steps.

[0054] The original audio material is obtained. When the original audio material carries the original text sequence corresponding to the pronunciation, the unit sentence containing the corresponding interested word is extracted from the original text sequence as the interested text sequence. The interested audio material is the material segment corresponding to the pronunciation of the unit sentence in the original audio material.

[0055] In this embodiment, the acquisition of the text sequence of interest can be performed on the basis of the obtained original audio material. Specifically, the original audio material can include audio and video with subtitles, storytelling videos, etc., that is, a large number of existing audio data such as audio and video with subtitles, storytelling videos, etc. can be collected as original audio material. Then, if the original audio material carries the original text sequence corresponding to the pronunciation, for example, the original audio material has been annotated with the corresponding subtitles, the annotated subtitles correspond to the original text sequence, in this case, the text sequence of interest can be found and extracted from the original text sequence. Specifically, the unit sentence containing the aforementioned corresponding text of interest is located in the original text sequence, for example, the original text sequence is “Hello, I am a person from Changchun, my name is Zhang San. Today I come here for……”, the text of interest is “long”, and the unit sentence where the text of interest “long” is located is “I am a person from Changchun”. After determining the unit sentence where the text of interest in the original text sequence is located, the material segment corresponding to the pronunciation of the unit sentence can be cut from the original audio material as the audio material of interest according to the corresponding relationship between the original text sequence and the original audio material in time. Through the scheme of this embodiment, the text sequence of interest and its corresponding audio material of interest can be quickly and accurately obtained in the case that the original audio material obtained carries the corresponding original text sequence, and the training database is constructed based on the unit sentence and the material segment corresponding to the pronunciation of the unit sentence, which can make the text sequence and the audio material extracted from the training database in the model training stage meet the requirements of the model training on the length and content of the training sample on the premise of containing the pronunciation corresponding to the target text, and ensure the model training effect.

[0056] In one embodiment, it also includes: in the case that the original audio material does not carry the original text sequence, the original audio material is converted into the corresponding original audio content text, and the original text sequence is obtained according to the original audio content text.

[0057] In the case that the original audio material does not carry the corresponding original text sequence, such as the obtained original audio material is an audio-video without labeled subtitles, the original audio material can be converted into the corresponding original audio content text through a speech recognition model / speech recognition tool. For example, a piece of original audio material is input into a speech recognition model / speech recognition tool, and the original audio content text output by the speech recognition model / speech recognition tool can be "Hello, I am a person from Changchun, my name is Zhang San. The reason why I come here today is…". Then, the original audio content text "Hello, I am a person from Changchun, my name is Zhang San. The reason why I come here today is…" can be directly used as the original text sequence. After obtaining the original text sequence, the method for finding the corresponding text sequence of interest disclosed in the above embodiment can be used to obtain the required text sequence of interest and the corresponding audio material of interest. Thus, the original audio material without labeled subtitles is used to extract the text sequence of interest and the audio material of interest to construct a training database for model training, so as to expand the range of original audio material and enrich the model training samples.

[0058] For training of the character phoneme conversion model, in one embodiment, the above method further includes:

[0059] When the training completion condition is not met, another text sequence containing the target character and the audio material of another text sequence corresponding to the pronunciation are obtained from the training database, and the training of the to-be-trained character phoneme conversion model is continued based on the another text sequence and the audio material of another text sequence corresponding to the pronunciation.

[0060] In this embodiment, the training completion condition can be a loss threshold condition under a certain number of model parameter adjustments. Specifically, in the foregoing embodiment, the model parameters are adjusted during the training process of the text phoneme conversion model to be trained using the text sequence containing the target character and the corresponding audio material. After adjustment, the text sequence and the audio material can be used again to train the model after parameter adjustment and determine whether the loss threshold condition is met. If not, continue to adjust the parameters and continue training. However, when the parameter adjustment for the model reaches a certain number of times and still does not meet the loss threshold condition, it is determined that it belongs to the case of not meeting the training completion condition in this embodiment. At this time, another text sequence containing the target character and the audio material corresponding to the pronunciation of the another text sequence are obtained from the training database. For example, the target character is "long", the original training uses "the snow in Changchun is very beautiful" and the audio material corresponding to the pronunciation to train the model. At this time, "long in the sentry post" or "long gate again misjudges the period of sound clear and sad" is obtained from the training database as another text sequence and the audio material corresponding to the pronunciation of the another text sequence. Then, the newly obtained "long in the sentry post" or "long gate again misjudges the period of sound clear and sad" and the audio material corresponding to the pronunciation are used to continue training the text phoneme conversion model to be trained, to ensure the accuracy and effectiveness of the model training.

[0061] For the extraction of the phoneme sequence, in one embodiment, as shown in Figure 3 the step S102 of extracting the phoneme sequence corresponding to the audio material by the speech recognition model includes:

[0062] Step S301, determining the phoneme probability corresponding to each phoneme position in the audio material;

[0063] Step S302, selecting the highest probability phoneme path based on the phoneme probability corresponding to each phoneme position; the phoneme path is composed of each phoneme position in sequence;

[0064] Step S303, obtaining the phoneme sequence according to the highest probability phoneme path.

[0065] The foregoing steps S301 to S303 in this embodiment can be executed by the speech recognition model, in combination with Figure 4A description will be given. Specifically, in step S301, the phoneme probabilities corresponding to each phoneme position in the audio material can be recognized by the acoustic model in the speech recognition model. For example, for the audio material corresponding to the text "Changchun...", the acoustic model can recognize the phoneme probabilities corresponding to the phoneme positions where "chang / zhang" and "chun" are located. In this regard, in some embodiments, the phoneme position includes the target phoneme position corresponding to the above target text, and the phoneme probability corresponding to the target phoneme position includes the phoneme probabilities corresponding to the target text under various pronunciations. Exemplarily, for the target text "长", the target phoneme position is the phoneme position where "chang / zhang" is located, and the phoneme probability corresponding to the target phoneme position includes the phoneme probabilities corresponding to the target text "长" under the two pronunciations of "chang" and "zhang" respectively. As Figure 4 shown, the phoneme probability corresponding to the target text "长" under the pronunciation of "chang" is 9.7, and the phoneme probability corresponding to the pronunciation of "zhang" is 0.5. The solution of this embodiment can obtain the phoneme sequences of accurate pronunciations of various polyphonic characters for the training and optimization of the model for polyphonic character conversion.

[0066] For example Figure 4 in, step S301 can specifically obtain that the phoneme probabilities corresponding to the first phoneme position (the phoneme position where "chang / zhang" is located) include 9.7 and 0.5, and the phoneme probability corresponding to the second phoneme position (the phoneme position where "chun" is located) is 9.8, and so on. Step S302 is mainly to find the phoneme path with the highest probability. Among them, the phoneme path is a path composed of each phoneme position in sequence. For example Figure 4 in, the first phoneme position, the second phoneme position,... can form a phoneme path in sequence, but the first phoneme position corresponds to phoneme probabilities 9.7 and 0.5, that is, each phoneme position may correspond to one or more phoneme probabilities. Step S302 combines the phoneme probabilities corresponding to each phoneme position and finds the phoneme path with the highest probability as a whole. In practical applications, the Viterbi algorithm can be used to calculate the phoneme path with the highest probability. For example<​​​​​

[0068] In one application example, referring to Figure 5 , specifically, the repeated item such as the polyphonic word "long" can be found from the pronunciation dictionary first, the pronunciation of which can include "chang" and "zhang", and the polyphonic word "long" can be taken as the text of interest, and then the corresponding text sequence of interest "Changmen you misjudged the best period of sound varnish", "Long in the sentry post" and "The snow in Changchun is very beautiful" and the corresponding audio materials of interest are added to the training database, and these audio materials of interest are real human audio rather than machine-synthesized audio to ensure the effectiveness of model training. Based on this, in the model training stage, the polyphonic word "long" can be taken as the target word, and the text sequence such as "Changmen you misjudged the best period of sound varnish", "Long in the sentry post", "The snow in Changchun is very beautiful" and the corresponding audio materials of interest can be obtained from the training database. These audio materials can be taken as real human audio for phoneme sequence extraction, and the phoneme sequence corresponding to the audio material in accordance with the real human reading method such as "chang men you……", "zhang zai shao……" and "chang chun de……" and the like can be extracted by the speech recognition model as the phoneme sequence labeling result. Then the aforementioned text sequence such as "Changmen you misjudged the best period of sound varnish", "Long in the sentry post", "The snow in Changchun is very beautiful" can be input into the text-to-phoneme conversion model (G2P model) to be trained, and the text-to-phoneme conversion model to be trained outputs the phoneme sequence prediction result such as "zhang men you……", "zhang zai shao……" and "chang chun de……" and the like. Then the phoneme sequence prediction result is compared with the aforementioned corresponding phoneme sequence labeling result to find the polyphonic word error, and the phoneme sequence prediction deviation is obtained. Then the model parameter of the text-to-phoneme conversion model is adjusted according to the phoneme sequence prediction deviation, and the trained text-to-phoneme conversion model is obtained when the training completion condition is met.

[0069] The application example can quickly obtain a large amount of text-phoneme sequence as model training sample based on the existing massive and rich audio database by means of speech recognition technology, solve the problem of lack of text-phoneme data by means of rich existing audio data, apply the model training sample to the training of the text-to-phoneme conversion model, obtain the prediction deviation by comparing the phoneme sequence as the labeling result with the phoneme sequence prediction result, adjust the model parameter, realize the full automation of the whole process of the text-to-phoneme conversion model correction, and eliminate the manual intervention, so as to quickly check a large number of test samples, greatly improve the efficiency of the correction of the text-to-phoneme conversion model, greatly save the manual intervention and checking time, and quickly optimize the accuracy of the model conversion.

[0070] In one embodiment, a speech synthesis method is also provided, as shown in Figure 6 , the method comprises:

[0071] Step S601, obtaining a target text;

[0072] Step S602, inputting the target text into the character-to-phoneme conversion model trained according to the model training method as described in the above embodiments to obtain a phoneme conversion result corresponding to the target text output by the character-to-phoneme conversion model;

[0073] Step S603, generating a corresponding speech according to the phoneme conversion result.

[0074] The present embodiment is an application of the character-to-phoneme conversion model obtained based on the model training method as described in the above embodiments. Specifically, the target text can be a piece of character content input by a user, or a piece of character content that needs to be fed back to the user. The piece of character content is input into the character-to-phoneme conversion model, and the character-to-phoneme conversion model converts the piece of character content into a corresponding phoneme sequence, i.e., a phoneme conversion result corresponding to the target text, and finally generates an audio based on the phoneme conversion result for playing. The character-to-phoneme conversion model of the present embodiment is trained based on the model training method as described above, and has accurate phoneme conversion capability for multi-sound characters. Therefore, even if the target text has multi-sound characters, the present embodiment can still accurately generate a speech corresponding to the target text for playing by means of the character-to-phoneme conversion model, thereby improving the speech conversion and synthesis capability.

[0075] It should be understood that although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0076] In one embodiment, a computer device, which can be a server, is provided, and its internal structure diagram can be as shown in Figure 7As shown in the figure. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store text sequences, audio materials and other data. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a model training method and a speech synthesis method.

[0077] Those skilled in the art can understand that, Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0078] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the method embodiments described above.

[0079] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps in each of the method embodiments described above.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0081] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0082] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a text sequence containing a target character and audio material corresponding to pronunciation of the text sequence from a training database; the target character is a character with multiple pronunciations; extracting a phoneme sequence corresponding to the audio material through a speech recognition model; inputting the text sequence into a to-be-trained character-phoneme conversion model to obtain a phoneme sequence prediction result of the text sequence output by the to-be-trained character-phoneme conversion model; comparing the phoneme sequence as a phoneme sequence annotation result corresponding to the text sequence with the phoneme sequence prediction result to obtain a phoneme sequence prediction deviation; adjusting parameters of the to-be-trained character-phoneme conversion model according to the phoneme sequence prediction deviation, and obtaining a character-phoneme conversion model when a training completion condition is met; The method further comprises: finding a duplicate in a pronunciation dictionary to obtain multiple characters of interest with multiple pronunciations; obtaining original audio material; under the condition that the original audio material carries an original text sequence corresponding to pronunciation, extracting a unit sentence containing corresponding characters of interest from the original text sequence as a corresponding text sequence of interest, and extracting a material segment corresponding to pronunciation of the unit sentence from the original audio material as audio material of interest; wherein the text sequence of interest contains corresponding characters of interest and is associated with audio material of interest; based on the text sequence of interest corresponding to each character of interest and the audio material of interest associated with the text sequence of interest, constructing the training database.

2. The method of claim 1, wherein, Further comprising: under the condition that the original audio material does not carry the original text sequence, converting the original audio material into corresponding original audio content text, and obtaining the original text sequence according to the original audio content text.

3. The method of claim 1, wherein, The method further comprises: when the training completion condition is not met, obtaining another text sequence containing the target character and audio material corresponding to pronunciation of the another text sequence from the training database, and continuing to train the to-be-trained character-phoneme conversion model based on the another text sequence and the audio material corresponding to pronunciation of the another text sequence.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: determining a phoneme probability corresponding to each phoneme position in the audio material; based on the phoneme probability corresponding to each phoneme position, selecting a phoneme path with the highest probability; the phoneme path is composed of the phoneme positions in sequence; obtaining the phoneme sequence according to the phoneme path with the highest probability.

5. The method of claim 4, wherein, The phoneme position includes a target phoneme position corresponding to the target character; the phoneme probability corresponding to the target phoneme position includes a phoneme probability corresponding to the target character under multiple pronunciations.

6. A speech synthesis method characterized by, The method comprises: obtaining a target text; inputting the target text into a character-phoneme conversion model trained by the model training method according to any one of claims 1 to 5 to obtain a phoneme conversion result of the target text output by the character-phoneme conversion model; generating corresponding speech according to the phoneme conversion result. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

8. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images