Speech model training method, speech generation method, device, and storage medium

Abstract

The application relates to a speech model training method, a speech generation method, computer equipment and a storage medium. Text structure information and phoneme information of text sample data are input into a speech model, the speech model encodes the text structure information and the phoneme information respectively, predicted speech features are obtained based on text coding data obtained through the encoding, model parameters are adjusted according to the similarity of the predicted speech features and reference speech features, and a trained speech model is obtained when a condition is met; the speech model outputs corresponding predicted speech based on text structure information and phoneme information in target text input by a user. Compared with traditional speech obtained through speech synthesis, the scheme combines text structure information and phoneme information to generate speech, so that the generated speech tone pause is more accurate and more natural, and the naturalness of speech generation is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for training a speech model, a speech generation method, a computer device, a storage medium, and a computer program product. Background Technology

[0002] With the rapid development of deep learning and the explosive growth of hardware computing power, it is now possible to generate corresponding speech using technologies such as speech synthesis to simulate the effect of real people speaking. However, the generated speech currently still sounds unnatural compared to the intonation and rhythm of real speech.

[0003] Therefore, current speech generation methods suffer from the drawback of producing unnatural speech. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method for a speech model, a speech generation method, a computer device, a computer-readable storage medium, and a computer program product that can improve the naturalness of the generated speech, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training a speech model, the method comprising:

[0006] Obtain text sample data and the speech features of the target speech corresponding to the text sample data;

[0007] The text structure information and phoneme information of the text sample data are obtained, and the text structure information and phoneme information are input into the speech model to be trained. The text encoder of the speech model obtains text encoded data based on the text structure information and phoneme information, and the decoder of the speech model obtains predicted speech features based on the text encoded data.

[0008] The speech feature encoding data of the speech features is obtained by the speech encoder, and the speech feature encoding data is input into the speech model. The decoder obtains the reference speech features based on the speech feature encoding data.

[0009] Based on the similarity between the predicted speech features and the reference speech features, the model parameters of the speech model to be trained are adjusted until the preset training termination condition is met, thus obtaining the trained speech model.

[0010] In one embodiment, obtaining the text structure information of the text sample data includes:

[0011] Based on the sorting information of each character in the text sample data, the character information of the text sample data is obtained;

[0012] Based on the word component information corresponding to the text sample data, determine the syntactic information of the text sample data;

[0013] Based on the character information and the syntactic information, the text structure information is obtained.

[0014] In one embodiment, the text structure information includes character information and syntactic information; obtaining text encoded data based on the text structure information and phoneme information includes:

[0015] Based on the phoneme information corresponding to each character in the character information, the first text encoding data is obtained;

[0016] Based on the syntactic information, determine the word attributes of each word in the word information;

[0017] Based on the word attributes of each word, the second text encoding data is obtained;

[0018] Based on the text sample data, determine the predicted start and end times of the phoneme information;

[0019] The text encoding data is obtained based on the first text encoding data, the second text encoding data, and the start and end time prediction data of the phoneme information.

[0020] In one embodiment, the method further includes:

[0021] Obtain pre-annotated continuous word and sentence information from the text sample data; the pause time between each character in the continuous word and sentence information is less than a preset threshold;

[0022] The continuous word and sentence information is input into the speech model to be trained, and the speech model determines the start and end time prediction data of each phoneme in the phoneme information based on the continuous word and sentence information.

[0023] In one embodiment, obtaining the pre-annotated continuous word and phrase information in the text sample data includes:

[0024] The text sample data is input into the trained word segmentation model, and the pre-annotated continuous word and sentence information is obtained based on the word segmentation results corresponding to the text sample data output by the word segmentation model.

[0025] And / or,

[0026] Obtain the maximum pause time threshold between each group of adjacent characters in the text sample data, and obtain pre-annotated continuous word and sentence information based on adjacent characters whose maximum pause time threshold is less than the preset threshold;

[0027] And / or,

[0028] Obtain preset words and phrases for the input text sample data, and obtain pre-annotated continuous word and phrase information based on the preset words and phrases.

[0029] Secondly, this application provides a speech generation method, the method comprising:

[0030] Obtain the target text, and obtain the text structure information and phoneme information of the target text;

[0031] The target text is input into a trained speech model. The text encoder of the speech model obtains text encoding data based on the text structure information and phoneme information of the target text. The decoder of the speech model obtains predicted speech features based on the text encoding data. The predicted speech features are input into the vocoder in the speech model. The vocoder outputs the corresponding predicted speech based on the predicted speech features. The speech model is trained according to the above method.

[0032] The speech corresponding to the target text is obtained based on the predicted speech.

[0033] In one embodiment, obtaining the target text includes:

[0034] Get the raw text input by the user;

[0035] Obtain continuous word and phrase information from the original text;

[0036] The target text is obtained from the original text carrying the information of the continuous words and phrases.

[0037] In one embodiment, obtaining continuous word and phrase information from the original text includes:

[0038] The original text is input into the trained word segmentation model, and continuous word and sentence information is obtained based on the word segmentation results corresponding to the original text output by the word segmentation model.

[0039] And / or,

[0040] Obtain the maximum pause time threshold between each group of adjacent characters in the original text, and obtain continuous word and sentence information based on adjacent characters whose maximum pause time threshold is less than the preset threshold;

[0041] And / or,

[0042] Obtain preset words and phrases for the original text input, and obtain continuous word and phrase information based on the preset words and phrases.

[0043] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0044] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0045] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0046] The aforementioned speech model training method, speech generation method, computer equipment, storage medium, and computer program product input text structure information and phoneme information from text sample data into the speech model. The speech model encodes the text structure information and phoneme information respectively, obtains predicted speech features based on the encoded text data, and adjusts the model parameters according to the similarity between the predicted speech features and reference speech features until a trained speech model is obtained. The speech model then outputs the corresponding predicted speech based on the text structure information and phoneme information in the target text input by the user. Compared to traditional speech synthesis, this scheme combines text structure information and phoneme information for speech generation, resulting in more accurate and natural pauses and intonation in the generated speech, thus improving the naturalness of the generated speech. Attached Figure Description

[0047] Figure 1 This is a diagram illustrating the application environment of a speech model training method in one embodiment.

[0048] Figure 2 This is a flowchart illustrating the training method for a speech model in one embodiment;

[0049] Figure 3 This is a flowchart illustrating the pause control steps in one embodiment;

[0050] Figure 4 This is a flowchart illustrating a speech generation method in one embodiment;

[0051] Figure 5 This is a flowchart illustrating the training method for a speech model in another embodiment;

[0052] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The speech model training method and speech generation method provided in this application embodiment can be applied to, for example... Figure 1 The application environment is illustrated. The terminal communicates with the server via a network. The data storage system stores the data the server needs to process. This system can be integrated onto the server, or it can be hosted in the cloud or on other network servers. The server trains a speech model based on text sample data and the corresponding target speech, resulting in a trained speech model. The terminal receives the user's input target text and sends it to the server. The server then outputs the corresponding predicted speech based on the trained speech model and the target text, and sends it back to the terminal. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0055] In one embodiment, such as Figure 2 As shown, a method for training a speech model is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0056] Step S202: Obtain the text sample data and the speech features of the target speech corresponding to the text sample data.

[0057] The text sample data can be pre-stored in a server database and used to train the speech model. The text sample data can be determined based on text input by the user at historical times. The speech model can then be used to output corresponding speech based on the text.

[0058] The server can also acquire the target speech corresponding to the text sample data and obtain the speech features of the target speech. The target speech can be speech that conforms to natural intonation, pauses, and speaking habits corresponding to the text sample data. This target speech can be speech pre-generated manually based on the text sample data, or it can be the correct output speech corresponding to the text sample data during historical speech generation. The server can extract the speech features of the target speech using machine learning.

[0059] During the training phase, the server can combine text sample data and the speech features of the corresponding target speech to train the speech model. This enables the speech model to build an automatic mapping capability, obtaining the mapping relationship between text and its corresponding speech. This mapping relationship includes pause relationships between words. Thus, the trained speech model can output speech with natural pauses and consistent with natural speaking habits based on the input text.

[0060] Step S204: Obtain the text structure information and phoneme information of the text sample data, input the text structure information and phoneme information into the speech model to be trained, the text encoder of the speech model obtains the text encoded data based on the text structure information and phoneme information, and the decoder of the speech model obtains the predicted speech features based on the text encoded data.

[0061] The text sample data can include various types of information. The server can obtain the text structure information and phoneme information of the text sample data. The text structure information represents the information about each character in the text sample data and the relationships between them. This text structure information can include various types of information, such as character information and syntactic information. Character information represents each character in the text sample data, while syntactic information represents the attributes of words and sentences. Therefore, the server can determine the individual characters in the text sample data and the relationships between them based on the text structure information. The server can also obtain the phoneme information corresponding to the text sample data, such as the phonemes and tones of each character in the text sample data.

[0062] The server can input the acquired text structure information, phoneme information, and speech feature information into the speech model to be trained. The speech model can encode the text structure information and phoneme information respectively, and obtain predicted speech features based on the encoded text data. The speech model can include a text encoder and a decoder. The text encoder can correspond to the information input into the speech model; for example, text structure information corresponds to a text structure encoder, and phoneme information corresponds to a phoneme encoder. The speech model encodes the corresponding information using its respective text encoder to form encoded text data. The speech features can be input into the decoder, so that the speech model, using the speech features as a reference, generates corresponding predicted speech features using the encoded text data.

[0063] The aforementioned text structure information and phoneme information can be obtained through corresponding methods. For example, in one embodiment, for text structure information, the server can obtain the character information of the text sample data based on the ordering information of each character in the text sample data. Furthermore, the server can also determine the syntactic information of the text sample data based on the word component information corresponding to the text sample data. Thus, the server obtains the text structure information based on the character information and syntactic information. For phoneme information, the server can extract it from the text sample data using tools.

[0064] Specifically, phoneme information can include phonemes and tones, which the server can extract from text sample data using tools such as jieba. Phoneme formats can take various forms, such as the International Phonetic Alphabet (IPA), initials, and finals in Pinyin. Tones refer to the four initials (high level, rising, falling-rising, falling, and neutral). Each pronunciation in the target speech can correspond to a specific phoneme. The server can pre-align the phoneme information with the target speech, for example, through automatic alignment or manual annotation, to obtain the start and end times of each phoneme.

[0065] In a text encoder, the server can train a speech model to predict the start and end times of speech corresponding to phoneme information. For example, the speech model predicts the start and end times of speech based on phoneme information, obtains the predicted start and end times, and then adjusts the model parameters based on phoneme prediction by comparing the similarity of the predicted start and end times with the pre-aligned start and end times. This process is iterated until the similarity is greater than or equal to a preset start and end time similarity threshold within a preset number of training iterations, or when the preset number of training iterations is reached. At this point, the training of the speech model to predict the start and end times of speech corresponding to phonemes is considered complete.

[0066] For character information, this can include information about each character arranged sequentially in the input text sample data. For example, the sentence "China's history and culture are long and profound, with a splendid civilization of five thousand years" contains character information. When training the speech model on the text structure information, the text encoder can jointly model the phoneme information corresponding to each character, fusing coarse-grained character information with fine-grained phoneme information.

[0067] Syntactic information can include word and sentence attributes, such as subject, verb, object, nouns, verbs, and adjectives. The server can perform natural language processing on the input text sample data to obtain syntactic dependency parsing results. For example, the server can use tools such as hanlp (Han Language Processing), StanfordNLP (Stanford Natural Language Processing), and LTP (Language Technology Platform) to extract syntactic information from the text sample data. The server can then combine this higher-dimensional syntactic information, character information, phoneme information, and speech features to generate corresponding predicted speech features. The speech model to be trained is then trained based on these predicted speech features.

[0068] Step S206: Obtain speech feature encoding data of speech features through speech encoder, input speech feature encoding data into speech model, and obtain reference speech features by decoder based on speech feature encoding data.

[0069] Both the predicted speech features and the speech features of the target speech can be a spectrum, such as a Mel spectrogram. After the speech model generates the predicted speech features through the decoder, it can compare the predicted speech features with the reference speech features of the target speech. The encoder in the server can include a speech encoder and a text encoder. The speech features of the target speech can be encoded by the speech encoder and then decoded by the decoder to obtain the reference speech features.

[0070] Step S206: Adjust the model parameters of the speech model to be trained according to the similarity between the predicted speech features and the reference speech features until the preset training termination condition is met, and obtain the trained speech model.

[0071] The server can adjust the model parameters based on the comparison between the reference speech features and the predicted speech features. For example, the server inputs the predicted speech features and the target speech features into a preset loss function, and determines the similarity between the predicted speech features and the predicted speech features by detecting whether the function value of the loss function converges. The server adjusts the model parameters of the speech model to be trained based on the similarity. It then returns to the step of inputting text structure information and phoneme information into the speech model to be trained for the next training iteration until a preset training termination condition is met. At this point, the server can stop training and obtain the trained speech model. The preset training termination condition can include various factors, such as the similarity being greater than or equal to a preset similarity threshold within a preset number of training iterations, or the number of training iterations reaching a preset number of training iterations.

[0072] In addition, in some embodiments, the above-mentioned speech model also includes a vocoder. After the speech model generates predicted speech features, the corresponding predicted speech can be generated and output in the vocoder based on the predicted speech features, thereby realizing text-based speech generation. Specifically, the server trains the speech model by inputting the above-mentioned character information, syntactic information, and phoneme information. This provides not only the speech segment information corresponding to each phoneme, which can train the model's ability to predict the start and end times of phonemes, but also the corresponding character information and the relationship information between characters. This trains the model's ability to generate predicted speech that conforms to the text and the ability to predict the pause relationships between characters in the predicted speech, so that the generated speech conforms to the rules of human speech.

[0073] In the training method of the aforementioned speech model, the text structure information and phoneme information of the text sample data are input into the speech model. The speech model encodes the text structure information and phoneme information respectively, and predicts speech features based on the encoded text data. The model parameters are adjusted according to the similarity between the predicted speech features and the reference speech features until the conditions are met to obtain the trained speech model. The speech model outputs the corresponding predicted speech based on the text structure information and phoneme information in the target text input by the user. Compared with the speech obtained by traditional speech synthesis, this scheme combines text structure information and phoneme information for speech generation, making the generated speech more accurate and natural in tone and pauses, thus improving the naturalness of speech generation.

[0074] In one embodiment, obtaining text-encoded data based on text structure information and phoneme information includes: obtaining first text-encoded data based on character information and phoneme information; obtaining second text-encoded data based on syntax information and character information; and obtaining text-encoded data based on start and end time prediction data of the first text-encoded data, the second text-encoded data, and phoneme information.

[0075] In this embodiment, the aforementioned text structure information may include character information and syntactic information in the text. The server can use a text encoder in the speech model to encode the character information, syntactic information, and phoneme information. The text encoder can be of various types, such as character information text encoders, syntactic text encoders, and phoneme text encoders, and the speech model can use different text encoders to encode the corresponding types of information.

[0076] During the encoding process, the aforementioned text structure information and phoneme information can influence each other. For example, the server obtains the first text encoded data based on the aforementioned character information and phoneme information using a speech model. The server obtains the second text encoded data based on the aforementioned syntactic information and character information using a speech model. Furthermore, the server can also determine the start and end time prediction data for phoneme information using a speech model. For instance, by training the speech model to match phoneme information with target speech, the speech model identifies the start and end times of each phoneme in the target speech based on the target speech, and then predicts the start and end time prediction data for the phoneme information. Thus, for a trained speech model, the server can predict the corresponding start and end time prediction data based on the input phoneme information.

[0077] The first and second text encoded data can be encoded in different ways. For example, in one embodiment, the server can obtain the phoneme information corresponding to each character in the above-mentioned character information using a speech model. That is, the speech model matches the character information with the phoneme information and encodes the first text encoded data according to the phoneme information corresponding to each character. In another embodiment, the server can use the speech model to determine the word attributes of each word in the above-mentioned character information based on the above-mentioned syntactic information, such as subject-verb-object, noun, verb, adjective, etc. The speech model encodes the second text encoded data according to the word attributes of each word.

[0078] In this context, the first text-encoded data, the second text-encoded data, and the start-end time prediction data of phoneme information can all be vector data. The server uses a speech model to obtain text-encoded data based on the first text-encoded data, the second text-encoded data, and the start-end time prediction data of phoneme information. Specifically, the server uses a speech model to align phoneme information with corresponding speech, obtaining start-end time prediction data for each phoneme. The speech model then jointly models the phoneme information corresponding to each character, fusing coarse-grained characters with fine-grained phonemes for the first encoding. The speech model then jointly models syntactic information with character information and phoneme information for the second encoding. Thus, the speech model can obtain speech segment information corresponding to each phoneme, character information corresponding to each phoneme, and the relationship information between characters based on text sample data. The server can input the various vector data obtained from the above encoding into a decoder. In the decoder, targeting the speech features of the target speech, the server uses the first text-encoded data, the second text-encoded data, and the start-end time prediction data of phoneme information to generate the spectrum of the corresponding predicted speech features. The speech model is trained by comparing the speech features of the target speech with the predicted speech features.

[0079] Through the above embodiments, the server encodes various text structure information and phoneme information using a speech model, generates predictive speech features based on the text encoding data, and trains the speech model so that the speech generated by the trained speech model is more in line with the rules of human speech, thereby improving the naturalness of speech generation.

[0080] In one embodiment, the method further includes: acquiring pre-annotated continuous word and sentence information from text sample data; ensuring that the pause time between each character in the continuous word and sentence information is less than a preset threshold; inputting the continuous word and sentence information into the speech model to be trained, and having the speech model determine the start and end time prediction data of each phoneme in the phoneme information based on the continuous word and sentence information.

[0081] In this embodiment, the aforementioned text sample data contains predetermined pause relationships, such as pause relationships determined by punctuation marks. The text between each two pauses can belong to continuous word / sentence information; that is, within continuous word / sentence information, the pauses between characters cannot be too long, for example, the pause time between characters in continuous word / sentence information is less than a preset threshold. Since the pause relationships required for a word vary in different contexts, there may be errors in the pause relationships in the aforementioned text sample data. The server can pre-acquire the annotation of continuous word / sentence information in the text sample data, thereby determining the start and end time prediction data of each phoneme in the phoneme information based on the text sample data.

[0082] After the server obtains pre-annotated continuous word and sentence information from the text sample data, it can input this information into the speech model to be trained. The speech model then determines the start and end time prediction data for each phoneme in the phoneme information based on the continuous word and sentence information. For example, based on the aforementioned continuous word and sentence information, the speech model determines which pause times between words in the text sample data need to be less than a preset threshold, and further determines which phoneme start and end time intervals need to be less than the preset threshold. Under these conditions, the speech model generates the start and end time prediction data for each phoneme in the phoneme information.

[0083] In this embodiment, the server can pre-annotate the input text sample data with continuous words and phrases, so that the speech model can combine the correct pause relationships to generate predictive speech features. This makes the predicted speech output by the trained speech model conform to the rules of human speech and improves the naturalness of speech generation.

[0084] In one embodiment, obtaining pre-annotated continuous word and sentence information in text sample data includes: inputting the text sample data into a trained word segmentation model, and obtaining pre-annotated continuous word and sentence information based on the word segmentation results corresponding to the text sample data output by the word segmentation model.

[0085] In this embodiment, there are several ways to pre-annotate continuous word and sentence information in text sample data. For example, the server can perform word segmentation on the text sample data. The server can input the text sample data into a trained word segmentation model, and the word segmentation model will output the corresponding word segmentation result for the text sample data. Based on the word segmentation result, the server obtains the pre-annotated continuous word and sentence information.

[0086] The server can also determine continuous word and phrase information by controlling the pause duration. For example, in one embodiment, the server can obtain the maximum pause time threshold between each group of adjacent characters in the text sample data. This maximum pause time threshold can be preset by the relevant personnel. The server can detect the maximum pause time threshold between each group of adjacent characters and obtain pre-labeled continuous word and phrase information based on adjacent characters whose maximum pause time threshold is less than the preset threshold.

[0087] The server can also determine continuous word and phrase information by setting preset phrases. For example, in one embodiment, the server can obtain preset phrases for text sample data input, which can be pre-input by the user or relevant personnel, and the server can obtain pre-annotated continuous word and phrase information based on the preset phrases.

[0088] Specifically, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating the pause control steps in one embodiment. The server can pre-determine the continuous word / phrase information of the input text, and then the speech model processes the annotated text. The continuous word / phrase information can be determined in various ways. For example, the server can determine the continuous word / phrase information using a word segmentation model to achieve pause control. That is, the server can identify the input text through the word segmentation model, combining elements that may become words or phrases. The start and end times of phonemes within these words or phrases need to be limited, for example, within the aforementioned preset threshold, to avoid excessively long pauses. The server can also control pauses using a duration threshold; that is, the server can set a maximum pause duration between characters, such as the aforementioned preset threshold; thus, the server can prevent long pauses within a single sentence. Alternatively, the server can control pauses using preset word / phrase information. That is, when inputting text, the server simultaneously inputs the desired continuous words or phrases to form continuous word / phrase information, so that the speech model can adjust the corresponding speech pauses based on the annotated continuous word / phrase information to avoid excessively long pauses.

[0089] The server can determine continuous word and sentence information based on any one of the above methods, or it can determine continuous word and sentence information based on a combination of the above methods.

[0090] Through this embodiment, the server can annotate the continuous word and sentence information in the above-mentioned text sample data in various ways, so that the speech model can combine the continuous word and sentence information to generate predicted speech that conforms to natural pause relationships, thereby improving the naturalness of speech generation.

[0091] In one embodiment, such as Figure 4 As shown, a speech generation method is provided, which can be applied to... Figure 1Taking the server in the example, the following steps are included:

[0092] Step S302: Obtain the target text, and obtain the text structure information and phoneme information of the target text.

[0093] The target text can be text entered by the user on the terminal. The terminal can send the user-input target text to the server. After obtaining the target text, the server can extract the text structure information and phoneme information from the target text. The text structure information can include character information and syntactic information. The server can extract the phoneme information from the target text using tools such as jieba, determine the character information by recognizing the sequentially arranged characters in the target text, and identify the syntactic information in the target text using natural language processing technology.

[0094] Step S304: Input the target text into the trained speech model. The text encoder of the speech model obtains text encoding data based on the text structure information and phoneme information of the target text. The decoder of the speech model obtains predicted speech features based on the text encoding data. The predicted speech features are input into the vocoder in the speech model. The vocoder outputs the corresponding predicted speech based on the predicted speech features. The speech model is trained according to the above speech model training method. The speech corresponding to the target text is obtained based on the predicted speech.

[0095] The server can pre-train a speech model. The server inputs the target text into the speech model, which then outputs the predicted speech based on the textual structure information of the target text, namely the aforementioned character and syntactic information, and combined with phoneme information. This speech model can be trained using the methods described above.

[0096] The aforementioned speech model may include a text encoder, a decoder, and a vocoder. The speech model encodes character information, syntactic information, and phoneme information. The decoder decodes the encoded text data, generating predicted speech features based on the decoded data. The vocoder then generates the corresponding predicted speech based on these predicted speech features. The server can also return the output predicted speech to the terminal for playback, allowing the user to obtain the synthesized speech corresponding to the input target text.

[0097] Specifically, when a user needs to generate speech, they can input the text they want to hear, such as a sentence or an article. The terminal can display the text information entered by the user. After the user clicks the synthesis button on the terminal, the terminal sends the text to the server. After the server generates the speech, it can return the generated speech to the terminal, which then plays the speech. During the playback, the text color will highlight according to the actual progress of the sound, so that the user can easily understand the text progress corresponding to the currently playing speech in real time.

[0098] In the aforementioned speech generation method, the text structure information and phoneme information of the target text are input into a speech model, which then outputs the corresponding predicted speech based on the text structure information and phoneme information in the user-input target text. Compared to speech generated through traditional speech synthesis, this scheme combines text structure information and phoneme information for speech generation, resulting in more accurate and natural intonation and pauses in the generated speech, thus improving the naturalness of the generated speech.

[0099] In one embodiment, obtaining target text includes: obtaining the original text input by the user; obtaining continuous word and sentence information in the original text; and obtaining the target text based on the original text carrying the continuous word and sentence information.

[0100] In this embodiment, the user can pre-set the pause relationships between words in the text. The server can obtain the original text input by the user, and the user can annotate the original text with continuous word and sentence information. Thus, the server can obtain the continuous word and sentence information in the original text and obtain the target text based on the original text carrying continuous word and sentence information.

[0101] There are various ways to set the aforementioned pause relationships. For example, in one embodiment, the server can input the original text into a trained word segmentation model, and obtain continuous word and sentence information based on the word segmentation results corresponding to the original text output by the word segmentation model. Specifically, the server inputs the input text into the word segmentation model, identifies combinations that may become words or phrases, and the start and end times of phonemes between the characters in these combinations need to be limited, for example, within the aforementioned preset threshold, to avoid excessive length and the problem of pause errors, thereby achieving the annotation of continuous word and sentence information.

[0102] The server can also receive user controls on duration thresholds. For example, in one embodiment, the server obtains the user's maximum pause time threshold between each group of adjacent characters in the original text, and obtains continuous word / sentence information based on adjacent characters whose maximum pause time threshold is less than a preset threshold. Specifically, by setting a maximum pause duration between characters, the user enables the server to determine continuous word / sentence information based on adjacent characters whose maximum pause duration is less than the preset threshold. This avoids the problem of significant long pauses within a single sentence.

[0103] The server can also receive user-preset phrases. For example, in one embodiment, the server can obtain user-preset phrases for the original text input and obtain continuous phrase information based on the preset phrases. Specifically, when inputting text, the user can simultaneously input desired continuous phrases or sentences to form continuous phrase information, so that the speech model can adjust the pauses between characters in these continuous phrase information to avoid excessively long pauses.

[0104] In speech models, when generating predicted speech based on target text, two situations may arise requiring external control of pauses. The first is potential pause errors during model prediction; the second is that the model's prediction is correct but contradicts human subjective intentions. For the first situation, because the model relies on probability statistics for prediction, it may not have encountered some rare characters or words, leading to poor modeling. For the second situation, in specific scenarios, human subjective intentions may dictate the pauses between certain characters. In these cases, users can annotate the original text to obtain the target text carrying continuous word and sentence information.

[0105] Additionally, users can annotate the original text while inputting it to obtain the target text. In some embodiments, users can also modify the pause relationships in the target text after the server generates the predicted speech, resulting in a new target text. The server then generates a new predicted speech based on the new target text, thus updating the predicted speech. Specifically, when a user needs to adjust pauses, they can click on the corresponding text content on the terminal screen. The terminal displays a pop-up window where the user can edit the text, adding the corresponding information where they need to link words or insert extra pauses. Then, by clicking the "Save and Synthesize" button, the terminal sends the inserted linking or extra pause information to the server. The server obtains new continuous word / sentence information based on this information, and then uses a speech model to generate a new predicted speech, returning it to the terminal for real-time updates. Through this method, users can enjoy a fully automated text listening experience or manually edit text to achieve personalized and controllable effects.

[0106] Through the above embodiments, the server can determine the pause relationships in the text based on multiple methods, and the pause relationships can be updated in real time by the user. The server generates corresponding predicted speech by combining the above pause relationships with a speech model, thereby improving the naturalness of the generated speech.

[0107] In one embodiment, such as Figure 5 As shown, Figure 5 This is a flowchart illustrating the training method for a speech model in another embodiment. This embodiment includes a training phase and an application phase. In the training phase, text structure information such as character and syntactic information of the input text, as well as phoneme information of the input text, are acquired. The character and syntactic information are encoded using a text encoder, and the start and end times of each phoneme in the phoneme information are predicted by combining the text encoder with the target speech. For example, the start and end time prediction capability of the phoneme information is trained using a corresponding loss function.

[0108] The server inputs the encoded text data into the decoder. During the training phase, the server trains the speech model by combining the speech features of the target speech. For example, in the decoder, the speech model generates corresponding predicted speech features based on the aforementioned encoded text data. Thus, the server can train the speech model by comparing the predicted speech features with the target speech features, using the target speech's speech features as the target. This allows the model to not only know the pronunciation of each character but also the specific character and the context in which it is pronounced. After combining the aforementioned character, syntactic, and phoneme information to generate predicted speech features through encoding and decoding, the speech model can also generate corresponding predicted speech based on these predicted speech features using a vocoder.

[0109] In the application phase, after the server obtains the phoneme, character, and syntactic information of the input text, it can directly encode these information items using a text encoder. The encoded text data is then input into a decoder. The speech model in the decoder generates corresponding predicted speech features based on the phonemes corresponding to each character in the encoded text data, the start and end times of each phoneme, and the word attributes of each character. A vocoder then generates the corresponding predicted speech based on these predicted speech features. Furthermore, in some embodiments, the server can also obtain continuous word and sentence information input by the user and combine this information to determine the pause relationships in the generated speech. This speech generation can be processed in a computer backend or in the cloud, offering high processing efficiency and fast operation.

[0110] Through the above embodiments, the server adds character and syntactic information to the speech synthesis scheme, enabling the model's text encoder to simultaneously encode information at three granularities: phonemes, characters, and syntax. This provides the model with richer input information, helping it better understand the syntactic and semantic information of the input text, as well as the corresponding speech information, resulting in a more natural speech synthesis effect with pauses. Furthermore, by using control information based on word segmentation, duration thresholds, and custom phrases, the pauses can be controlled, improving the accuracy of the predicted speech and the naturalness of the spoken sound.

[0111] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0112] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores text sample data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a speech model training method and a speech generation method.

[0113] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0114] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech model training method and speech generation method.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method for training the speech model and the method for generating speech.

[0116] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described speech model training method and speech generation method.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training a speech model, characterized in that, The method includes: Obtain text sample data and the speech features of the target speech corresponding to the text sample data; The text structure information and phoneme information of the text sample data are obtained, and the text structure information and phoneme information are input into the speech model to be trained. The text encoder of the speech model obtains text encoded data based on the text structure information and phoneme information, and the decoder of the speech model obtains predicted speech features based on the text encoded data. The text structure information includes character information and syntactic information. The character information represents the information of each character arranged in sequence in the text sample data, and the syntactic information represents the word component information corresponding to the text sample data. The text encoded data is obtained based on the first text encoded data, the second text encoded data, and the start and end time prediction data of the phoneme information. The first text encoded data is obtained based on the phoneme information corresponding to each character in the character information, the second text encoded data is obtained based on the word attributes of each word in the character information, the word attributes are obtained based on the syntactic information, and the start and end time prediction data of the phoneme information is determined based on the text sample data. The speech feature encoding data of the speech features is obtained by the speech encoder, and the speech feature encoding data is input into the speech model. The decoder obtains the reference speech features based on the speech feature encoding data. Based on the similarity between the predicted speech features and the reference speech features, the model parameters of the speech model to be trained are adjusted until the preset training termination condition is met, thus obtaining the trained speech model.

2. The method according to claim 1, characterized in that, The method further includes: Obtain pre-annotated continuous word and sentence information from the text sample data; the pause time between each character in the continuous word and sentence information is less than a preset threshold; The continuous word and sentence information is input into the speech model to be trained, and the speech model determines the start and end time prediction data of each phoneme in the phoneme information based on the continuous word and sentence information.

3. The method according to claim 2, characterized in that, The step of obtaining pre-annotated continuous word and phrase information from the text sample data includes: The text sample data is input into the trained word segmentation model, and the pre-annotated continuous word and sentence information is obtained based on the word segmentation results corresponding to the text sample data output by the word segmentation model.

4. The method according to claim 2, characterized in that, The step of obtaining pre-annotated continuous word and phrase information from the text sample data further includes: Obtain the maximum pause time threshold between each group of adjacent characters in the text sample data, and obtain pre-annotated continuous word and sentence information based on adjacent characters whose maximum pause time threshold is less than the preset threshold.

5. The method according to claim 2, characterized in that, The step of obtaining pre-annotated continuous word and phrase information from the text sample data further includes: Obtain preset words and phrases for the input text sample data, and obtain pre-annotated continuous word and phrase information based on the preset words and phrases.

6. A speech generation method, characterized in that, The method includes: Obtain the target text, and obtain the text structure information and phoneme information of the target text; The target text is input into a trained speech model. The text encoder of the speech model obtains text encoded data based on the text structure information and phoneme information of the target text. The decoder of the speech model obtains predicted speech features based on the text encoded data. The predicted speech features are input into the vocoder in the speech model, and the vocoder outputs the corresponding predicted speech based on the predicted speech features. The text structure information includes character information and syntactic information. The character information represents the information of each character arranged in sequence in the target text, and the syntactic information represents the word component information corresponding to the target text. The text encoded data is obtained based on first text encoded data, second text encoded data, and start and end time prediction data of the phoneme information. The first text encoded data is obtained based on the phoneme information corresponding to each character in the character information. The second text encoded data is obtained based on the word attributes of each word in the character information. The word attributes are obtained based on the syntactic information. The start and end time prediction data of the phoneme information is determined based on text sample data. The speech model is trained according to the method described in any one of claims 1 to 5. The speech corresponding to the target text is obtained based on the predicted speech.

7. The method according to claim 6, characterized in that, The acquisition of the target text includes: Get the raw text input by the user; Obtain continuous word and phrase information from the original text; The target text is obtained from the original text carrying the information of the continuous words and phrases.

8. The method according to claim 7, characterized in that, The step of obtaining continuous word and phrase information in the original text includes: The original text is input into the trained word segmentation model, and continuous word and sentence information is obtained based on the word segmentation results corresponding to the original text output by the word segmentation model. And / or, Obtain the maximum pause time threshold between each group of adjacent characters in the original text, and obtain continuous word and sentence information based on adjacent characters whose maximum pause time threshold is less than a preset threshold; And / or, Obtain preset words and phrases for the original text input, and obtain continuous word and phrase information based on the preset words and phrases.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images