Text-to-voice conversion method and device
By compressing the corpus importance value and using a non-autoregressive model, the problem of high training cost of text to speech model is solved, achieving high accuracy and low cost to generate high-quality speech.
Patent Information
- Application Number
- CN202411279443.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
AI Technical Summary
When improving accuracy, existing text-to-speech models require a lot of computing resources and long-term training, resulting in high costs and difficulty in taking into account accuracy and reducing training costs.
By obtaining the importance of each unit object in the corpus, the corpus is compressed, and the text-to-speech model is trained using the compressed corpus, including non-autoregressive models, multi-level pronunciation modeling units and spectrogram refinement units, to generate High-quality voice.
While reducing the amount of corpus data, it retains effective data, saves computing resources, shortens training time, and reduces costs, while ensuring model accuracy and achieving high-quality voice output.
Smart Images

Figure CN119091852A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of smart homes and intelligent families, for example, to a text-to-speech method and device. Background Art
[0002] Text-to-Speech (TTS) models can convert input text into natural and fluent human speech, and have been widely used in smart homes, voice assistants, car systems, etc. It is understandable that text-to-speech models need to be trained based on a corpus before they can have the ability to convert speech.
[0003] In the related art, in order to improve the accuracy of the output results of the text-to-speech model, the data volume of the corpus is usually large. However, using a corpus with a large data volume to train the text-to-speech model requires more computing resources and takes a long time, resulting in a high cost for model training. Therefore, the related art cannot simultaneously take into account the two requirements of high accuracy of the text-to-speech model and reducing the cost of model training.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.
[0006] The embodiments of the present disclosure provide a text-to-speech method and device, which can take into account both the requirements of higher accuracy of the text-to-speech model and reduced model training costs.
[0007] According to a first aspect of the present disclosure, a text-to-speech method is provided, comprising:
[0008] Obtaining the importance value of each unit object in the corpus, wherein the unit object includes phonemes and function words;
[0009] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus, wherein the text-to-speech model is a non-autoregressive model;
[0010] The target text to be converted is input into the trained text-to-speech model, and the text-to-speech model is used to output the corresponding speech.
[0011] In some embodiments, obtaining the importance value of each unit object in the corpus includes:
[0012] Obtain the frequency ratio of each unit object in the corpus, wherein the frequency ratio of each unit object is positively correlated with the number of occurrences of the unit object;
[0013] Based on the frequency ratio of each unit object, the importance value of each unit object is determined.
[0014] The frequency weight of each unit object is the ratio of the number of occurrences of the unit object to the total number of occurrences of all unit objects in the corpus.
[0015] In some embodiments, based on the frequency weight of each unit object, determining the importance value of each unit object includes:
[0016] Among the frequency proportions of each unit object, determine the maximum frequency proportion and the minimum frequency proportion;
[0017] calculating a first difference between the maximum frequency ratio and the minimum frequency ratio, and a second difference between the frequency ratio of each unit object and the minimum frequency ratio;
[0018] The ratio of the second difference value to the first difference value corresponding to each unit object is used as the importance value of the unit object.
[0019] In some embodiments, taking the ratio of the second difference value to the first difference value corresponding to each unit object as the importance value of the unit object includes:
[0020] Calculate a first ratio of the second difference value to the first difference value corresponding to each unit object;
[0021] The product of the first ratio corresponding to each unit object and the preset control parameter is used as the importance value of the unit object, wherein the control parameter needs to be dynamically adjusted according to the training result information of the text-to-speech model.
[0022] In some embodiments, compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus includes:
[0023] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus;
[0024] Determine training result information of the text-to-speech model, and adjust model parameters and control parameters of the text-to-speech model based on the training result information;
[0025] Re-obtain the importance value of each unit object in the corpus;
[0026] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus.
[0027] In some embodiments, obtaining the importance value of each unit object in the corpus includes:
[0028] Get the information entropy of each unit object in the corpus;
[0029] The information entropy of each unit object is used as the importance value of the unit object.
[0030] In some embodiments, compressing the corpus based on the importance value of each unit object includes:
[0031] Determine the unit objects whose importance values meet the deletion conditions as deletable objects;
[0032] The deletable objects in the corpus are removed to achieve compression, and a compressed corpus is obtained.
[0033] In some embodiments, determining a unit object whose importance value meets the deletion condition as a deletable object includes:
[0034] Sort the unit objects in descending order of importance;
[0035] The other unit objects arranged after the Nth unit object are determined as deletable objects, where N is an integer greater than 1.
[0036] In some embodiments, determining a unit object whose importance value meets the deletion condition as a deletable object includes:
[0037] Compare the importance value of each unit object with the preset compression threshold;
[0038] Unit objects whose importance values are smaller than the compression threshold are determined as deletable objects.
[0039] In some embodiments, the compression threshold needs to be dynamically adjusted based on the training result information of the text-to-speech model.
[0040] In some embodiments, compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus includes:
[0041] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus;
[0042] Determine training result information of the text-to-speech model, and adjust model parameters and a compression threshold of the text-to-speech model based on the training result information;
[0043] Re-obtain the importance value of each unit object in the corpus;
[0044] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus.
[0045] In some embodiments, the text-to-speech model includes:
[0046] A multi-level prosody modeling unit, used to extract phoneme-level features and word-level features from the input text, and obtain multi-level prosody features based on the phoneme-level features and word-level features;
[0047] A spectrogram refinement unit, used for generating a mel-spectrogram, and refining an initially generated mel-spectrogram to obtain a target mel-spectrogram;
[0048] The speech generation unit is used to generate corresponding speech based on the multi-level prosodic features and the target Mel-spectrogram.
[0049] In some embodiments, the text-to-speech method further includes:
[0050] After the text-to-speech model is trained based on the compressed corpus, the style features of the target user are embedded in the corpus to obtain a dedicated corpus;
[0051] Fine-tune the text-to-speech model based on your own corpus.
[0052] In some embodiments, the target text to be converted is input into a trained text-to-speech model, and the corresponding speech is output using the text-to-speech model, including: inputting the target text to be converted into a fine-tuned text-to-speech model, and outputting the corresponding speech using the text-to-speech model.
[0053] According to a second aspect of the present disclosure, a text-to-speech device is provided, comprising:
[0054] An initial importance determination module is configured to obtain an importance value of each unit object in the corpus, the unit objects including phonemes and function words;
[0055] A conversion model training module is configured to compress the corpus based on the importance value of each unit object, and train a text-to-speech model based on the compressed corpus, wherein the text-to-speech model is a non-autoregressive model;
[0056] The text-to-speech conversion module is configured to input the target text to be converted into a trained text-to-speech model, and output the corresponding speech using the text-to-speech model.
[0057] According to a third aspect of the present disclosure, a text-to-speech device is provided, comprising a processor and a memory storing program instructions, wherein the processor is configured to execute the text-to-speech method provided by the first aspect of the present disclosure when running the program instructions.
[0058] The text-to-speech method and device provided in the embodiments of the present disclosure can achieve the following technical effects:
[0059] The text-to-speech method provided by the embodiment of the present disclosure compresses the corpus based on the importance value of each unit object, which can remove information of low importance in the corpus, and helps to reduce the amount of data in the corpus while retaining the valid data in the corpus as much as possible. Since the compressed corpus has a smaller amount of data than the original corpus, training the text-to-speech model based on the compressed corpus can save computing resources, shorten the training time, and help reduce the cost of model training. In addition, the compressed corpus retains the valid data in the corpus as much as possible, which helps to ensure that the text-to-speech model has a high accuracy. Therefore, based on the text-to-speech method provided by the embodiment of the present disclosure, it is possible to take into account both the high accuracy of the text-to-speech model and the reduction of the cost of model training.
[0060] The foregoing general description and the following description are exemplary and explanatory only and are not intended to limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:
[0062] Figure 1 It is a flowchart of a text-to-speech method provided by an embodiment of the present disclosure;
[0063] Figure 2 is a schematic diagram of a text-to-speech model provided by an embodiment of the present disclosure;
[0064] Figure 3 is a flowchart of another text-to-speech method provided by an embodiment of the present disclosure;
[0065] Figure 4 is a flowchart of another text-to-speech method provided by an embodiment of the present disclosure;
[0066] Figure 5is a flowchart of another text-to-speech method provided by an embodiment of the present disclosure;
[0067] Figure 6 is a structural schematic diagram of a text-to-speech device provided by an embodiment of the present disclosure;
[0068] Figure 7 is a structural schematic diagram of another text-to-speech device provided by an embodiment of the present disclosure;
[0069] Figure 8 It is a structural schematic diagram of another text-to-speech device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0070] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.
[0071] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.
[0072] Unless otherwise stated, the term "plurality" means two or more.
[0073] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.
[0074] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.
[0075] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.
[0076] The embodiments of the present disclosure provide a text-to-speech method, which can compress a corpus, use the compressed corpus to train a text-to-speech model, and then use the text-to-speech model to convert the text into corresponding speech.
[0077] Combination Figure 1 As shown, the embodiment of the present disclosure provides a text-to-speech method, the execution subject of the method is a text-to-speech device (hereinafter referred to as the device), and the text-to-speech method includes:
[0078] S101, the device obtains the importance value of each unit object in the corpus.
[0079] In the disclosed embodiment, the unit object includes phonemes and function words. Here, phonemes are the smallest phonetic units in speech. For example, in English, the word "cat" can be decomposed into three phonemes [k], [ae] and [t]. Function words are words that play a functional role in a sentence, including prepositions and conjunctions, etc. For example, in English, function words include "and" and "the".
[0080] In the disclosed embodiment, the importance value is a quantitative index used to measure the relative importance of a unit object in a corpus, which reflects the importance of the unit object for constructing a meaningful language expression. It is understandable that the importance of a unit object is positively correlated with the importance value of the unit object. Here, the features that can reflect the importance of the unit object can be quantified to obtain the importance value of the unit object. For example, the importance value of the unit object can be determined based on the number of occurrences of the unit object, or the information entropy of the unit object can be used as the importance value of the unit object.
[0081] S102, the device compresses the corpus based on the importance value of each unit object, and trains the text-to-speech model based on the compressed corpus.
[0082] As mentioned above, the importance value of a unit object is used to reflect the importance of the unit object. Therefore, compressing the corpus based on the importance value of each unit object can remove information of low importance in the corpus, which helps to reduce the amount of data in the corpus while retaining as much valid data as possible.
[0083] In some embodiments, compressing the corpus based on the importance value of each unit object includes: determining the unit objects whose importance values meet the deletion conditions as deletable objects; removing the deletable objects in the corpus to achieve compression, and obtaining a compressed corpus. Here, the unit objects whose importance values meet the deletion conditions are determined to be unit objects with lower importance, and thus can be used as deletable objects.
[0084] In the disclosed embodiment, the deletion condition can be set based on the ranking result of the importance value of the unit object, and the ranking of the importance value of the unit object can reflect the importance of the unit object. Alternatively, the deletion condition can be set based on a compression threshold, wherein the compression threshold is used to compare with the importance value of the unit object, and whether the importance value of the unit object is greater than the compression threshold can reflect the importance of the unit object.
[0085] Optionally, determining the unit objects whose importance values meet the deletion condition as deletable objects includes: sorting the unit objects in descending order of importance values; and determining the other unit objects after the Nth unit object as deletable objects. N is an integer greater than 1.
[0086] In the embodiment of the present disclosure, the compression threshold may be determined according to actual design requirements or may be adjusted dynamically. For example, N may be determined according to the total amount of unit object types, for example, N may be 80% of the total amount of unit object types.
[0087] Optionally, determining unit objects whose importance values meet the deletion condition as deletable objects includes: comparing the importance value of each unit object with a preset compression threshold, and determining unit objects whose importance values are less than the compression threshold as deletable objects.
[0088] In the disclosed embodiment, the compression threshold may be determined according to actual design requirements, or may be adjusted dynamically.
[0089] In the embodiment of the present disclosure, the text-to-speech model is a non-autoregressive model. Optionally, the non-autoregressive model may be CMDTTS (Coarse Mean Detail Text to Speech). Figure 2 As shown, the text-to-speech model includes a multi-level prosody modeling unit, a spectrogram refinement unit, and a speech generation unit. The multi-level prosody modeling unit is used to extract phoneme-level features and word-level features from the input text, and obtain multi-level prosody features based on the phoneme-level features and word-level features. For example, a reference encoder can be used to capture multi-level prosody features in phoneme-level features and word-level features. The spectrogram refinement unit is used to generate a Mel-spectrogram, and the initially generated Mel-spectrogram is refined to obtain a target Mel-spectrogram. The speech generation unit is used to generate corresponding speech based on the multi-level prosody features and the target Mel-spectrogram.
[0090] Optionally, the spectrum graph refinement unit may be a DDPM (Denoising Diffusion Probabilistic Model).
[0091] In the embodiment of the present disclosure, the phoneme-level features extracted from the input text can be expressed by the following formula: Phoneme-level embeddings = E phoneme (text sequence). Phoneme-level embeddings represent phoneme-level features, text sequence represents text, E phoneme (text sequence) means obtaining phoneme-level features based on the input text.
[0092] In the embodiment of the present disclosure, word-level features extracted from the input text can be expressed by the following formula: Word-level embeddings = E word (text sequence). Word-level embeddings represent word-level features, text sequence represents text, E word (text sequence) Obtain word-level features based on the input text.
[0093] In the disclosed embodiment, refining the initially generated Mel-spectrogram to obtain the target Mel-spectrogram can be expressed by the following formula: Mel-spectrogram = DDPM (Initial Mel-spectrogram). Mel-spectrogram represents the target Mel-spectrogram, Initial Mel-spectrogram represents the initially generated Mel-spectrogram, and DDPM (Initial Mel-spectrogram) represents using a denoising diffusion probability model to refine the initially generated Mel-spectrogram to obtain the target Mel-spectrogram.
[0094] In the disclosed embodiment, the generation of corresponding speech based on multi-level prosodic features and target Mel-spectrogram can be expressed by the following formula: Synthesized Speech = CMDTTS (Phoneme-level embeddings, Word-level embeddings, Word-level embeddings, Mel-spectrogram). Phoneme-level embeddings, Word-level embeddings, and Word-level embeddings constitute multi-level prosodic features, Synthesized Speech represents the generated speech, and CMDTTS (Phoneme-level embeddings, Word-level embeddings, Word-level embeddings, Mel-spectrogram) represents the use of CMDTTS to generate corresponding speech based on multi-level prosodic features and target Mel-spectrogram.
[0095] Optionally, the spectrogram refinement unit may be a VAE (Variational Autoencoder). VAE can generate a high-fidelity spectrogram by modeling data distribution, which helps to improve the quality of speech synthesis.
[0096] Optionally, the non-autoregressive model may be a self-supervised learning model, which can learn representations from unlabeled data and then be combined with a non-autoregressive structure to further improve the generation speed and quality of the model.
[0097] S103, the device inputs the target text to be converted into the trained text-to-speech model, and outputs the corresponding speech using the text-to-speech model.
[0098] The text-to-speech method provided by the embodiment of the present disclosure compresses the corpus based on the importance value of each unit object, which can remove information of low importance in the corpus, and helps to reduce the amount of data in the corpus while retaining the valid data in the corpus as much as possible. Since the compressed corpus has a smaller amount of data than the original corpus, training the text-to-speech model based on the compressed corpus can save computing resources, shorten the training time, and help reduce the cost of model training. In addition, the compressed corpus retains the valid data in the corpus as much as possible, which helps to ensure that the text-to-speech model has a high accuracy. Therefore, based on the text-to-speech method provided by the embodiment of the present disclosure, it is possible to take into account both the high accuracy of the text-to-speech model and the reduction of the cost of model training.
[0099] In addition, in order to further avoid the high accuracy of the text-to-speech model being affected by the reduction in the amount of data in the compressed corpus, the embodiment of the present disclosure defines the text-to-speech model as a non-autoregressive model, and the text-to-speech model includes a multi-level prosodic modeling unit and a spectrogram refinement unit, so that multi-level prosodic features and finer Mel-spectrograms can be obtained. Speech is generated based on the multi-level prosodic features and finer Mel-spectrograms, which can improve the naturalness and clarity of the synthesized speech. Even in the case of a compressed corpus, high-quality speech output can still be maintained, which helps to ensure that the text-to-speech model has a high accuracy.
[0100] In some embodiments, obtaining the importance value of each unit object in the corpus includes: obtaining the frequency ratio of each unit object in the corpus, wherein the frequency ratio of each unit object is positively correlated with the number of occurrences of the unit object; and determining the importance value of each unit object based on the frequency ratio of each unit object.
[0101] Combination Figure 3 As shown, the embodiment of the present disclosure provides another text-to-speech method, the text-to-speech method comprising:
[0102] S301, the device obtains the frequency ratio of each unit object in the corpus.
[0103] In the disclosed embodiment, the frequency weight of each unit object is positively correlated with the number of occurrences of the unit object. That is, the more the number of occurrences of the unit object, the greater the frequency weight of the unit object, and at the same time, the greater the importance value of the unit object.
[0104] Optionally, the frequency weight of each unit object is the ratio of the number of occurrences of the unit object to the total number of occurrences of all unit objects in the corpus. Taking the unit objects of a corpus including phoneme [k], phoneme [ae], phoneme [t], function word "and" and function word "the" as an example, the number of occurrences of each unit object is shown in Table 1.
[0105] Table 1
[0106] Phonemes or function words Occurrences [k] 150 [ar] 100 [t] 200 and 300 the 400
[0107] In the embodiment of the present disclosure, the total number of occurrences of the unit object can be expressed as f T , f T The calculation process of f is as follows: T =150+100+200+300+400=1150.
[0108] Taking phoneme [k] as an example, the frequency ratio of phoneme [k] can be expressed as fk , f k The calculation process of f is as follows: k =150 / 1150=0.13.
[0109] S302: The device determines the importance value of each unit object based on the frequency ratio of each unit object.
[0110] In the embodiment of the present disclosure, the importance value of each unit object is positively correlated with the frequency weight of the unit object. That is, the greater the frequency weight of the unit object, the greater the importance value of the unit object.
[0111] S303, the device compresses the corpus based on the importance value of each unit object, and trains the text-to-speech model based on the compressed corpus.
[0112] S304, the device inputs the target text to be converted into the trained text-to-speech model, and outputs the corresponding speech using the text-to-speech model.
[0113] In some embodiments, based on the frequency ratio of each unit object, the importance value of each unit object is determined, including: determining the maximum frequency ratio and the minimum frequency ratio among the frequency ratios of each unit object; calculating a first difference between the maximum frequency ratio and the minimum frequency ratio, and a second difference between the frequency ratio of each unit object and the minimum frequency ratio; and taking the ratio of the second difference corresponding to each unit object to the first difference as the importance value of the unit object.
[0114] In the embodiment of the present disclosure, the maximum frequency proportion is the largest frequency proportion among the frequency proportions of all unit objects, and the minimum frequency proportion is the smallest frequency proportion among the frequency proportions of all unit objects. For ease of understanding and expression, the difference between the maximum frequency proportion and the minimum frequency proportion is defined as the first difference, and the difference between the frequency proportion of each unit object and the minimum frequency proportion is defined as the second difference.
[0115] The maximum frequency ratio is expressed as f max , the minimum frequency weight is expressed as f min , taking phoneme [k] as an example, the importance value of phoneme [k] is expressed as R k , R k It can be calculated by the following formula:
[0116]
[0117] In some embodiments, the ratio of the second difference value to the first difference value corresponding to each unit object is used as the importance value of the unit object, including: calculating a first ratio of the second difference value to the first difference value corresponding to each unit object; and multiplying the first ratio corresponding to each unit object by a preset control parameter as the importance value of the unit object. The control parameter needs to be dynamically adjusted according to the training result information of the text-to-speech model.
[0118] The control parameter is represented as α. Taking phoneme [k] as an example, the importance value of phoneme [k] is represented as R k , R k It can be calculated by the following formula:
[0119]
[0120] In some embodiments, a corpus is compressed based on the importance value of each unit object, and a text-to-speech model is trained based on the compressed corpus, including: compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus; determining training result information of the text-to-speech model, and adjusting model parameters and control parameters of the text-to-speech model based on the training result information; reacquiring the importance value of each unit object in the corpus; compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus.
[0121] In the disclosed embodiment, the control parameters are dynamically adjusted through the training result information of the text-to-speech model, so that the compression amount of the corpus can be changed in real time, so that the data volume of the compressed corpus conforms to the actual situation of the model, which helps to achieve the best balance between the accuracy of the text-to-speech model and the training of the model.
[0122] In some embodiments, obtaining the importance value of each unit object in the corpus includes: obtaining the information entropy of each unit object in the corpus, and using the information entropy of each unit object as the importance value of the unit object.
[0123] Combination Figure 4 As shown, the embodiment of the present disclosure provides another text-to-speech method, the text-to-speech method comprising:
[0124] S401, the device obtains the information entropy of each unit object in the corpus.
[0125] S402: The device uses the information entropy of each unit object as the importance value of the unit object.
[0126] S403, the device compresses the corpus based on the importance value of each unit object, and trains the text-to-speech model based on the compressed corpus.
[0127] S404, the device inputs the target text to be converted into the trained text-to-speech model, and outputs the corresponding speech using the text-to-speech model.
[0128] In some embodiments, determining a unit object whose importance value meets the deletion condition as a deletable object includes: comparing the importance value of each unit object with a preset compression threshold, and determining a unit object whose importance value is less than the compression threshold as a deletable object. The compression threshold needs to be dynamically adjusted according to the training result information of the text-to-speech model.
[0129] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus, including: compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus; determining the training result information of the text-to-speech model, and adjusting the model parameters and compression threshold of the text-to-speech model based on the training result information; re-obtaining the importance value of each unit object in the corpus; compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus.
[0130] In the disclosed embodiments, the compression threshold is dynamically adjusted through the training result information of the text-to-speech model, so that the compression amount of the corpus can be changed in real time, so that the data volume of the compressed corpus conforms to the actual situation of the model, which helps to achieve the best balance between the accuracy of the text-to-speech model and the training of the model.
[0131] In some embodiments, a corpus is compressed based on the importance value of each unit object, and a text-to-speech model is trained based on the compressed corpus, including: compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus; determining training result information of the text-to-speech model, and adjusting model parameters, control parameters and compression thresholds of the text-to-speech model based on the training result information; reacquiring the importance value of each unit object in the corpus; reacquiring the importance value of each unit object in the corpus; compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus.
[0132] In the disclosed embodiments, the control parameters and compression thresholds are dynamically adjusted through the training result information of the text-to-speech model, so that the compression amount of the corpus can be changed in real time, so that the data volume of the compressed corpus conforms to the actual situation of the model, which helps to achieve the best balance between the accuracy of the text-to-speech model and the training of the model.
[0133] Here, obtaining the importance value of each unit object in the corpus includes: obtaining the frequency ratio of each unit object in the corpus; determining the maximum frequency ratio and the minimum frequency ratio among the frequency ratios of each unit object; calculating a first difference between the maximum frequency ratio and the minimum frequency ratio, and a second difference between the frequency ratio of each unit object and the minimum frequency ratio; calculating a first ratio of the second difference corresponding to each unit object to the first difference; and taking the product of the first ratio corresponding to each unit object and a preset control parameter as the importance value of the unit object.
[0134] The corpus is compressed based on the importance value of each unit object, including: comparing the importance value of each unit object with a preset compression threshold, determining the unit object whose importance value is less than the compression threshold as a deletable object; removing the deletable objects in the corpus to achieve compression, and obtaining a compressed corpus.
[0135] In some embodiments, determining training result information of a text-to-speech model includes: evaluating the quality of speech output by the text-to-speech model, and using the quality of the speech as the training result information.
[0136] Optionally, the MOS (Mean Opinion Score) value and the WER (Word Error Rate) value of the speech may be calculated, and the quality of the speech output by the text-to-speech model may be evaluated based on the MOS value and the WER value.
[0137] In the embodiments of the present disclosure, in a speech synthesis or speech communication system, MOS is used to measure user satisfaction with speech quality, and a higher MOS value indicates better speech quality. The MOS value can be calculated using the following formula:
[0138]
[0139] In the above formula, MOS represents the MOS value, N refers to the number of speech samples involved in the scoring, and MOS j represents the opinion score of the jth speech sample.
[0140] In the disclosed embodiment, WER is an indicator for evaluating the accuracy of an automatic speech recognition system. It determines the error rate by calculating the difference between the recognized text and the reference text, including the number of replacements, omissions, and extra words. The lower the WER value, the higher the accuracy of the speech recognition system. The WER value can be calculated by the following formula:
[0141]
[0142] In the above formula, Substitutions represents the number of substitution errors, Deletions represents the number of omission errors, Insertions represents the number of redundant errors, and Total number of words represents the total number of words in the text.
[0143] In some embodiments, a reinforcement learning method may be applied to dynamically adjust control parameters and compression thresholds.
[0144] In some embodiments, the text-to-speech method provided by the embodiments of the present disclosure also includes: after training the text-to-speech model based on the compressed corpus, embedding the style features of the target user in the corpus to obtain a dedicated corpus; and fine-tuning the text-to-speech model based on the dedicated corpus.
[0145] In some embodiments, the target text to be converted is input into a trained text-to-speech model, and the corresponding speech is output using the text-to-speech model, including: inputting the target text to be converted into a fine-tuned text-to-speech model, and outputting the corresponding speech using the text-to-speech model.
[0146] Combination Figure 5 As shown, the embodiment of the present disclosure provides another text-to-speech method, the text-to-speech method comprising:
[0147] S501, the device obtains the importance value of each unit object in the corpus.
[0148] S502, the device compresses the corpus based on the importance value of each unit object, and trains the text-to-speech model based on the compressed corpus.
[0149] S503, after the device completes training the text-to-speech model based on the compressed corpus, the device embeds the style features of the target user into the corpus to obtain a dedicated corpus.
[0150] S504, the device fine-tunes the text-to-speech model based on the dedicated corpus.
[0151] In the disclosed embodiment, the device can use CGANs (Conditional Generative Adversarial Networks) to fine-tune the text-to-speech model based on a dedicated corpus. CGANs includes a generator and a discriminator, and the generator is used to generate speech data that is as realistic as possible. The discriminator is used to distinguish between generated data and real data, and to improve the effect of the generator through adversarial training. It can be understood that it is the generator in the text-to-speech model.
[0152] In the embodiment of the present disclosure, the discriminator loss function in the fine-tuning process is as follows:
[0153] CGANsLoss=E real [log D(real)]+E fake [log(1-D(G(fake)))].
[0154] In the above formula, CGANsLoss represents the loss value, D(real) represents the probability of the discriminator outputting real data, and E real [log D(real)] represents the expected value of real data, G(fake) represents the data generated by the generator, D(G(fake)) represents the probability of the discriminator outputting the generated data, and E fake [log(1-D(G(fake)))] represents the expected value of the generated data (fake data).
[0155] In the disclosed embodiment, after the fine-tuning of the text-to-speech model is completed, the fine-tuning effect of the text-to-speech model can be evaluated. Here, the quality of the speech output by the text-to-speech model can be determined based on the calculation method of the target similarity (Target Similarity), and the fine-tuning effect of the text-to-speech model can be evaluated based on the quality of the speech.
[0156] In the disclosed embodiment, the device can use the contrastive learning method to fine-tune the text-to-speech model based on the dedicated corpus. Specifically, by constructing positive and negative sample pairs, the model can capture the characteristics of the target speaker, and through the contrastive learning loss function, the generator can be guided to generate more natural and similar speech data.
[0157] S505, the device inputs the target text to be converted into the fine-tuned text-to-speech model, and outputs the corresponding speech using the text-to-speech model.
[0158] In the disclosed embodiment, by fine-tuning the text-to-speech model using a dedicated corpus containing the style features of the target user, the speech features of the text-to-speech model can be made to have a high degree of similarity with the voice features of the target user, allowing the text-to-speech model to provide users with personalized, high-quality voice services.
[0159] In the embodiments of the present disclosure, corpora of different language types (such as Chinese and English) can be used to train the text-to-speech model, so that the text-to-speech model has multi-language support capabilities and can effectively adapt to the speech generation needs in a multi-language environment.
[0160] In the embodiments of the present disclosure, a meta-learning method is used to enable the text-to-speech model to quickly adapt to data in different languages. Through a small amount of corpus adjustment, multi-language support and personalized synthesis are achieved, thereby improving the generalization ability of the model.
[0161] Combination Figure 6 As shown, the embodiment of the present disclosure provides a text-to-speech device 600 , which includes an initial importance determination module 601 , a conversion model training module 602 , and a text-to-speech conversion module 603 .
[0162] The initial importance determination module 601 is configured to obtain the importance value of each unit object in the corpus, where the unit objects include phonemes and function words.
[0163] The conversion model training module 602 is configured to compress the corpus based on the importance value of each unit object, and train the text-to-speech model based on the compressed corpus, where the text-to-speech model is a non-autoregressive model.
[0164] The text-to-speech conversion module 603 is configured to input the target text to be converted into a trained text-to-speech model, and output the corresponding speech using the text-to-speech model.
[0165] The text-to-speech device provided by the embodiment of the present disclosure compresses the corpus based on the importance value of each unit object, which can remove information of low importance in the corpus, and helps to reduce the data volume of the corpus while retaining the valid data in the corpus as much as possible. Since the compressed corpus has a smaller data volume than the original corpus, training the text-to-speech model based on the compressed corpus can save computing resources, shorten the training time, and help reduce the model training cost. In addition, the compressed corpus retains the valid data in the corpus as much as possible, which helps to ensure that the text-to-speech model has a high accuracy. Therefore, the text-to-speech method provided by the embodiment of the present disclosure can take into account both the high accuracy of the text-to-speech model and the reduction of the model training cost.
[0166] In some embodiments, the initial importance determination module 601 is configured to:
[0167] Obtain the frequency ratio of each unit object in the corpus, wherein the frequency ratio of each unit object is positively correlated with the number of occurrences of the unit object;
[0168] Based on the frequency ratio of each unit object, the importance value of each unit object is determined.
[0169] In some embodiments, the frequency weight of each unit object is the ratio of the number of occurrences of the unit object to the total number of occurrences of all unit objects in the corpus.
[0170] In some embodiments, the initial importance determination module 601 is configured to:
[0171] Among the frequency proportions of each unit object, determine the maximum frequency proportion and the minimum frequency proportion;
[0172] calculating a first difference between the maximum frequency ratio and the minimum frequency ratio, and a second difference between the frequency ratio of each unit object and the minimum frequency ratio;
[0173] The ratio of the second difference value to the first difference value corresponding to each unit object is used as the importance value of the unit object.
[0174] In some embodiments, the initial importance determination module 601 is configured to:
[0175] Calculate a first ratio of the second difference value to the first difference value corresponding to each unit object;
[0176] The product of the first ratio corresponding to each unit object and the preset control parameter is used as the importance value of the unit object, wherein the control parameter needs to be dynamically adjusted according to the training result information of the text-to-speech model.
[0177] In some embodiments, the conversion model training module 602 is configured to:
[0178] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus;
[0179] Determine training result information of the text-to-speech model, and adjust model parameters and control parameters of the text-to-speech model based on the training result information;
[0180] Re-obtain the importance value of each unit object in the corpus;
[0181] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus.
[0182] In some embodiments, the initial importance determination module 601 is configured to:
[0183] Get the information entropy of each unit object in the corpus;
[0184] The information entropy of each unit object is used as the importance value of the unit object.
[0185] In some embodiments, the conversion model training module 602 is configured to:
[0186] Determine the unit objects whose importance values meet the deletion conditions as deletable objects;
[0187] The deletable objects in the corpus are removed to achieve compression, and a compressed corpus is obtained.
[0188] In some embodiments, the conversion model training module 602 is configured to:
[0189] Sort the unit objects in descending order of importance;
[0190] The other unit objects arranged after the Nth unit object are determined as deletable objects, where N is an integer greater than 1.
[0191] In some embodiments, the conversion model training module 602 is configured to:
[0192] Compare the importance value of each unit object with the preset compression threshold;
[0193] Unit objects whose importance values are smaller than the compression threshold are determined as deletable objects.
[0194] In some embodiments, the compression threshold needs to be dynamically adjusted based on the training result information of the text-to-speech model.
[0195] In some embodiments, the conversion model training module 602 is configured to:
[0196] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus;
[0197] Determine training result information of the text-to-speech model, and adjust model parameters and a compression threshold of the text-to-speech model based on the training result information;
[0198] Re-obtain the importance value of each unit object in the corpus;
[0199] The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus.
[0200] In some embodiments, the text-to-speech model includes:
[0201] A multi-level prosody modeling unit, used to extract phoneme-level features and word-level features from the input text, and obtain multi-level prosody features based on the phoneme-level features and word-level features;
[0202] A spectrogram refinement unit, used for generating a mel-spectrogram, and refining an initially generated mel-spectrogram to obtain a target mel-spectrogram;
[0203] The speech generation unit is used to generate corresponding speech based on the multi-level prosodic features and the target Mel-spectrogram.
[0204] Combination Figure 7 As shown, the embodiment of the present disclosure provides another text-to-speech device 600 , which includes an initial importance determination module 601 , a conversion model training module 602 , and a text-to-speech conversion module 603 , and further includes a model fine-tuning module 604 .
[0205] The model fine-tuning module 604 is configured to: after completing the training of the text-to-speech model based on the compressed corpus, embed the style features of the target user in the corpus to obtain a dedicated corpus; and fine-tune the text-to-speech model based on the dedicated corpus.
[0206] The text-to-speech conversion module 603 is configured to: input the target text to be converted into the fine-tuned text-to-speech model, and output the corresponding speech using the text-to-speech model.
[0207] Combination Figure 8 As shown, the embodiment of the present disclosure provides another text-to-speech device 800, which includes a processor 801 and a memory 802. Optionally, the text-to-speech device 800 may also include a communication interface 803 and a bus 804. The processor 801, the communication interface 803, and the memory 802 may communicate with each other through the bus 804. The communication interface 803 may be used for information transmission. The processor 801 may call the logic instructions in the memory 802 to execute the text-to-speech method of the above embodiment.
[0208] In addition, the logic instructions in the memory 802 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.
[0209] The memory 802 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 801 executes the function application and data processing by running the program instructions / modules stored in the memory 802, that is, the text-to-speech method in the above embodiment is implemented.
[0210] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 802 may include a high-speed random access memory and may also include a non-volatile memory.
[0211] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned text-to-speech method.
[0212] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.
[0213] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0214] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.
[0215] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0216] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A text-to-speech method, characterized in that: include: Obtaining the importance value of each unit object in the corpus, wherein the importance value is a quantitative indicator used to measure the relative importance of the unit object in the corpus; Compressing the corpus based on the importance value of each unit object, and training the text-to-speech model based on the compressed corpus, wherein the text-to-speech model is a non-autoregressive model; The target text to be converted is input into the trained text-to-speech model, and the text-to-speech model is used to output the corresponding speech.
2. The text-to-speech method according to claim 1, characterized in that: Get the importance value of each unit object in the corpus, including: Obtain the frequency ratio of each unit object in the corpus, wherein the frequency ratio of each unit object is positively correlated with the number of occurrences of the unit object; Based on the frequency ratio of each unit object, the importance value of each unit object is determined.
3. The text-to-speech method according to claim 2, characterized in that: Based on the frequency ratio of each unit object, the importance value of each unit object is determined, including: Among the frequency proportions of each unit object, determine the maximum frequency proportion and the minimum frequency proportion; calculating a first difference between the maximum frequency ratio and the minimum frequency ratio, and a second difference between the frequency ratio of each unit object and the minimum frequency ratio; The ratio of the second difference value to the first difference value corresponding to each unit object is used as the importance value of the unit object.
4. The text-to-speech method according to claim 3, characterized in that: The ratio of the second difference value to the first difference value corresponding to each unit object is used as the importance value of the unit object, including: Calculate a first ratio of the second difference value to the first difference value corresponding to each unit object; The product of the first ratio corresponding to each unit object and the preset control parameter is used as the importance value of the unit object, wherein the control parameter needs to be dynamically adjusted according to the training result information of the text-to-speech model.
5. The text-to-speech method according to claim 4, characterized in that: The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus, including: The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus; Determine training result information of the text-to-speech model, and adjust model parameters and control parameters of the text-to-speech model based on the training result information; Re-obtain the importance value of each unit object in the corpus; The corpus is compressed based on the importance value of each unit object, and the text-to-speech model is trained based on the compressed corpus.
6. The text-to-speech method according to any one of claims 1 to 5, characterized in that: The corpus is compressed based on the importance value of each unit object, including: Determine the unit objects whose importance values meet the deletion conditions as deletable objects; The deletable objects in the corpus are removed to achieve compression, and a compressed corpus is obtained.
7. The text-to-speech method according to any one of claims 1 to 5, characterized in that: The text-to-speech model includes: A multi-level prosody modeling unit, used to extract phoneme-level features and word-level features from the input text, and obtain multi-level prosody features based on the phoneme-level features and word-level features; A spectrogram refinement unit, used for generating a mel-spectrogram, and refining an initially generated mel-spectrogram to obtain a target mel-spectrogram; The speech generation unit is used to generate corresponding speech based on the multi-level prosodic features and the target Mel-spectrogram.
8. The text-to-speech method according to any one of claims 1 to 5, characterized in that: Also includes: After the text-to-speech model is trained based on the compressed corpus, the style features of the target user are embedded in the corpus to obtain a dedicated corpus; Fine-tune the text-to-speech model based on your own corpus.
9. A text-to-speech device, characterized in that: include: An initial importance determination module is configured to obtain an importance value of each unit object in the corpus, the unit objects including phonemes and function words; A conversion model training module is configured to compress the corpus based on the importance value of each unit object, and train a text-to-speech model based on the compressed corpus, wherein the text-to-speech model is a non-autoregressive model; The text-to-speech conversion module is configured to input the target text to be converted into a trained text-to-speech model, and output the corresponding speech using the text-to-speech model.
10. A text-to-speech device, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the text-to-speech method according to any one of claims 1 to 8 when running the program instructions.
