A speech model training method, a speech synthesis method, and related devices thereof
By constructing a speech network that includes a large language model and a pinyin embedding layer, and introducing pronunciation loss during training, the problem of pronunciation errors in large speech models when dealing with Chinese polyphonic characters is solved, and higher pronunciation accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU QUWAN NETWORK TECH CO LTD
- Filing Date
- 2025-07-30
- Publication Date
- 2026-07-14
AI Technical Summary
Existing large speech models are prone to errors when processing the pronunciation of Chinese polyphonic characters. The main reason is that splitting Chinese characters into sub-words makes it impossible for the model to accurately associate character shape with pronunciation.
A speech network is constructed, including a large language model, a conditional flow matching model, a vocoder, and an attention module. The network branches are trained using Pinyin data, and a pronunciation loss is introduced to correct pronunciation errors of polyphonic characters. A Pinyin embedding layer is added to provide a reference for correct pronunciation.
This improves the pronunciation accuracy of the speech model when dealing with Chinese polyphonic characters, ensuring the correct pronunciation of the synthesized audio.
Smart Images
Figure CN120726989B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech synthesis technology, and in particular to a speech model training method, a speech synthesis method, and related apparatus. Background Technology
[0002] Text-to-speech (TTS), as an important topic in generative artificial intelligence, has made rapid progress in recent years. To achieve efficient synthesis of natural and high-quality human speech, many institutions and companies have conducted research on related projects. These include NaturalSpeech (NaturalSpeech 2) launched by the Machine Learning Group of Microsoft Research Asia and the Microsoft Azure Speech team, and Voicebox released by Meta. Both represent the use of big data, large models, and zero-shot synthesis technology to achieve diversity in timbre, rhythm, and style in speech synthesis.
[0003] Currently, the introduction of large language model technology into the TTS field has greatly improved the emotion, naturalness, and anthropomorphism of synthesized audio. Examples include mature large speech models in China such as SeedTTTS, IndexTTS, and Cosyvoice. However, existing large speech models are prone to errors when handling the pronunciation of polyphonic Chinese characters. This is mainly because these models use Byte Pair Encoding (BPE) to split Chinese characters into subwords, causing the model to fail to accurately associate character form with pronunciation. In Chinese speech models using BPE encoding, pronunciation errors of polyphonic characters are a common problem. Summary of the Invention
[0004] This application provides a speech model training method, a speech synthesis method, and related apparatus to improve the technical problem that existing large speech models are prone to pronunciation errors when processing Chinese polyphonic characters.
[0005] In view of this, the first aspect of this application provides a speech model training method, comprising:
[0006] A speech network is constructed, comprising a large language model, a conditional flow matching model, a vocoder, an attention module, and a decoder. The attention module, the decoder, and the vocoder are sequentially connected in series to form a first network branch, and the large language model, the conditional flow matching model, and the vocoder are sequentially connected in series to form a second network branch.
[0007] Obtain the first pinyin data and the corresponding first target audio of the first training text, use the first pinyin data as input data, use the first target audio as the training target to train the first network branch, and obtain the trained first network branch;
[0008] Obtain reference audio, use the acoustic markers of the reference audio and the second training text as input data, and train the second network branch with the second target audio corresponding to the second training text as the training target to obtain the trained second network branch;
[0009] Load the attention module and decoder network parameters from the trained first network branch, as well as the large language model, conditional flow matching model, and vocoder network parameters from the trained second network branch, into the speech network to obtain the initialized speech network; use the acoustic markers, the second training text, and the second pinyin data corresponding to the second training text as input data, and use the second target audio as the training target to train the initialized speech network to obtain the trained speech model.
[0010] Optionally, the step of training the first network branch using the first pinyin data as input data and the first target audio as the training target to obtain the trained first network branch includes:
[0011] The first pinyin data is input into the first network branch, the weight information of the first pinyin data is calculated by the attention module, and the first pinyin data is weighted by the weight information; the first metronome is generated by the decoder based on the weighted first pinyin data.
[0012] The first Mel spectrogram is reconstructed into audio using the vocoder to obtain the first generated audio.
[0013] Calculate a first pronunciation loss based on the first target audio and the first generated audio; update the network parameters of the first network branch using the first pronunciation loss to obtain the trained first network branch.
[0014] Optionally, the step of training the second network branch using the acoustic markers of the reference audio and the second training text as input data, and using the second target audio corresponding to the second training text as the training target, to obtain the trained second network branch, includes:
[0015] The acoustic tag of the reference audio and the second training text are input into the second network branch, and the large language model generates semantic tags containing speech information based on the acoustic tag and the second training text; the conditional flow matching model models a second Mel spectrogram containing speech information based on the semantic tags.
[0016] The second Mel spectrogram is reconstructed into audio using the vocoder to obtain the second generated audio.
[0017] The second pronunciation loss is calculated based on the second target audio and the second generated audio; the network parameters of the second network branch are updated using the second pronunciation loss to obtain the trained second network branch.
[0018] Optionally, the step of training the initialized speech network using the acoustic markers, the second training text, and the second pinyin data corresponding to the second training text as input data, and using the second target audio as the training target to obtain a trained speech model, includes:
[0019] The acoustic markers and the second training text are input into the large language model of the initialized speech network, and the second pinyin data corresponding to the second training text is input into the attention module of the initialized speech network.
[0020] The large language model generates semantic tags containing speech information based on the acoustic tags and the second training text; the conditional flow matching model models a second Mel spectrogram containing speech information based on the semantic tags;
[0021] The attention module calculates the weight information of the second pinyin data and weights the second pinyin data accordingly; the decoder then decodes and generates a third Mel spectrogram based on the weighted second pinyin data.
[0022] The second Mel spectrogram and the third Mel spectrogram are reconstructed into audio using the vocoder to obtain the third generated audio.
[0023] The third pronunciation loss is calculated based on the second target audio and the third generated audio; the network parameters of the initialized speech network are updated using the third pronunciation loss until the initialized speech network converges, thus obtaining a trained speech model.
[0024] Optionally, calculating the first pronunciation loss based on the first target audio and the first generated audio includes:
[0025] The first target audio and the first generated audio are converted into pinyin respectively to obtain the target pinyin sequence and the generated pinyin sequence;
[0026] Calculate the CTC loss of the initials and finals of the target pinyin sequence and the generated pinyin sequence to obtain the phoneme-level loss;
[0027] Calculate the cross-entropy between the target pinyin sequence and the generated pinyin sequence to obtain the syllable-level loss;
[0028] The mean square error of the tones in the target pinyin sequence and the generated pinyin sequence is calculated to obtain the tone level loss;
[0029] The first pronunciation loss is obtained by weighted summation of the phoneme-level loss, the syllable-level loss, and the tone-level loss.
[0030] Optionally, the method further includes:
[0031] During the training of the speech network, the generated audio output by the speech network is converted into pinyin to obtain a generated pinyin sequence;
[0032] Calculate the KL divergence loss of the generated pinyin sequence and the second pinyin data, and adjust the hyperparameters of the speech network based on the KL divergence loss.
[0033] A second aspect of this application provides a speech synthesis method, including:
[0034] Obtain reference audio of the target speaker, extract the timbre and style features of the target speaker from the reference audio, and obtain the target acoustic tag;
[0035] The target text, the corresponding pinyin data of the target text, and the target acoustic marker are input into a trained speech model, and audio is synthesized through the speech model; wherein the speech model is trained using any of the speech model training methods described in the first aspect.
[0036] A third aspect of this application provides a speech model training device, comprising:
[0037] A network construction unit is used to construct a speech network, which includes a large language model, a conditional flow matching model, a vocoder, an attention module, and a decoder. The attention module, the decoder, and the vocoder are connected in series to form a first network branch, and the large language model, the conditional flow matching model, and the vocoder are connected in series to form a second network branch.
[0038] The first training unit is used to acquire the first pinyin data of the first training text and the corresponding first target audio, and to train the first network branch with the first pinyin data as input data and the first target audio as the training target, so as to obtain the trained first network branch.
[0039] The second training unit is used to acquire reference audio, use the acoustic markers of the reference audio and the second training text as input data, and use the second target audio corresponding to the second training text as the training target to train the second network branch, so as to obtain the trained second network branch.
[0040] The third training unit is used to load the network parameters of the attention module and the decoder in the trained first network branch, as well as the network parameters of the large language model, the conditional flow matching model, and the vocoder in the trained second network branch, into the speech network to obtain the initialized speech network; and to train the initialized speech network with the acoustic markers, the second training text, and the second pinyin data corresponding to the second training text as input data, and with the second target audio as the training target, to obtain the trained speech model.
[0041] A fourth aspect of this application provides a speech synthesis apparatus, comprising:
[0042] The feature extraction unit is used to acquire the reference audio of the target speaker, extract the timbre and style features of the target speaker from the reference audio, and obtain the target acoustic tag.
[0043] A speech synthesis unit is used to input target text, the corresponding pinyin data of the target text, and the target acoustic marker into a trained speech model, and synthesize audio through the speech model; wherein the speech model is trained using any of the speech model training methods described in the first aspect.
[0044] A fifth aspect of this application provides an electronic device, the device including a processor and a memory;
[0045] The memory is used to store program code and transmit the program code to the processor;
[0046] The processor is configured to execute any of the speech model training methods described in the first aspect, or to execute the speech synthesis method described in the second aspect, according to the instructions in the program code.
[0047] As can be seen from the above technical solutions, this application has the following advantages:
[0048] The speech model training method provided in this application adds pinyin annotations to Chinese text in the training data, adds a pinyin embedding layer to the speech network, and further introduces pronunciation loss during the network training process. This allows the speech network to provide a correct pronunciation reference to the network when it encounters a Chinese polyphonic character with a pronunciation error, based on the correct pronunciation of the corresponding character, and finally synthesizes the correct audio, thereby improving the pronunciation accuracy of the speech model. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0050] Figure 1 A schematic flowchart of a speech model training method provided in this application embodiment;
[0051] Figure 2 A schematic diagram of the structure of a voice network provided in an embodiment of this application;
[0052] Figure 3 A schematic flowchart of a speech synthesis method provided in this application embodiment;
[0053] Figure 4 A schematic diagram of a speech model training device provided in an embodiment of this application;
[0054] Figure 5 This is a schematic diagram of a speech synthesis device provided in an embodiment of this application. Detailed Implementation
[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0056] For easier understanding, please refer to Figure 1 This application provides a speech model training method, including:
[0057] Step 110: Construct a speech network. The speech network includes a large language model, a conditional flow matching model, a vocoder, an attention module, and a decoder. The attention module, decoder, and vocoder are connected in series to form the first network branch, and the large language model, conditional flow matching model, and vocoder are connected in series to form the second network branch.
[0058] Please refer to the speech network constructed in this embodiment of the application. Figure 2, The speech network includes a large language model (Text-to-token LM), a conditional flow matching model (Flow Match), an acoustic vocoder (Acoustic Vocoder), an attention module (Attention), and a decoder (Transformer Decoder). Among them, the attention module, the decoder, and the acoustic vocoder are connected in series in sequence to form a first network branch, and the large language model, the conditional flow matching model, and the acoustic vocoder are connected in series in sequence to form a second network branch. The input of the large language model includes the text content (text Tokens) and acoustic tokens (speechTokens) in the training text, and the acoustic tokens can be extracted from the reference audio (Reference) through a speech tokenizer (Speech Tokenizer).
[0059] Step 120: Obtain the first pinyin data of the first training text and the corresponding first target audio, use the first pinyin data as the input data, and use the first target audio as the training target to train the first network branch to obtain the trained first network branch;
[0060] A large number of training texts and their corresponding target audios can be obtained from an existing speech synthesis database. After obtaining several first training texts from the database in the embodiments of this application, the first training texts are predicted through a pre-trained language model (such as BERT-Pinyin) to obtain the pinyin data corresponding to the first training texts; the first training texts can also be pinyin-labeled through an ASR model to obtain the corresponding pinyin data; the pinyin data predicted by the language model and the pinyin data obtained by the ASR model can also be combined. When there are different pinyins for the same text, manual verification can be performed to obtain the final pinyin data.
[0061] The first training text and its corresponding first pinyin data can be hierarchically labeled. For example, hello[ni3hao3], and the text and pinyin are distinguished by special marks.
[0062] Input the first pinyin data into the first network branch, calculate the weight information of the first pinyin data through the attention module, and weight the first pinyin data through the weight information to obtain the weighted first pinyin data. The attention module determines the proportion of each pinyin in the input when synthesizing each frame of audio; the decoder decodes and generates the first mel spectrogram (mel) according to the weighted first pinyin data; the acoustic vocoder reconstructs the first mel spectrogram into audio to obtain the first generated audio; calculate the first pronunciation loss according to the first target audio and the first generated audio; update the network parameters of the first network branch through the first pronunciation loss to obtain the trained first network branch; save the network parameters of the attention module and the decoder in the trained first network branch to obtain the first training parameters.
[0063] The calculation process for the first pronunciation loss is as follows: The first target audio and the first generated audio are converted into Pinyin, respectively, to obtain the target Pinyin sequence and the generated Pinyin sequence; the CTC loss of the initials and finals in the target Pinyin sequence and the generated Pinyin sequence is calculated to obtain the phoneme-level loss; the cross-entropy of the target Pinyin sequence and the generated Pinyin sequence is calculated to obtain the syllable-level loss; the mean square error of the tones in the target Pinyin sequence and the generated Pinyin sequence is calculated to obtain the tone-level loss; the phoneme-level loss, syllable-level loss, and tone-level loss are weighted and summed to obtain the first pronunciation loss. The weight coefficients of each loss term can be set according to the actual situation, and their specific values are not limited here.
[0064] Step 130: Obtain reference audio, use the acoustic markers of the reference audio and the second training text as input data, and train the second network branch with the second target audio corresponding to the second training text as the training target to obtain the trained second network branch.
[0065] The reference audio can be parsed using a speech tokenizer to obtain its acoustic tokens, which determine the timbre and global style of the synthesized audio. A large amount of second training text and its corresponding target audio can be obtained from a database. The large language model receives the acoustic tokens of the second training text and the reference audio, and then generates semantic tokens containing speech information. A second Mel spectrogram containing speech information is modeled based on the semantic tokens using a conditional flow matching model. The second Mel spectrogram is reconstructed into audio using a vocoder to obtain the second generated audio. A second articulation loss is calculated based on the second target audio and the second generated audio. The network parameters of the second network branch are updated using the second articulation loss to obtain the trained second network branch. The network parameters of each module (large language model, conditional flow matching model, and vocoder) in the trained second network branch are saved to obtain the second training parameters. The specific calculation process of the second articulation loss is similar to that of the first articulation loss, and will not be elaborated further here.
[0066] The first training text and the second training text can be completely identical or partially identical.
[0067] Step 140: Load the network parameters of the attention module and decoder in the first network branch and the network parameters of the large language model, conditional flow matching model and vocoder in the second network branch into the speech network to obtain the initialized speech network; use the acoustic markers, the second training text and the second pinyin data corresponding to the second training text as input data, and the second target audio as the training target to train the initialized speech network to obtain the trained speech model.
[0068] The first and second training parameters are loaded into the speech network to obtain the initialized speech network. Then, the acoustic markers and the second training text are input into the large language model in the initialized speech network, and the second pinyin data corresponding to the second training text is input into the attention module in the initialized speech network. The large language model generates semantic markers containing speech information based on the acoustic markers and the second training text. The conditional flow matching model models a second Mel spectrogram containing speech information based on the semantic markers. The attention module calculates the weight information of the second pinyin data and weights the second pinyin data accordingly. The decoder decodes the weighted second pinyin data to generate a third Mel spectrogram. The vocoder reconstructs the second and third Mel spectrograms into audio to obtain the third generated audio. The third pronunciation loss is calculated based on the second target audio and the third generated audio. The network parameters of the initialized speech network are updated using the third pronunciation loss until the initialized speech network converges (e.g., reaching the maximum number of training iterations, or the training accuracy / error converges to a certain value), resulting in a trained speech model.
[0069] The calculation process for the third pronunciation loss is similar to that for the first pronunciation loss, and will not be elaborated further here. The second training text can be predicted using a pre-trained language model to obtain the corresponding second pinyin data; alternatively, the second training text can be annotated with pinyin using an ASR model to obtain the corresponding second pinyin data; or, the pinyin data predicted by the language model and the pinyin data obtained by the ASR model can be combined. When different pinyin versions exist for the same character, manual verification can be performed to obtain the final pinyin data.
[0070] Furthermore, during the training of the initialized speech network, the generated audio output by the initialized speech network is converted into Pinyin to obtain a generated Pinyin sequence;
[0071] Calculate the KL divergence loss of the generated pinyin sequence and the second pinyin data, and adjust the hyperparameters (such as the learning rate) of the speech network based on the KL divergence loss. For example, in the early stage of training, when the KL divergence loss is large, the learning rate can be increased; in the later stage of training, when the KL divergence loss gradually decreases and falls below the threshold, the learning rate can be decreased.
[0072] Furthermore, after obtaining the trained speech model, its effectiveness can be validated. Evaluation metrics for the speech model can be obtained through a validation set, such as the accuracy of pinyin (including initials, finals, and tones), and the error rate of characters with pinyin constraints (this can be achieved by comparing the number of incorrect pinyin in the pinyin sequence corresponding to the synthesized audio of the speech model with the number of incorrect pinyin in the input data of the speech model, and then calculating the ratio of the number of incorrect pinyin to the total number of pinyin / total number of characters in the pinyin data). The naturalness of pronunciation can also be manually evaluated (e.g., scored from 1 to 5). If a certain evaluation metric does not meet the requirements, the speech network can be retrained using the above steps until the speech model meets the actual requirements, and then the speech model can be deployed to actual engineering applications.
[0073] The speech model training method provided in this application adds pinyin annotations to Chinese text in the training data, adds a pinyin embedding layer to the speech network, and further introduces pronunciation loss during the network training process. This allows the speech network to provide a correct pronunciation reference to the network when it encounters a Chinese polyphonic character with a pronunciation error, based on the correct pronunciation of the corresponding character, and finally synthesizes the correct audio, thereby improving the pronunciation accuracy of the speech model.
[0074] Please refer to Figure 3 This application also provides a speech synthesis method, including:
[0075] Step 310: Obtain the reference audio of the target speaker, extract the timbre and style features of the target speaker from the reference audio, and obtain the target acoustic tag;
[0076] After obtaining the reference audio of the target speaker, the reference audio can be parsed using a speech tagger to obtain the target acoustic tags of the reference audio. The reference audio determines the timbre and global style of the synthesized audio.
[0077] Step 320: Input the target text, the corresponding pinyin data, and the target acoustic markers into the trained speech model, and synthesize audio through the speech model;
[0078] The target acoustic markers and target text are input into a large language model within a pre-trained speech model. The corresponding pinyin data for the target text is input into the attention module of the pre-trained speech model. The large language model generates semantic markers containing speech information based on the target acoustic markers and target text. A conditional flow matching model models a Mel spectrogram containing speech information based on the semantic markers. The attention module calculates the weight information of the pinyin data and weights the pinyin data accordingly. A decoder decodes the weighted pinyin data to generate a Mel spectrogram. A vocoder combines the Mel spectrogram generated by the conditional flow matching model and the Mel spectrogram generated by the decoder according to a certain weight, and then reconstructs the audio to obtain synthesized audio. The speech model is trained using the speech model training method described in the aforementioned method embodiment.
[0079] The speech synthesis method provided in this application adds pinyin annotations to Chinese text and injects prior pinyin knowledge into the speech model by adding a pinyin embedding layer (i.e., the first network branch), which helps to improve the pronunciation accuracy of synthesized audio.
[0080] Please refer to Figure 4 This application also provides a speech model training device, including:
[0081] The network building unit 410 is used to build a speech network. The speech network includes a large language model, a conditional flow matching model, a vocoder, an attention module, and a decoder. The attention module, decoder, and vocoder are connected in series to form the first network branch, and the large language model, conditional flow matching model, and vocoder are connected in series to form the second network branch.
[0082] The first training unit 420 is used to acquire the first pinyin data of the first training text and the corresponding first target audio, and train the first network branch with the first pinyin data as input data and the first target audio as the training target to obtain the trained first network branch.
[0083] The second training unit 430 is used to acquire reference audio, use the acoustic markers of the reference audio and the second training text as input data, and use the second target audio corresponding to the second training text as the training target to train the second network branch, so as to obtain the trained second network branch.
[0084] The third training unit 440 is used to train the speech network with acoustic tags and the second training text as input data for the trained second network branch, the second pinyin data corresponding to the second training text as input data for the trained first network branch, and the second target audio as the training target, so as to obtain the trained speech model.
[0085] As a further improvement, the device also includes: a parameter adjustment unit, used for:
[0086] When training the speech network, the generated audio output by the speech network is converted into Pinyin to obtain a generated Pinyin sequence;
[0087] Calculate the KL divergence loss of the generated pinyin sequence and the second pinyin data, and adjust the hyperparameters of the speech network based on the KL divergence loss.
[0088] Please refer to Figure 5 This application also provides a speech synthesis device, including:
[0089] The feature extraction unit 510 is used to acquire the reference audio of the target speaker, extract the timbre and style features of the target speaker from the reference audio, and obtain the target acoustic tag.
[0090] The speech synthesis unit 520 is used to input the target text, the corresponding pinyin data of the target text, and the target acoustic marker into the trained speech model, and synthesize audio through the speech model; wherein, the speech model is trained using the speech model training method in the aforementioned method embodiment.
[0091] This application also provides an electronic device, which includes a processor and a memory;
[0092] The memory is used to store program code and transfer the program code to the processor;
[0093] The processor is used to execute the speech model training method or speech synthesis method in the foregoing method embodiments according to the instructions in the program code.
[0094] This application also provides a computer-readable storage medium for storing program code, which, when executed by a processor, implements the speech model training method or speech synthesis method in the aforementioned method embodiments.
[0095] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0096] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.
[0097] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0098] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0102] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A speech model training method, characterized in that, include: A speech network is constructed, comprising a large language model, a conditional flow matching model, a vocoder, an attention module, and a decoder. The attention module, the decoder, and the vocoder are sequentially connected in series to form a first network branch, and the large language model, the conditional flow matching model, and the vocoder are sequentially connected in series to form a second network branch. Obtain the first pinyin data and the corresponding first target audio of the first training text, use the first pinyin data as input data, use the first target audio as the training target to train the first network branch, and obtain the trained first network branch; Obtain reference audio, use the acoustic markers of the reference audio and the second training text as input data, and train the second network branch with the second target audio corresponding to the second training text as the training target to obtain the trained second network branch; Load the attention module and decoder network parameters from the trained first network branch, as well as the large language model, conditional flow matching model, and vocoder network parameters from the trained second network branch, into the speech network to obtain an initialized speech network; use the acoustic markers, the second training text, and the second pinyin data corresponding to the second training text as input data, and use the second target audio as the training target to train the initialized speech network to obtain a trained speech model; The process of training the initialized speech network using the acoustic markers, the second training text, and the second pinyin data corresponding to the second training text as input data, and using the second target audio as the training target, to obtain a trained speech model includes: The acoustic markers and the second training text are input into the large language model of the initialized speech network, and the second pinyin data corresponding to the second training text is input into the attention module of the initialized speech network. The large language model generates semantic tags containing speech information based on the acoustic tags and the second training text; the conditional flow matching model models a second Mel spectrogram containing speech information based on the semantic tags; The attention module calculates the weight information of the second pinyin data and weights the second pinyin data accordingly; the decoder then decodes and generates a third Mel spectrogram based on the weighted second pinyin data. The second Mel spectrogram and the third Mel spectrogram are reconstructed into audio using the vocoder to obtain the third generated audio. The third pronunciation loss is calculated based on the second target audio and the third generated audio; the network parameters of the initialized speech network are updated using the third pronunciation loss until the initialized speech network converges, thus obtaining a trained speech model.
2. The speech model training method according to claim 1, characterized in that, The step of training the first network branch using the first pinyin data as input data and the first target audio as the training target to obtain the trained first network branch includes: The first pinyin data is input into the first network branch, the weight information of the first pinyin data is calculated by the attention module, and the first pinyin data is weighted by the weight information; the first metronome is generated by the decoder based on the weighted first pinyin data. The first Mel spectrogram is reconstructed into audio using the vocoder to obtain the first generated audio. Calculate a first pronunciation loss based on the first target audio and the first generated audio; update the network parameters of the first network branch using the first pronunciation loss to obtain the trained first network branch.
3. The speech model training method according to claim 2, characterized in that, The step of calculating the first pronunciation loss based on the first target audio and the first generated audio includes: The first target audio and the first generated audio are converted into pinyin respectively to obtain the target pinyin sequence and the generated pinyin sequence; Calculate the CTC loss of the initials and finals of the target pinyin sequence and the generated pinyin sequence to obtain the phoneme-level loss; Calculate the cross-entropy between the target pinyin sequence and the generated pinyin sequence to obtain the syllable-level loss; The mean square error of the tones in the target pinyin sequence and the generated pinyin sequence is calculated to obtain the tone level loss; The first pronunciation loss is obtained by weighted summation of the phoneme-level loss, the syllable-level loss, and the tone-level loss.
4. The speech model training method according to claim 1, characterized in that, The method further includes: During the training of the speech network, the generated audio output by the speech network is converted into pinyin to obtain a generated pinyin sequence; Calculate the KL divergence loss of the generated pinyin sequence and the second pinyin data, and adjust the hyperparameters of the speech network based on the KL divergence loss.
5. A speech synthesis method, characterized in that, include: Obtain reference audio of the target speaker, extract the timbre and style features of the target speaker from the reference audio, and obtain the target acoustic tag; The target text, the corresponding pinyin data, and the target acoustic marker are input into a trained speech model, and audio is synthesized through the speech model; wherein the speech model is trained using the speech model training method described in any one of claims 1-4.
6. A speech model training device, characterized in that, include: A network construction unit is used to construct a speech network, which includes a large language model, a conditional flow matching model, a vocoder, an attention module, and a decoder. The attention module, the decoder, and the vocoder are connected in series to form a first network branch, and the large language model, the conditional flow matching model, and the vocoder are connected in series to form a second network branch. The first training unit is used to acquire the first pinyin data of the first training text and the corresponding first target audio, and to train the first network branch with the first pinyin data as input data and the first target audio as the training target, so as to obtain the trained first network branch. The second training unit is used to acquire reference audio, use the acoustic markers of the reference audio and the second training text as input data, and use the second target audio corresponding to the second training text as the training target to train the second network branch, so as to obtain the trained second network branch. The third training unit is used to load the network parameters of the attention module and the decoder in the first network branch and the network parameters of the large language model, the conditional flow matching model and the vocoder in the second network branch into the speech network to obtain the initialized speech network; and to train the initialized speech network with the acoustic marker, the second training text and the second pinyin data corresponding to the second training text as input data and the second target audio as the training target to obtain the trained speech model. The process of training the initialized speech network using the acoustic markers, the second training text, and the second pinyin data corresponding to the second training text as input data, and using the second target audio as the training target, to obtain a trained speech model includes: The acoustic markers and the second training text are input into the large language model of the initialized speech network, and the second pinyin data corresponding to the second training text is input into the attention module of the initialized speech network. The large language model generates semantic tags containing speech information based on the acoustic tags and the second training text; the conditional flow matching model models a second Mel spectrogram containing speech information based on the semantic tags; The attention module calculates the weight information of the second pinyin data and weights the second pinyin data accordingly; the decoder then decodes and generates a third Mel spectrogram based on the weighted second pinyin data. The second Mel spectrogram and the third Mel spectrogram are reconstructed into audio using the vocoder to obtain the third generated audio. The third pronunciation loss is calculated based on the second target audio and the third generated audio; the network parameters of the initialized speech network are updated using the third pronunciation loss until the initialized speech network converges, thus obtaining a trained speech model.
7. A speech synthesis device, characterized in that, include: The feature extraction unit is used to acquire the reference audio of the target speaker, extract the timbre and style features of the target speaker from the reference audio, and obtain the target acoustic tag. A speech synthesis unit is used to input target text, the corresponding pinyin data of the target text, and the target acoustic marker into a trained speech model, and synthesize audio through the speech model; wherein the speech model is trained using the speech model training method according to any one of claims 1-4.
8. An electronic device, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the speech model training method according to any one of claims 1-4, or execute the speech synthesis method according to claim 5, according to the instructions in the program code.