Speech synthesis method, speech synthesis model training method, electronic device, and computer program product

Abstract

The application provides a speech synthesis method, a speech synthesis model training method, an electronic device and a computer program product. The text features and speech demand information of a target text can be input into a speech synthesis model to generate synthesized speech of the target text read by a target dialect. The speech synthesis model includes an encoding module, a dialect discrimination module and a decoding module. The dialect discrimination module can output decision information to the encoding module, and then correct the initial prior information output by the encoding module or improve the quality of the initial prior information by means of the decision information to obtain target prior information with more dialect characteristics. Finally, the decoding module generates synthesized speech based on the target prior information, so that the dialect characteristics of the synthesized speech are more prominent, and the effect of the synthesized dialect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech synthesis method, a training method of a speech synthesis model, an electronic device, and a computer program product. BACKGROUND

[0002] In a country with a vast territory and numerous ethnic groups, people's communication is increasingly frequent, and dialects in different regions have become an important element in communication. The emergence of speech synthesis technology involving dialects provides a more convenient and natural way of interaction. Among them, dialect, also known as "local language" or "local accent", refers to a language in a certain region that is different from the standard language.

[0003] Currently, a trained speech synthesis model can be used to generate synthesized speech of dialects. During the training phase, the speech synthesis model needs to be trained using a large amount of dialect speech, so as to learn the unique speech characteristics and grammatical structures of each dialect.

[0004] However, there are usually many types of dialects. According to statistics, some countries with a vast territory may have more than one hundred types of dialects, which makes it extremely complex to train a speech synthesis model covering various dialects. For example, the tones, finals, initials, etc. of different dialects differ greatly, which requires a large amount of data and complex algorithms for processing. Limited by training data and algorithms, current speech synthesis models mostly use conventional codecs to synthesize dialect speech, resulting in general effects of synthesizing dialects and not enough obvious dialect characteristics. SUMMARY

[0005] Based on the above technical status, the present application provides a speech synthesis method, a training method of a speech synthesis model, an electronic device, and a computer program product.

[0006] According to a first aspect of an embodiment of the present application, a speech synthesis method is provided, the method comprising:

[0007] obtaining text features of a target text and speech requirement information; wherein the speech requirement information comprises an identifier of a target dialect;

[0008] inputting the text features and the speech requirement information into a speech synthesis model to obtain synthesized speech of the target text read by the target dialect;

[0009] The speech synthesis model comprises an encoding module, a dialect discrimination module and a decoding module. The encoding module is configured to generate initial prior information according to the text feature and the speech requirement information. The dialect discrimination module is configured to generate decision information according to the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is further configured to generate target prior information according to the text feature, the speech requirement information and the decision information, and input the target prior information into the decoding module. The decoding module is configured to generate the synthesized speech according to the target prior information.

[0010] According to a second aspect of the embodiment of the present application, a training method of a speech synthesis model is provided, and the method comprises:

[0011] obtaining a text feature and speech requirement information of sample data; wherein the speech requirement information comprises an identifier of a target dialect;

[0012] training a speech synthesis model according to the text feature and the speech requirement information;

[0013] updating model parameters of the speech synthesis model based on model loss;

[0014] The speech synthesis model comprises an encoding module, a dialect discrimination module and a decoding module. The encoding module is configured to generate initial prior information according to the text feature and the speech requirement information. The dialect discrimination module is configured to generate decision information according to the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is further configured to generate target prior information according to the text feature, the speech requirement information and the decision information, and input the target prior information into the decoding module. The decoding module is configured to generate the synthesized speech according to the target prior information, and the synthesized speech comprises synthesized speech of the sample data read by the target dialect.

[0015] According to a third aspect of the embodiment of the present application, an electronic device is provided, comprising a memory and a processor; the memory is connected with the processor, and is configured to store a program; the processor is configured to realize the speech synthesis method according to the first aspect or the training method of the speech synthesis model according to the second aspect by running the program in the memory.

[0016] According to a fourth aspect of the embodiment of the present application, a storage medium is provided, and the storage medium stores a computer program. When the computer program is run by a processor, the speech synthesis method according to the first aspect or the training method of the speech synthesis model according to the second aspect is realized.

[0017] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the speech synthesis method as described in the first aspect or the training method of the speech synthesis model as described in the second aspect.

[0018] In an embodiment of the present application, the text features and speech requirement information of the target text are input into a speech synthesis model, and a synthesized speech that reads the target text in the target dialect can be generated. The speech synthesis model includes: an encoding module, a dialect discrimination module, and a decoding module. The dialect discrimination module can output judgment information to the encoding module, and then use the judgment information to correct or improve the quality of the initial prior information output by the encoding module to obtain target prior information with more dialect characteristics. Finally, the decoding module generates synthesized speech based on the target prior information, making the dialect characteristics of the synthesized speech more prominent and improving the effect of the synthesized dialect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0020] Figure 1 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the structure of the speech synthesis model provided in the embodiment of the present application;

[0022] Figure 3 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of a speech synthesis model training device provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] Overview

[0028] As mentioned in the background, the emergence of speech synthesis technology for dialects has enabled the output of text-based synthesized speech in a dialect, i.e., dialect speech. Currently, there are three main approaches to synthesizing dialect speech. Approach 1: Unit-based synthesis: This involves segmenting dialect speech into several basic units, such as phonemes, syllables, or words. Based on the input text, appropriate units are selected from a pre-recorded unit library and spliced ​​together to produce a speech waveform. Approach 2: Synthesis based on statistical parameters: This involves analyzing large amounts of dialect speech data to establish acoustic and language models. The acoustic model predicts the acoustic features of speech, such as fundamental frequency and spectrum; the language model processes the linguistic information of the text, such as syntax and semantics. Based on the input text, these two models are used to generate corresponding acoustic feature parameters, which are then converted into a speech waveform by a synthesizer. Approach 3: End-to-end synthesis based on deep learning: This employs a deep neural network to directly convert the input text into a speech waveform, eliminating the need for intermediate acoustic feature parameter conversion. Approach 3's processing logic is relatively simple, requiring only the collection of appropriate samples for model training.

[0029] However, due to the large variety of dialects, each with unique phonetic characteristics and grammatical structures, traditional neural network models (such as recurrent neural networks and long short-term memory networks) are generally ineffective in synthesizing dialects, and the dialectal characteristics are not obvious enough.

[0030] In view of the above-mentioned technical status, the inventors propose that a discriminator can be introduced into the neural network model, so that the encoded data in the model can be corrected or the quality of the encoded data can be improved with the help of the signal fed back by the discriminator, so that it has more obvious dialect characteristics. Therefore, an embodiment of the present application proposes a speech synthesis method. The text features and speech demand information of the target text are input into the speech synthesis model, and a synthesized speech that reads the target text in the target dialect can be generated. Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The dialect discrimination module can output judgment information to the encoding module, and then use the judgment information to correct the initial prior information output by the encoding module or improve the quality of the initial prior information, so as to obtain target prior information with more dialect characteristics. Finally, the decoding module generates a synthesized speech based on the target prior information, so that the dialect characteristics of the synthesized speech are more obvious, thereby improving the effect of the synthesized dialect.

[0031] Exemplary Methods

[0032] See also Figure 1 In an exemplary embodiment, a speech synthesis method is provided, which may include:

[0033] S101: Acquire text features and speech requirement information of a target text; wherein the speech requirement information includes an identifier of a target dialect.

[0034] In this step, the target text is the text to be converted into dialect speech. The semantic content of the target text is not limited here. For example, the target text can be an article, a speech, or a news report. In another example, the target text can be text converted from a piece of speech data.

[0035] The text features of the target text may include pronunciation features of the target text in the target dialect, semantic features of the target text, etc. In some embodiments, the text features include: a target phoneme sequence of the target text that is characteristic of the target dialect. The target dialect characteristics include characteristics or regularities of the target dialect in pronunciation, prosody, and intonation.

[0036] The speech requirement information includes: the requirement for the final synthesized speech. In some embodiments, the requirement for the final synthesized speech includes: the requirement for the dialect type of the final synthesized speech. For example, the requirement may be to synthesize a dialect speech under a target dialect. The target dialect may be a dialect of any region. The identifier of the dialect may be a unique identifier of the dialect. In the process of synthesizing speech, the identifier may be used to determine the dialect speech that needs to be synthesized, which is the dialect speech under the target dialect indicated by the identifier. In some embodiments, the identifier of the dialect may be a dialect label. It is understandable that the speech requirement information includes but is not limited to the requirement for the dialect type of the final synthesized speech.

[0037] S102: input the text feature and the speech requirement information into the speech synthesis model to obtain a synthesized speech of reading the target text in the target dialect.

[0038] In this step, the speech synthesis model includes an encoding module, a dialect discrimination module, and a decoding module. The encoding module is configured to generate initial prior information based on the text feature and the speech requirement information. The dialect discrimination module is configured to generate decision information based on the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is further configured to generate target prior information based on the text feature, the speech requirement information, and the decision information, and input the target prior information into the decoding module. The decoding module is configured to generate a synthesized speech based on the target prior information.

[0039] It can be understood that the initial prior information is an intermediate representation generated by the module based on the text feature and the speech requirement information. The encoding module may, for example, be an encoder in a neural network model, which is configured to capture semantic information, syntactic information, contextual information, dialect information, and the like and convert them into a form easy to process, i.e., initial prior information. For example, the network architecture of the encoding module can be the same as that of the encoding part of the SOVITS model.

[0040] The discrimination module can discriminate the initial prior information generated by the encoding module, so as to determine the quality of the initial prior information. The higher the quality, the more accurate the content represented; otherwise, the lower the quality, the greater the deviation of the content represented. By generating decision information and feeding it back to the encoding module, the initial prior information is corrected, and the quality of the prior information generated by the encoding module is improved. Here, the encoding module will generate target prior information with higher quality based on the decision information. In this embodiment, the more accurate the content represented by the prior information output by the encoding module, the more prominent or accurate the dialect characteristics under the target dialect. In some embodiments, the network architecture of the discrimination module can be similar to that of the SOVITS model, for example, it can be that the multi-resolution discriminator of the SOVITS model is removed and the remaining part is retained. In this way, the original audio signal is converted to 2D after passing through a 6-layer 1D convolution module, and then passes through a 2D, 5-layer multi-period convolution module (the scales of the multi-period convolution are 2, 3, 5, 7, 11), and the returned result can include two types of final generated decision information and layer output feature decision results.

[0041] The decoding module may, for example, be a decoder in a neural network model, which can generate the final synthesized speech based on the intermediate representation output by other modules. In some embodiments, the network architecture of the decoding module can be the same as that of the decoding part of the SOVITS model.

[0042] In an embodiment of the present application, the text features and speech requirement information of the target text are input into a speech synthesis model, and a synthesized speech that reads the target text in the target dialect can be generated. The speech synthesis model includes: an encoding module, a dialect discrimination module, and a decoding module. The dialect discrimination module can output judgment information to the encoding module, and then use the judgment information to correct or improve the quality of the initial prior information output by the encoding module to obtain target prior information with more dialect characteristics. Finally, the decoding module generates synthesized speech based on the target prior information, making the dialect characteristics of the synthesized speech more prominent and improving the effect of the synthesized dialect.

[0043] In order to further enhance the dialect characteristics of the synthesized speech, in some embodiments of the present application, the speech synthesis model further includes: a prosody encoding module, wherein the prosody encoding module is used to generate a prosody encoding of the target text in the target dialect based on text features and speech requirement information;

[0044] The encoding module inputs the target prior information into the decoding module, including:

[0045] The encoding module concatenates the prosody code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0046] It should be noted that the prosodic code represents the prosodic information of the target text in the target dialect. By combining the target prior information with the prosodic code, we can obtain intermediate features that are more consistent with the dialect's characteristics, namely the encoding features. Furthermore, the synthesized speech generated using the encoding features has a more pronounced dialectal character.

[0047] In some embodiments, the prosody encoding module may be, for example, a large language model. The large language model is trained separately so that it can output prosody encoding based on input text features and dialect identification.

[0048] In the embodiment of the present application, a prosody coding module dedicated to providing prosody coding is provided in the speech synthesis model. Through the processing of the prosody coding model, the dialect characteristics of the synthesized speech can be further enhanced.

[0049] In some embodiments of the present application, obtaining text features of the target text includes:

[0050] Based on the phoneme pronunciation rules of the target dialect, the target text is converted into an initial phoneme sequence; based on the prosody control rules and / or tone control rules of the target dialect, the phonemes in the initial phoneme sequence are arranged and combined to generate a target phoneme sequence with the characteristics of the target dialect.

[0051] It should be noted that the initial phoneme sequence and the target phoneme sequence are sequences composed of phonemes. A phoneme is the smallest unit of speech divided according to the natural properties of speech. Each pronunciation action within a syllable is analyzed, with each action constituting a phoneme. For example, the Chinese syllable "啊" (ā) has only one phoneme, "爱" (ài) has two phonemes, and "代" (dai) has three phonemes.

[0052] It is understandable that different dialects have different phoneme pronunciation rules, rhythm control rules and tone control rules. Therefore, the phoneme pronunciation rules, rhythm control rules and tone control rules can be pre-created for each dialect. Furthermore, after determining the target text and the target dialect, a phoneme sequence with the characteristics of the target dialect can be quickly generated based on the corresponding rules. In some embodiments, dialect data can be collected in advance, and then the pronunciation phonemes, pronunciation rhythm changes, and tone changes unique to each dialect can be filtered and analyzed to determine the pronunciation rules of the phonemes and the control rules of rhythm and tone. Taking a certain dialect as an example, there may be specific pronunciation changes of initials and finals. The pronunciation rules of the phonemes can be used to accurately convert the characters in the text into corresponding phoneme sequences, and then combine them according to the specific rhythm and tone control rules to generate a phoneme sequence with the characteristics of the dialect.

[0053] In the embodiment of the present application, with the help of phoneme pronunciation rules, rhythm control rules and tone control rules, a factor sequence with the characteristics of the target dialect can be quickly generated, and then used as a text feature to enhance the dialect characteristics of the final synthesized speech.

[0054] In some embodiments of the present application, the speech requirement information further includes a character identifier; inputting the text features and the speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect includes:

[0055] The text features and speech requirement information are input into a speech synthesis model to obtain a synthesized speech of a target person reading a target text in a target dialect, wherein the target person includes the person indicated by the person identifier.

[0056] It should be noted that the character identifier is the identifier of the target person, which can be, for example, the target person's name, but is not limited to this. This character identifier can be used to specify the timbre of the synthesized speech. For example, if the speech requirement information includes: Dialect A, Zhang San. Then, the synthesized speech will be the speech data of Zhang San reading the target text in Dialect A. Compared to reading the target text with a fixed timbre, this embodiment can be applied to more application scenarios and meet users' timbre requirements.

[0057] It's understandable that during training, the speech synthesis model uses character and dialect identifiers to indicate the timbre and dialect type of the synthesized speech. Therefore, during model inference, the speech synthesis model outputs synthesized speech of the target character reading the target text in the target dialect, based on text features, character identifiers, and the target dialect identifier.

[0058] In some embodiments, when a speech synthesis model outputs synthesized speech of a target person reading a target text in a target dialect based on text features, a character identifier, and a target dialect identifier, the target person's audio data is first determined based on the character identifier, and then voiceprint features of the audio data are extracted to obtain the target person's voiceprint features. Finally, based on the text features, the character identifier, the target person's voiceprint features, and the target dialect identifier, synthesized speech of the target person reading the target text in the target dialect is output.

[0059] In an embodiment of the present application, the timbre in the synthesized speech can be specified, and the synthesized speech with the timbre of the target person can be output, so that it can be applied to more application scenarios and meet the user's related needs regarding timbre.

[0060] In some embodiments of the present application, the speech requirement information further includes a style identifier; inputting the text features and the speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect includes:

[0061] The text features and speech requirement information are input into a speech synthesis model to obtain synthesized speech that reads the target text in a target dialect under a target style, wherein the target style includes the style indicated by the style identifier.

[0062] It should be noted that the style identifier is an identifier of the target style, for example, it can be the name of the target style, but is not limited thereto. The style here is related to emotion or mood, for example, the style can include happiness, sadness, excitement, tension, etc. The target style can be any style. Through the style identifier, the performance style of the character in the synthesized speech can be specified. For example, the speech requirement information includes: dialect A, tension. Then the synthesized speech will be the speech data of the target text read aloud in dialect A under a nervous mood. Compared with reading the target text in a fixed style, this embodiment can be applicable to more application scenarios and meet the user's relevant needs regarding style.

[0063] It's understood that during training, the speech synthesis model uses style and dialect identifiers to indicate the style and dialect type of the synthesized speech. Therefore, during model inference, the speech synthesis model outputs synthesized speech that reads the target text in the target dialect and in the target style, based on the text features, style identifier, and target dialect identifier.

[0064] In an embodiment of the present application, the style of the character in the synthesized speech can be specified, and the synthesized speech read in the target dialect under the target style can be output, so that it can be applied to more application scenarios and meet the user's related needs regarding style.

[0065] In some embodiments, the speech requirement information includes a character identifier, a style identifier, and a target dialect identifier. By inputting the text features and the speech requirement information into the speech synthesis model, a synthesized speech of the target character reading the target text in the target dialect under the target style can be obtained. For example Figure 2 As shown, the speech synthesis model includes: a dialect synthesis front end 201, a large language model 202, an encoder 203, and a decoder 204.

[0066] After the text data and dialect label are input into the dialect synthesis front-end 201, the front-end 201 searches for the phoneme pronunciation rules, prosody control rules, and pitch control rules corresponding to the target dialect indicated by the dialect label. Based on these rules, the front-end 201 converts the text data into a target phoneme sequence that reflects the characteristics of the target dialect.

[0067] After the emotion label, timbre label, dialect label and target phoneme sequence are input into the large language model 202, the large language model 202 will generate a prosodic code. The generation process of the prosodic code is described in the above embodiment and will not be repeated here.

[0068] After the emotion label, timbre label, dialect label, target phoneme sequence and prosody code are input to the encoder 203, the encoder 203 will generate target prior information based on the emotion label, timbre label, dialect label and target phoneme sequence. The encoder 203 is similar to the encoding module in the above embodiment, and the process of generating the target prior information is also similar, which will not be repeated here. The encoder 203 splices the generated target prior information with the prosody code to obtain a coding feature, and then inputs the coding feature into the decoder 204.

[0069] After receiving the encoded features, decoder 204 performs speech decoding based on the encoded features to generate synthesized speech, i.e., synthesized speech in which the target person reads the target text in the target style and in the target dialect. The emotion tag is equivalent to the style identifier in the above-mentioned embodiment, the timbre tag is equivalent to the person identifier in the above-mentioned embodiment, and the dialect tag is equivalent to the dialect label in the above-mentioned embodiment. The target person is the person indicated by the timbre tag, the target style is the emotion or style indicated by the emotion tag, and the target dialect is the dialect indicated by the dialect tag.

[0070] See also Figure 3 According to another aspect of the present application, in an exemplary embodiment, a method for training a speech synthesis model is provided, which may include:

[0071] S301: Obtain text features of sample data and voice demand information; wherein the voice demand information comprises an identification of a target dialect.

[0072] S302: Train a voice synthesis model according to the text features and the voice demand information.

[0073] S303: Update model parameters of the voice synthesis model based on model loss.

[0074] The voice synthesis model comprises an encoding module, a dialect discrimination module, and a decoding module. The encoding module is configured to generate initial prior information according to the text features and the voice demand information. The dialect discrimination module is configured to generate decision information according to the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is further configured to generate target prior information according to the text features, the voice demand information, and the decision information, and input the target prior information into the decoding module. The decoding module is configured to generate synthesized voice according to the target prior information, and the synthesized voice comprises synthesized voice of the sample data read by the target dialect.

[0075] It should be noted that S101-S102 in the above embodiments are processing procedures in the model inference process, and S301-S302 in the present embodiment are processing procedures in the model training process, and the similarities will not be repeated. In the model training process, the sample data used is pre-collected data for training the voice synthesis model. In addition, in the model training scenario, the model parameters need to be updated in reverse using the model loss, which will not be described in detail here.

[0076] In the present embodiment, the voice synthesis model comprises an encoding module, a dialect discrimination module, and a decoding module. The dialect discrimination module can output decision information to the encoding module, and then correct or improve the quality of the initial prior information output by the encoding module by means of the decision information, to obtain target prior information with more dialect characteristics. Finally, the decoding module generates synthesized voice based on the target prior information, so that the dialect characteristics of the synthesized voice are more prominent, and the effect of the synthesized dialect is improved. Therefore, the voice synthesis model trained by the present embodiment has better dialect voice synthesis effect.

[0077] In some embodiments of the present application, sample data can be collected in the following ways. Collecting Mandarin data: by collecting voice data on the Internet, including data from sources such as radio stations, documentaries, and open source data sets. Collecting dialect data: expanding the dialect corpus through various means, such as cooperating with local cultural institutions to collect folk dialect voice materials; using crowdsourcing platforms to invite volunteers to record dialect voices, etc. For the collected voice data, score the voice data through MosNet, and filter out voice data with scores higher than the threshold, for example, filter out voice data with scores lower than 3.5 points. Use speaker checking technology to determine how many speakers there are in each sentence, and more than one person will be filtered out. For the filtered voice data, generate text corresponding to the voice through the language recognition system ASR.

[0078] In some embodiments, speaker clustering models can also be used to obtain data with speaker information. This allows for determining which segments of unlabeled speech data correspond to the same speaker. Furthermore, for each speaker, voiceprint features can be extracted from their speech data.

[0079] Collect a large amount of Mandarin text corpus containing various styles, assign a corresponding style label to each text sample (the text converted by ASR mentioned above), and establish a mapping relationship between text content and style labels. You can also use open source models to extract unsupervised Hubertian features from the collected Mandarin and dialect data and use them for model pre-training.

[0080] In some embodiments of the present application, the speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate prosody coding of the sample data in the target dialect according to text features and speech requirement information;

[0081] The encoding module inputs the target prior information into the decoding module, including:

[0082] The encoding module concatenates the prosody code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0083] It should be noted that prosodic coding refers to the prosodic information of the sample data in the target dialect. Combining the target prior information with the prosodic coding yields intermediate features that are more consistent with the dialect's characteristics, known as coding features. Furthermore, synthesized speech generated using these coding features exhibits a more pronounced dialectal character.

[0084] In some embodiments, the prosody encoding module can be, for example, a large language model. This large language model is trained separately to output prosody encoding based on input text features and dialect identifiers. For example, a GPT model based on the Transformer architecture is used as the base model. By pre-training on large-scale data, extensive language knowledge is acquired, and fine-tuning is performed to adapt to specific tasks, achieving outstanding performance. The training process is as follows:

[0085] Phase 1: Baseline Mandarin model training.

[0086] The model is trained using Mandarin data so that the trained model can output prosodic coding according to the style label. In the label coding process, the voiceprint features of the sample data can be encoded into a coding module of 192*1024 size; the style label of the sample data can be encoded into a coding module of 10*1024 size. In the word segmentation coding process: the pre-trained Tokenizer can be used to segment and encode the input sequence and convert it into an input format acceptable to the model. The input sequence can be a phoneme sequence of the sample data or a phoneme sequence with dialect characteristics generated by the above-mentioned dialect synthesis front end. In the process of updating the model parameters based on the model loss, the model loss can be generated based on the prosodic coding guidance generated by the pre-trained codec model. The codec model can generate synthesized speech in a specified dialect based on text data, and its intermediate features include prosodic coding.

[0087] Phase 2: Fine-tuning and training of high-quality Mandarin and dialect data models.

[0088] Using high-quality Mandarin data as samples, we continue training based on the first-stage model. During the label encoding process, we add an additional encoding module for the dialect label, which can be 100*1024. The remaining training process is essentially the same as the first stage and will not be repeated here.

[0089] The third stage: fine-tuning and training of high-quality Mandarin data and dialect data models.

[0090] Using high-quality Mandarin data as samples, we continue training based on the second-stage model. During the label encoding process, we add an additional encoding module for the timbre label, which can be 100*1024. The remaining training process is essentially the same as the first stage and will not be repeated here.

[0091] It is understandable that, depending on business needs or voice demand information, only the first and second stages of training may be performed during the training process. Alternatively, the first, second, and third stages may be performed during the training process. It is worth noting that the training data used during the training process includes Mandarin data, dialect data, timbre labels, style labels, dialect labels, voiceprint features, text features, etc., and varies in each stage, so I will not elaborate on this here.

[0092] In the embodiment of the present application, a prosody coding module dedicated to providing prosody coding is provided in the speech synthesis model. Through the processing of the prosody coding model, the dialect characteristics of the synthesized speech can be further enhanced.

[0093] In some embodiments of the present application, training a speech synthesis model based on text features and speech demand information includes:

[0094] The initial speech synthesis model is trained based on the Mandarin data to obtain an intermediate speech synthesis model, which is used to generate speech data corresponding to the Mandarin data;

[0095] The intermediate speech synthesis model is trained according to the text features and speech demand information to obtain a trained speech synthesis model.

[0096] It should be noted that the speech synthesis model has the basic ability to generate synthesized speech for text. Since dialect data is difficult to organize and obtain. This embodiment will use easily available Mandarin data to train the speech synthesis model so that it has basic capabilities. How to train the model so that it has the ability to generate corresponding synthesized speech based on the input text will not be described in detail here. In this embodiment, a speech synthesis model with basic capabilities, that is, an intermediate speech synthesis model, is used as the basis. Sample data is then used for training so that the trained speech synthesis model can generate dialect speech, that is, synthesized speech that reads text in dialect. In some embodiments, the network architecture of the initial speech synthesis model can be the same as the network architecture of the SOVITS model, but is not limited to this.

[0097] In the embodiment of the present application, a multi-stage training method is adopted. First, the basic capabilities of the model are trained using easily available Mandarin data, and then dialect data is used as samples for training on this basis. This can improve the efficiency of model training and shorten the model training time.

[0098] In some embodiments of the present application, an initial speech synthesis model is trained based on Mandarin data to obtain an intermediate speech synthesis model, including:

[0099] The initial speech synthesis model is trained according to the Mandarin data and the style identifier, to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin and the style identifier. The speech data corresponding to the Mandarin and the style identifier includes speech data of reading the Mandarin data in the style indicated by the style identifier.

[0100] It should be noted that, regarding the style identifier, the same description as described above in the embodiments of the speech synthesis method can be referred to, and details are not described herein. The style identifier is added in the synthesized speech output by the speech synthesis model, even if the model has the ability to generate speech according to the specified style. The style identifier needs to be added to indicate the model. In the model training process, due to the limited dialect data and the difficulty in obtaining the data. Therefore, when training the intermediate speech synthesis model by using the Mandarin data, the ability of the model to generate speech according to the specified style can be trained. The Mandarin data in the training process can be the Mandarin data provided in the above embodiments, but is not limited thereto.

[0101] In the embodiments of the present application, the ability of the model to generate speech according to the specified style on the easily obtained Mandarin data can be applied synchronously on the dialect data without the style label, thereby reducing the complexity of the model training.

[0102] In some embodiments of the present application, the speech requirement information further includes a style identifier; the intermediate speech synthesis model is trained according to the text feature and the speech requirement information, to obtain the trained speech synthesis model, including:

[0103] The intermediate speech synthesis model is trained according to the text feature, the identifier of the target dialect, and the style identifier, to obtain the trained speech synthesis model.

[0104] It should be noted that, although the intermediate speech synthesis model has the ability to generate speech according to the specified style, it does not involve dialect information. Therefore, it is necessary to add dialect information to enable it to generate speech according to the specified dialect. In the training process of this stage, the style identifier is continuously added, which will greatly improve the ability of the model to generate speech according to the specified style.

[0105] In the embodiments of the present application, the ability of the model to generate speech according to the specified style is improved in two training stages, thereby greatly improving the ability of the model to generate speech according to the style.

[0106] In some embodiments of the present application, the initial speech synthesis model is trained according to the Mandarin data, to obtain the intermediate speech synthesis model, including:

[0107] The initial speech synthesis model is trained according to Mandarin data and voiceprint features to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to Mandarin and voiceprint features. The speech data corresponding to Mandarin and voiceprint features includes: speech data of reading Mandarin data according to voiceprint features.

[0108] It should be noted that voiceprint features can be considered as timbre information, and different voiceprint features can indicate different timbre. Therefore, speech data of Mandarin data read aloud using a voiceprint feature can be considered as speech data of a person reading Mandarin data with the timbre indicated by that voiceprint feature. In this way, the synthesized speech output by the model can be a dialect speech with a specified timbre.

[0109] When a speech synthesis model is required to generate speech with a specified timbre, voiceprint features need to be added to the model to provide guidance. During model training, dialect data is limited and difficult to obtain. Therefore, when training the intermediate speech synthesis model using Mandarin data, the model's ability to generate speech with specified timbre / voiceprint features can be trained. The Mandarin data used in the training process can be, but is not limited to, the Mandarin data provided in the aforementioned embodiments.

[0110] In the embodiment of the present application, the ability of the model to generate speech according to a specified timbre is trained on easily accessible Mandarin data, and can be applied simultaneously to dialect data without timbre labels, thereby reducing the complexity of model training.

[0111] In some embodiments of the present application, the speech demand information further includes a character identifier; and training an intermediate speech synthesis model based on the text features and the speech demand information to obtain a trained speech synthesis model includes:

[0112] An intermediate speech synthesis model is trained according to text features, voiceprint features, target dialect identification, and character identification to obtain a trained speech synthesis model.

[0113] It should be noted that while the intermediate speech synthesis model is capable of generating speech with a specified timbre, the voiceprint's ability to control timbre is generally limited. Therefore, it is necessary to add a character identifier to the intermediate speech synthesis model. For details on character identifiers, please refer to the same description of the speech synthesis method embodiment above and will not be repeated here. During this stage of training, adding a character identifier to control timbre will significantly improve the model's ability to generate speech with a specified timbre.

[0114] In the embodiment of the present application, the model's ability to generate speech according to the specified timbre is improved in both training stages, thereby greatly improving the model's ability to generate speech according to the timbre.

[0115] To facilitate understanding, the following example illustrates the model training process, showing how the model acquires and improves its ability to generate speech based on timbre, style, and dialect.

[0116] The training process is as follows:

[0117] Phase 1: Mandarin basic model training.

[0118] First, select the model's network architecture, for example, the SOVITS model. Then, perform model training based on the Mandarin data, voiceprint features, style labels, and text features collected in the aforementioned embodiments. During the encoding process during training, the voiceprint features are encoded into a 192*1024 encoding module; the style labels are encoded into a 10*1024 encoding module, and then converted into an acceptable input format for the model. Regarding model loss, the audio adversarial loss makes the speech generated by the generator in the network architecture more similar to real speech, improving the quality and naturalness of the synthesized speech. Feedback from the discriminator in the network architecture helps the generator learn the characteristics of real speech, thereby continuously improving the synthesis effect. The feature matching loss encourages the generator to learn the feature distribution of real speech at different levels, preventing the generator from focusing too much on the final output and neglecting the learning of intermediate processes. This helps to improve the detail and authenticity of the synthesized speech. The Mel-Spectrogram Loss calculates the difference between the mel-spectrum of the synthesized speech and the real speech. The model can learn the frequency characteristics of real speech, improving the timbre and sound quality of the synthesized speech. In some embodiments, the training data can be divided into small batches for iterative training to improve training efficiency and stability. In addition, when evaluating the performance of the model on the validation set, the similarity between the original recording and the generated speech can be measured to check the accuracy of the encoding generated by the model.

[0119] Phase 2: Multi-dialect model training.

[0120] Based on the dialect data, voiceprint features, style labels, dialect labels and text features collected in the aforementioned embodiments, the model trained in the first stage is further trained. In this stage, a dialect discriminator is added to the original network architecture, thereby introducing a coding adversarial loss, so that the code generated by the generator is closer to the code sequence that matches the dialect features. The structure of the dialect discriminator is basically similar to that of SOVITS, except that the multi-resolution discriminator is removed and the rest is retained. In this way, the original audio signal passes through a 6-layer 1D convolution module, is converted to 2D, and then passes through a 2D, 5-layer multi-cycle convolution module (the scale of the multi-cycle convolution is 2, 3, 5, 7, 11). The returned result can include two categories: the final generated judgment information and the output feature judgment results of each layer. In some embodiments, the training data is divided into small batches and iterative training is performed to improve training efficiency and stability. When evaluating the performance of the model on the validation set, the similarity between the original recording and the generated speech is measured to check the accuracy of the code generated by the model. After training, specifying a style label can generate speech of the corresponding style; specifying a voiceprint feature can generate speech with the timbre characteristics of the voiceprint, and the style label and voiceprint feature can be combined to control the generated speech.

[0121] The third stage: fine-tuning training of multi-dialect models, using a small amount of high-quality target speaker data and some high-quality dialect data for fine-tuning training.

[0122] Based on the data collected in the aforementioned embodiments, a small amount of high-quality target speaker data and some high-quality dialect data, voiceprint features, timbre labels, dialect labels and text features, the model after the second stage training is further trained / fine-tuned. During the training process, the parameters of the style encoding module are fixed; the timbre label encoding module size of 100*1024 is added on the basis of the initial model. Since the speaker timbre control based on voiceprint is weak, in the fine-tuning stage, the speaker label is adjusted to better control the target speaker timbre. In some embodiments, the training data can be divided into small batches for iterative training to improve training efficiency and stability. When evaluating the performance of the model on the validation set, the similarity between the original recording and the generated speech is measured to check the accuracy of the model generation coding; the generated speech can be controlled in combination by the style label and timbre label. In some embodiments, the intermediate features of the trained model include prosody coding, which can be used to guide the training of the large language model in the above embodiments.

[0123] It should be noted that although technology is constantly advancing, there is still a gap in naturalness between synthesized dialectal speech and real human pronunciation. There may be mechanical feeling, incoherence or lack of emotional expression, and it is difficult to completely simulate the rich changes of human speech. Especially in expressing complex emotions and subtle changes in tone, synthesized speech often appears stiff. In order to improve the accuracy and naturalness of multi-dialectal speech synthesis, the key technology of speech synthesis is improved, in the aspect of prosody modeling, a modeling method based on dialectal prosodic features is adopted to generate more natural dialectal speech. In the aspect of emotion and pragmatic information modeling, the dialectal speech synthesis can express emotion and pragmatic information, and enhance the expressiveness of the speech. A multi-stage training method is adopted to learn the prosodic characteristics of each dialect, and the model is pre-trained through massive data to obtain pre-trained model parameters, which can learn the basic natural pronunciation prosody. Then, a small amount of high-quality data annotated by artificial labeling is used to optimize the model parameters to obtain an optimized model, so that the pronunciation and prosodic characteristics of each dialect can be accurately learned.

[0124] In the embodiments of the present application, a multi-stage training scheme is adopted, a baseline Mandarin model can learn basic language information on large-scale data; a high-quality Mandarin and dialect data model adds dialect data and continues to train on higher quality data to ensure model stability; fine data fine-tuning training can learn the pronunciation details of the target speaker more finely to ensure the synthesis effect. For tone control, first control the tone based on voiceprint, and then use speaker information to control the tone in the fine data fine-tuning training model stage, so that the tone in the synthesized speech can be completely controlled, and the problem of tone control in large data training is solved. In addition, dialect / dialect category information is added at the encoding level to ensure that the generated code is more consistent with the language characteristics of the dialect.

[0125] In some embodiments of the present application, the text features include: a target phoneme sequence of the sample data with target dialect characteristics.

[0126] It should be noted that after collecting various data in the above-mentioned embodiments, the collected data can be used to create a customized front end, i.e., a dialect synthesis front end. The dialect label can be specified, and the front end engine outputs text features conforming to the dialect according to the dialect label, including phoneme, prosody and pitch information.

[0127] In some embodiments, customized mapping dictionaries can be constructed based on the phonemes, tones, and rhythms of different dialects. By collecting a small amount of dialect data, all the pronunciation characteristics of that dialect can be covered. By combining manual and automatic annotation, accurate customized dictionaries for different dialects can be obtained at a low manual annotation cost. Each dialect-specific dictionary can then be used to build a front-end for each dialect, resulting in a dialect synthesis front-end. For example, dialect data can be filtered and analyzed to identify the unique pronunciation phonemes, rhythmic variations, and tonal variations of each dialect, thereby determining the pronunciation rules, rhythmic, and tonal control rules for each dialect. Based on the dialect and its corresponding pronunciation rules and control rules, a dialect synthesis front-end is created. When a text is input into the dialect synthesis front-end and the dialect is specified, a phoneme sequence with the characteristics of that dialect can be output. For example, using the pronunciation rules of a particular dialect, the characters in the text can be accurately converted into corresponding phoneme sequences. These characters are then combined according to specific rhythmic and tonal control rules to generate a phoneme sequence with the characteristics of that dialect.

[0128] In the embodiment of the present application, the text data also has dialect characteristics, which can further improve the effect of the model's final synthesis of dialect speech.

[0129] Exemplary devices

[0130] Accordingly, the present invention also provides a speech synthesis device, see Figure 4 As shown, the device includes:

[0131] The first acquisition module 401 is used to acquire text features and speech requirement information of the target text; wherein the speech requirement information includes an identifier of the target dialect;

[0132] The speech module 402 is used to input text features and speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect;

[0133] Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information based on text features and speech demand information. The dialect discrimination module is used to generate decision information based on the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information based on the text features, speech demand information and decision information, and input the target prior information into the decoding module. The decoding module is used to generate synthesized speech based on the target prior information.

[0134] In some embodiments, the speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate a prosody coding of the target text in the target dialect based on text features and speech requirement information;

[0135] The encoding module inputs the target prior information into the decoding module, including:

[0136] The encoding module concatenates the prosody code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0137] In some embodiments, the text features include: a target phoneme sequence of the target text having characteristics of the target dialect.

[0138] In some embodiments, the first acquisition module 401 is specifically used to convert the target text into an initial phoneme sequence based on the phoneme pronunciation rules of the target dialect; and to arrange and combine the phonemes in the initial phoneme sequence based on the prosody control rules and / or tone control rules of the target dialect to generate a target phoneme sequence with the characteristics of the target dialect.

[0139] In some embodiments, the speech demand information also includes a character identification; the speech module 402 is specifically used to input text features and speech demand information into a speech synthesis model to obtain a synthesized speech of the target person reading the target text in the target dialect, wherein the target person includes the person indicated by the character identification.

[0140] In some embodiments, the speech requirement information also includes a style identifier; the speech module 402 is specifically used to input text features and speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in a target dialect under a target style, wherein the target style includes the style indicated by the style identifier.

[0141] The speech synthesis device provided in this embodiment is based on the same concept as the speech synthesis method provided in the above embodiments of this application. It can execute the speech synthesis method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects. For technical details not fully described in this embodiment, please refer to the specific processing content of the speech synthesis method provided in the above embodiments of this application and will not be repeated here.

[0142] Accordingly, the present application also provides a training device for a speech synthesis model, see Figure 5 As shown, the device includes:

[0143] The second acquisition module 501 is used to obtain text features and speech requirement information of the sample data; wherein the speech requirement information includes the identification of the target dialect;

[0144] A training module 502 is used to train a speech synthesis model based on text features and speech demand information;

[0145] An updating module 503, configured to update model parameters of the speech synthesis model based on the model loss;

[0146] Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information based on text features and speech demand information. The dialect discrimination module is used to generate decision information based on the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information based on the text features, speech demand information and decision information, and input the target prior information into the decoding module. The decoding module is used to generate synthesized speech based on the target prior information. The synthesized speech includes synthesized speech by reading sample data in the target dialect.

[0147] In some embodiments, the speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate prosody coding of the sample data in the target dialect based on text features and speech requirement information;

[0148] The encoding module inputs the target prior information into the decoding module, including:

[0149] The encoding module concatenates the prosody code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0150] In some embodiments, the training module 502 includes:

[0151] A first training unit is used to train the initial speech synthesis model based on the Mandarin data to obtain an intermediate speech synthesis model, and the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin data;

[0152] The second training unit is used to train the intermediate speech synthesis model according to the text features and speech demand information to obtain a trained speech synthesis model.

[0153] In some embodiments, the first training unit is specifically used to train an initial speech synthesis model based on Mandarin data and its style identifier to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to Mandarin and its style identifier. The speech data corresponding to Mandarin and its style identifier includes: speech data of reading Mandarin data in the style indicated by the style identifier.

[0154] In some embodiments, the speech requirement information also includes a style identifier; the second training unit is specifically used to train the intermediate speech synthesis model according to the text features, the identifier of the target dialect and the style identifier to obtain a trained speech synthesis model.

[0155] In some embodiments, the first training unit is specifically configured to train the initial speech synthesis model according to the Mandarin data and the voiceprint feature to obtain an intermediate speech synthesis model, and the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin and the voiceprint feature, and the speech data corresponding to the Mandarin and the voiceprint feature includes speech data of reading the Mandarin data according to the voiceprint feature.

[0156] In some embodiments, the voice demand information further includes a character identifier; and the second training unit is specifically configured to train the intermediate speech synthesis model according to the text feature, the voiceprint feature, the identifier of the target dialect, and the character identifier to obtain the trained speech synthesis model.

[0157] In some embodiments, the text feature includes a target phoneme sequence of the sample data with the target dialect characteristics.

[0158] The training device of the speech synthesis model provided in the embodiment belongs to the same application concept as the training method of the speech synthesis model provided in the above embodiments of the application, can execute the training method of the speech synthesis model provided in any of the above embodiments of the application, and has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the embodiment can be referred to the specific processing content of the training method of the speech synthesis model provided in the above embodiments of the application, which will not be described here.

[0159] It should be understood that the modules in the above device can be implemented in the form of processor calling software. For example, the device includes a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of the units of the device, wherein the processor can be a general processor such as CPU or microprocessor, and the memory can be an internal memory or an external memory of the device. Alternatively, the units in the device can be implemented in the form of hardware circuit. The functions of part or all of the units can be implemented by designing the hardware circuit. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is ASIC, and the functions of part or all of the units are implemented by designing the logical relationship of elements in the circuit. For another example, in another implementation, the hardware circuit can be implemented by PLD. Taking FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to implement the functions of part or all of the units. All units of the above device can be implemented in the form of processor calling software, or implemented in the form of hardware circuit, or partially implemented in the form of processor calling software and the remaining part implemented in the form of hardware circuit.

[0160] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP. In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0161] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0162] In addition, the various units in the above apparatus may be fully or partially integrated together, or may be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the apparatus. The at least one processor may be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0163] Exemplary electronic devices

[0164] The present application embodiment provides an electronic device, see Figure 6 As shown, the device includes:

[0165] Memory 600 and processor 610;

[0166] The memory 600 is connected to the processor 610 and is used to store programs;

[0167] The processor 610 is configured to implement the speech synthesis method or speech synthesis model training method disclosed in any one of the above embodiments by running the program stored in the memory 600 .

[0168] Specifically, the electronic device may further include: a bus, a communication interface 620 , an input device 630 and an output device 640 .

[0169] The processor 610, the memory 600, the communication interface 620, the input device 630 and the output device 640 are connected with each other through a bus. Among them:

[0170] The bus can include a path for transmitting information between various components of the computer system.

[0171] The processor 610 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-use programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0172] The processor 610 can include a main processor, and can also include a baseband chip, a modem, etc.

[0173] The memory 600 stores programs for executing the technical solutions of the present application, and can also store operating systems and other key services. Specifically, the program can include program code, and the program code includes computer operation instructions. More specifically, the memory 600 can include read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash, etc.

[0174] The input device 630 can include a device that receives data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer or a gravity sensor, etc.

[0175] The output device 640 can include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0176] The communication interface 620 can include a device using any transceiver, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc., to communicate with other devices or communication networks.

[0177] The processor 610 executes the program stored in the memory 600, and calls other devices, which can be used to implement each step of any one of the speech synthesis methods or the training methods of the speech synthesis model provided by the embodiments of the present application.

[0178] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs the program stored in the memory through the data interface to execute the speech synthesis method or the speech synthesis model training method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the embodiment introduction of the above-mentioned speech synthesis method or the speech synthesis model training method.

[0179] Exemplary computer program products and storage media

[0180] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the speech synthesis method or speech synthesis model training method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0181] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0182] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the speech synthesis method or speech synthesis model training method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0183] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0184] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0185] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0186] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0188] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0189] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0190] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0191] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0192] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0193] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquiring text features and speech requirement information of a target text; wherein the speech requirement information includes an identifier of a target dialect; Inputting the text features and the speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect; Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information based on the text features and the speech requirement information. The dialect discrimination module is used to generate decision information based on the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information based on the text features, the speech requirement information and the decision information, and input the target prior information into the decoding module. The decoding module is used to generate the synthesized speech based on the target prior information.

2. The method according to claim 1, characterized in that The speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate a prosody code of the target text in the target dialect according to the text features and the speech requirement information; The encoding module inputs the target prior information into the decoding module, including: The encoding module concatenates the prosody code and the target prior information to obtain encoding features, and inputs the encoding features into the decoding module.

3. The method according to claim 1, characterized in that The text features include: a target phoneme sequence of the target text having characteristics of a target dialect.

4. The method according to claim 3, characterized in that The acquiring of text features of the target text includes: Based on the phoneme pronunciation rules of the target dialect, converting the target text into an initial phoneme sequence; Based on the prosody control rules and / or tone control rules of the target dialect, the phonemes in the initial phoneme sequence are arranged and combined to generate a target phoneme sequence with the characteristics of the target dialect.

5. The method according to claim 1, wherein The speech requirement information further includes a character identifier; inputting the text features and the speech requirement information into a speech synthesis model to obtain a synthesized speech that reads the target text in the target dialect, including: The text features and the speech requirement information are input into a speech synthesis model to obtain a synthesized speech of a target person reading the target text in the target dialect, wherein the target person includes the person indicated by the person identifier.

6. The method according to claim 1, characterized in that The speech requirement information further includes a style identifier; inputting the text features and the speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect, including: The text features and the speech requirement information are input into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect under a target style, wherein the target style includes the style indicated by the style identifier.

7. A method for training a speech synthesis model, characterized in that: The method comprises: Acquiring text features and speech requirement information of sample data; wherein the speech requirement information includes an identifier of the target dialect; Training a speech synthesis model according to the text features and the speech demand information; Updating model parameters of the speech synthesis model based on the model loss; Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information based on the text features and the speech demand information. The dialect discrimination module is used to generate decision information based on the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information based on the text features, the speech demand information and the decision information, and input the target prior information into the decoding module. The decoding module is used to generate synthesized speech based on the target prior information. The synthesized speech includes synthesized speech that reads the sample data in the target dialect.

8. The method according to claim 7, characterized in that The speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate a prosody coding of the sample data in the target dialect according to the text features and the speech requirement information; The encoding module inputs the target prior information into the decoding module, including: The encoding module concatenates the prosody code and the target prior information to obtain encoding features, and inputs the encoding features into the decoding module.

9. The method according to claim 7 or 8, characterized in that Training a speech synthesis model according to the text features and the speech demand information includes: Training an initial speech synthesis model based on Mandarin data to obtain an intermediate speech synthesis model, wherein the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin data; The intermediate speech synthesis model is trained according to the text features and the speech demand information to obtain a trained speech synthesis model.

10. The method according to claim 9, characterized in that The initial speech synthesis model is trained based on Mandarin data to obtain an intermediate speech synthesis model, including: An initial speech synthesis model is trained based on Mandarin data and its style identifier to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin and its style identifier. The speech data corresponding to the Mandarin and its style identifier includes: speech data of reading the Mandarin data in the style indicated by the style identifier.

11. The method according to claim 10, characterized in that The speech requirement information further includes a style identifier; and training the intermediate speech synthesis model according to the text features and the speech requirement information to obtain a trained speech synthesis model includes: The intermediate speech synthesis model is trained according to the text features, the identifier of the target dialect and the style identifier to obtain a trained speech synthesis model.

12. The method according to claim 9, characterized in that The initial speech synthesis model is trained based on Mandarin data to obtain an intermediate speech synthesis model, including: An initial speech synthesis model is trained based on Mandarin data and voiceprint features to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin and voiceprint features. The speech data corresponding to the Mandarin and voiceprint features includes: speech data that reads the Mandarin data according to the voiceprint features.

13. The method according to claim 12, characterized in that The speech demand information further includes a character identifier; and training the intermediate speech synthesis model according to the text features and the speech demand information to obtain a trained speech synthesis model includes: The intermediate speech synthesis model is trained according to the text features, the voiceprint features, the identifier of the target dialect and the character identifier to obtain a trained speech synthesis model.

14. The method according to claim 7, wherein: The text features include: a target phoneme sequence of the sample data having characteristics of a target dialect.

15. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 14 by running the program in the memory.

16. A computer program product, characterized in that include: A computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images