Speech synthesis method, speech synthesis model training method, electronic device and computer program product

Abstract

The invention provides a speech synthesis method, a speech synthesis model training method, electronic equipment and a computer program product. The text features of the target text and the voice demand information can be input into a voice synthesis model to generate a synthetic voice of reading the target text through the target dialect. Wherein the speech synthesis model comprises an encoding module, a dialect judgment module and a decoding module, the dialect judgment module can output judgment information to the encoding module, and then initial prior information output by the encoding module is corrected or the quality of the initial prior information is improved by means of the judgment information; and target prior information with more dialect characteristics is obtained. And finally, the decoding module generates the synthetic speech based on the target prior information, so that the dialect characteristic of the synthetic speech is more remarkable, and the dialect synthesis effect is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, a training method for a speech synthesis model, an electronic device, and a computer program product. Background Art

[0002] In a country with a vast territory and many ethnic groups, people are communicating more and more frequently, and dialects from different regions have become an important element in communication. The emergence of speech synthesis technology involving dialects has provided people with a more convenient and natural way of interaction. Among them, dialects are also called "local dialects" or "local sounds", which refer to the language of a certain region that is different from the standard language.

[0003] At present, a trained speech synthesis model can usually be used to generate synthesized speech in a dialect. During the training phase, the speech synthesis model needs to be trained with a large amount of dialect speech to learn the unique speech characteristics and grammatical structure of each dialect.

[0004] However, there are usually many types of dialects. According to statistics, some countries with vast territories may have more than 100 dialects, which makes it extremely complicated to train a speech synthesis model that covers various dialects. For example, the tones, finals, and initial consonants of different dialects are very different, which requires a large amount of data and complex algorithms to process. Limited by training data and algorithms, most current speech synthesis models use conventional codecs to synthesize dialect speech, resulting in mediocre results in synthesizing dialects and insufficient dialect characteristics. Summary of the invention

[0005] Based on the above technical status, the present application proposes a speech synthesis method, a training method of a speech synthesis model, an electronic device and a computer program product.

[0006] According to a first aspect of an embodiment of the present application, a speech synthesis method is provided, the method comprising:

[0007] Acquire text features and speech requirement information of a target text; wherein the speech requirement information includes an identifier of a target dialect;

[0008] Inputting the text features and the speech requirement information into a speech synthesis model to obtain a synthesized speech that reads the target text in the target dialect;

[0009] Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module, the encoding module is used to generate initial prior information according to the text features and the speech requirement information, the dialect discrimination module is used to generate decision information according to the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information according to the text features, the speech requirement information and the decision information, and input the target prior information into the decoding module, and the decoding module is used to generate the synthesized speech according to the target prior information.

[0010] According to a second aspect of an embodiment of the present application, a method for training a speech synthesis model is provided, the method comprising:

[0011] Acquire text features and speech demand information of sample data; wherein the speech demand information includes an identifier of a target dialect;

[0012] Training a speech synthesis model according to the text features and the speech demand information;

[0013] Updating model parameters of the speech synthesis model based on the model loss;

[0014] Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module, the encoding module is used to generate initial prior information according to the text features and the speech demand information, the dialect discrimination module is used to generate decision information according to the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information according to the text features, the speech demand information and the decision information, and input the target prior information into the decoding module, and the decoding module is used to generate synthesized speech according to the target prior information, and the synthesized speech includes synthesized speech that reads the sample data in the target dialect.

[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the speech synthesis method as described in the first aspect or the speech synthesis model training method as described in the second aspect by running the program in the memory.

[0016] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method as described in the first aspect or the training method of the speech synthesis model as described in the second aspect is implemented.

[0017] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the speech synthesis method as described in the first aspect or the training method of the speech synthesis model as described in the second aspect.

[0018] In an embodiment of the present application, the text features and speech demand information of the target text are input into a speech synthesis model, and a synthesized speech that reads the target text in a target dialect can be generated. Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module, and a decoding module. The dialect discrimination module can output judgment information to the encoding module, and then use the judgment information to correct the initial prior information output by the encoding module or improve the quality of the initial prior information, so as to obtain target prior information with more dialect characteristics. Finally, the decoding module generates a synthesized speech based on the target prior information, so that the dialect characteristics of the synthesized speech are more significant, thereby improving the effect of synthesizing the dialect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the structure of the speech synthesis model provided in the embodiment of the present application;

[0022] Figure 3 A flowchart of a method for training a speech synthesis model provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of a training device for a speech synthesis model provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Overview

[0028] As described in the background technology, after the emergence of speech synthesis technology involving dialects, it is possible to output a synthesized speech that reads text in dialects based on text output, that is, dialect speech. At present, there are mainly three ways to synthesize dialect speech. Method 1, synthesis based on unit splicing: the dialect speech is divided into several basic units, such as phonemes, syllables or words. Then, according to the input text, appropriate units are selected from the pre-recorded unit library for splicing to synthesize the speech waveform. Method 2, synthesis based on statistical parameters: by analyzing a large amount of dialect speech data, an acoustic model and a language model are established. The acoustic model is used to predict the acoustic features of speech, such as fundamental frequency, spectrum, etc.; the language model is responsible for processing the linguistic information of the text, such as grammar, semantics, etc. Then, according to the input text, the two models are used to generate the corresponding acoustic feature parameters, and then these parameters are converted into speech waveforms through the synthesizer. Method 3, end-to-end synthesis based on deep learning: a deep neural network is used to directly convert the input text into a speech waveform without the need for an intermediate acoustic feature parameter conversion process. Among them, the processing logic of method 3 is relatively simple, and only the corresponding samples need to be collected for model training.

[0029] However, since there are many types of dialects, and each dialect has unique phonetic characteristics and grammatical structures, the effect of synthesizing dialects using traditional neural network models (such as recurrent neural networks, long short-term memory networks, etc.) for training is generally mediocre, and the dialect characteristics are not obvious enough.

[0030] In view of the above-mentioned technical status, the inventor proposes that a discriminator can be introduced into the neural network model, so as to correct the coded data in the model or improve the quality of the coded data with the help of the signal fed back by the discriminator, so that it has more significant dialect characteristics. Therefore, an embodiment of the present application proposes a speech synthesis method. The text features and speech demand information of the target text are input into the speech synthesis model, and a synthesized speech that reads the target text in the target dialect can be generated. Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The dialect discrimination module can output judgment information to the encoding module, and then correct the initial prior information output by the encoding module with the help of the judgment information or improve the quality of the initial prior information, so as to obtain target prior information with more dialect characteristics. Finally, the decoding module generates a synthesized speech based on the target prior information, so that the dialect characteristics of the synthesized speech are more significant, and the effect of synthesizing the dialect is improved.

[0031] Exemplary Methods

[0032] See also Figure 1 In an exemplary embodiment, a speech synthesis method is provided, which may include:

[0033] S101: Acquire text features and speech requirement information of a target text; wherein the speech requirement information includes an identifier of a target dialect.

[0034] In this step, the target text is the text that needs to be converted into dialect speech, and the semantic content of the target text is not limited here. For example, the target text can be an article, a speech, or a news report. For another example, the target text can also be a text converted from a piece of speech data.

[0035] The text features of the target text may include pronunciation features of the target text in the target dialect, semantic features of the target text, etc. In some embodiments, the text features include: a target phoneme sequence of the target text with target dialect characteristics. The target dialect characteristics include characteristics or rules of the target dialect in pronunciation, rhythm, and intonation.

[0036] The speech requirement information includes: the requirement for the final synthesized speech. In some embodiments, the requirement for the final synthesized speech includes: the requirement for the dialect type of the final synthesized speech. For example, the requirement may be to synthesize a dialect speech in a target dialect. The target dialect may be a dialect of any region. The identifier of the dialect may be a unique identifier of the dialect. In the process of synthesizing speech, the identifier may be used to determine the dialect speech that needs to be synthesized, which is the dialect speech in the target dialect indicated by the identifier. In some embodiments, the identifier of the dialect may be a dialect label. It is understandable that the speech requirement information includes but is not limited to the requirement for the dialect type of the final synthesized speech.

[0037] S102: Inputting text features and speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect.

[0038] In this step, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information based on text features and speech demand information. The dialect discrimination module is used to generate decision information based on the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information based on the text features, speech demand information and decision information, and input the target prior information into the decoding module. The decoding module is used to generate synthesized speech based on the target prior information.

[0039] It can be understood that the initial prior information is an intermediate representation generated by the module based on text features and speech demand information. The encoding module can be, for example, an encoder in a neural network model, which is used to capture semantic information, grammatical information, context information, dialect information, etc. and convert it into an easy-to-process form, namely, the initial prior information. For example, the network architecture of the encoding module can be the same as the network architecture of the encoding part in the SOVITS model.

[0040] The discriminant module can discriminate the initial prior information generated by the encoding module, thereby determining the quality of the initial prior information. The higher the quality, the more accurate the content represented; conversely, the lower the quality, the greater the deviation of the represented content. By generating judgment information and feeding it back to the encoding module, the initial prior information is corrected to improve the quality of the prior information generated by the encoding module. Here, the encoding module will regenerate higher quality target prior information based on the judgment information. In this embodiment, the more accurate the content represented by the prior information output by the encoding module, the more significant or accurate the dialect characteristics under the target dialect. In some embodiments, the network architecture of the discriminant module can be similar to the network architecture of the SOVITS model. For example, the multi-resolution discriminator of the SOVITS model can be removed and the rest can be retained. In this way, the original audio signal passes through a 6-layer 1D convolution module, is converted to 2D, and then passes through a 2D, 5-layer multi-cycle convolution module (the scale of the multi-cycle convolution is 2, 3, 5, 7, 11,). The returned result can include two categories: the final generated judgment information and the output feature judgment results of each layer.

[0041] The decoding module may be, for example, a decoder in a neural network model that can generate the final synthesized speech based on the intermediate representations output by other modules. In some embodiments, the network architecture of the decoding module may be the same as the network architecture of the decoding part in the SOVITS model.

[0042] In an embodiment of the present application, the text features and speech demand information of the target text are input into a speech synthesis model, and a synthesized speech that reads the target text in a target dialect can be generated. Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module, and a decoding module. The dialect discrimination module can output judgment information to the encoding module, and then use the judgment information to correct the initial prior information output by the encoding module or improve the quality of the initial prior information, so as to obtain target prior information with more dialect characteristics. Finally, the decoding module generates a synthesized speech based on the target prior information, so that the dialect characteristics of the synthesized speech are more significant, thereby improving the effect of synthesizing the dialect.

[0043] In order to further enhance the dialect characteristics of the synthesized speech, in some embodiments of the present application, the speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate a prosody coding of the target text in the target dialect according to the text features and the speech requirement information;

[0044] The encoding module inputs the target prior information into the decoding module, including:

[0045] The encoding module concatenates the prosodic code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0046] It should be noted that the prosodic code is the prosodic information of the target text in the target dialect. After the target prior information is combined with the prosodic code, an intermediate feature that is more in line with the dialect characteristics can be obtained, namely the coding feature. Then, the synthesized speech generated using the coding feature has more obvious dialect characteristics.

[0047] In some embodiments, the prosody encoding module may be, for example, a large language model, which is trained separately so that it can output prosody encoding based on input text features and dialect identification.

[0048] In the embodiment of the present application, a prosody coding module dedicated to providing prosody coding is provided in the speech synthesis model. Through the processing of the prosody coding model, the dialect characteristics of the synthesized speech can be further enhanced.

[0049] In some embodiments of the present application, obtaining text features of the target text includes:

[0050] Based on the phoneme pronunciation rules of the target dialect, the target text is converted into an initial phoneme sequence; based on the rhythm control rules and / or tone control rules of the target dialect, the phonemes in the initial phoneme sequence are arranged and combined to generate a target phoneme sequence with the characteristics of the target dialect.

[0051] It should be noted that the initial phoneme sequence and the target phoneme sequence are sequences composed of phonemes. Among them, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. For example, the Chinese syllable "ā" has only one phoneme, "ài" has two phonemes, "dài" has three phonemes, and so on.

[0052] It can be understood that different dialects have different phoneme pronunciation rules, rhythm control rules, and tone control rules. Therefore, the phoneme pronunciation rules, rhythm control rules, and tone control rules for each dialect can be pre-created. Furthermore, after determining the target text and the target dialect, a phoneme sequence with the characteristics of the target dialect can be quickly generated based on the corresponding rules. In some embodiments, dialect data can be pre-collected, and then the unique pronunciation phonemes, pronunciation rhythm changes, and tone changes of each dialect can be filtered and analyzed to determine the phoneme pronunciation rules, rhythm, and tone control rules. Taking a certain dialect as an example, there may be specific pronunciation changes of initials and finals. Through the phoneme pronunciation rules, the characters in the text can be accurately converted into the corresponding phoneme sequence, and then combined according to the specific rhythm and tone control rules to generate a phoneme sequence with the characteristics of this dialect.

[0053] In the embodiments of the present application, with the help of phoneme pronunciation rules, rhythm control rules, and tone control rules, a factor sequence with the characteristics of the target dialect can be quickly generated. Furthermore, taking this as a text feature can enhance the dialect characteristics of the final synthesized speech.

[0054] In some embodiments of the present application, the voice demand information further includes a person identifier; inputting the text feature and the voice demand information into a voice synthesis model to obtain a synthesized voice that reads the target text in the target dialect, including:

[0055] Inputting the text feature and the voice demand information into a voice synthesis model to obtain a synthesized voice in which the target person reads the target text in the target dialect, where the target person includes the person indicated by the person identifier.

[0056] It should be noted that the person identifier is the identifier of the target person. For example, it can be the name of the target person, but not limited to this. Through this person identifier, the timbre in the synthesized voice can be specified. For example, the voice demand information includes: Dialect A, Zhang San. Then the synthesized voice will be the voice data in which Zhang San reads the target text in Dialect A. Compared with reading the target text with a fixed timbre, this embodiment can be applied to more application scenarios and meet the user's related requirements regarding timbre.

[0057] It is understandable that during the training process, the speech synthesis model indicates the timbre and dialect type in the synthesized speech through the character identifier and the dialect identifier. Therefore, during the model inference process, the speech synthesis model outputs the synthesized speech of the target character reading the target text in the target dialect based on the text features, the character identifier and the target dialect identifier.

[0058] In some embodiments, when the speech synthesis model outputs the synthesized speech of the target person reading the target text in the target dialect according to the text features, the character identification and the identification of the target dialect, the audio data of the target person is first determined according to the character identification, and then the voiceprint features of the audio data are extracted to obtain the voiceprint features of the target person. Finally, based on the text features, the character identification, the voiceprint features of the target person, and the identification of the target dialect, the synthesized speech of the target person reading the target text in the target dialect is output.

[0059] In the embodiment of the present application, the timbre in the synthesized speech can be specified, and the synthesized speech with the timbre of the target person can be output, so that it can be applied to more application scenarios and meet the user's related needs regarding timbre.

[0060] In some embodiments of the present application, the speech requirement information further includes a style identifier; inputting the text features and the speech requirement information into a speech synthesis model to obtain a synthesized speech that reads the target text in a target dialect, including:

[0061] The text features and speech requirement information are input into a speech synthesis model to obtain synthesized speech that reads the target text in a target dialect under a target style, wherein the target style includes a style indicated by a style identifier.

[0062] It should be noted that the style identifier is an identifier of the target style, for example, it can be the name of the target style, but is not limited to this. The style here is related to emotion or mood, for example, the style can include happy, sad, excited, nervous, etc. The target style can be any style. Through the style identifier, the performance style of the character in the synthesized speech can be specified. For example, the speech requirement information includes: dialect A, nervous. Then the synthesized speech will be the speech data of reading the target text in dialect A under a nervous mood. Compared with reading the target text in a fixed style, this embodiment can be applicable to more application scenarios and meet the user's relevant needs regarding style.

[0063] It is understandable that during the training process, the speech synthesis model indicates the style and dialect type in the synthesized speech through the style identifier and the dialect identifier. Therefore, during the model inference process, the speech synthesis model outputs the synthesized speech that reads the target text in the target dialect under the target style according to the text features, the style identifier and the identifier of the target dialect.

[0064] In the embodiment of the present application, the style of the character in the synthesized speech can be specified, and the synthesized speech read in the target dialect under the target style can be output, so that it can be applied to more application scenarios and meet the user's related needs regarding style.

[0065] In some embodiments, the speech requirement information includes a character identifier, a style identifier, and an identifier of a target dialect. By inputting the text features and the speech requirement information into a speech synthesis model, a synthesized speech of a target character reading a target text in a target dialect under a target style can be obtained. Figure 2 As shown, the speech synthesis model includes: a dialect synthesis front end 201, a large language model 202, an encoder 203, and a decoder 204.

[0066] After the text data and the dialect label are input to the dialect synthesis front end 201, the dialect synthesis front end 201 will search for the phoneme pronunciation rules, prosody control rules, and tone control rules corresponding to the target dialect indicated by the dialect label. Then, based on the phoneme pronunciation rules, prosody control rules, and tone control rules corresponding to the target dialect, the text data is converted into a target phoneme sequence with the characteristics of the target dialect.

[0067] After the emotion label, timbre label, dialect label and target phoneme sequence are input into the large language model 202, the large language model 202 will generate a prosodic code. The generation process of the prosodic code is described in the above embodiment and will not be repeated here.

[0068] After the emotion label, timbre label, dialect label, target phoneme sequence and prosody coding are input to the encoder 203, the encoder 203 will generate target prior information based on the emotion label, timbre label, dialect label and target phoneme sequence. The encoder 203 is similar to the coding module in the above embodiment, and the process of generating the target prior information is also similar, which is not repeated here. The encoder 203 splices the generated target prior information with the prosody coding to obtain a coding feature, and then inputs the coding feature into the decoder 204.

[0069] After receiving the coding features, the decoder 204 performs speech decoding processing based on the coding features to generate synthesized speech, that is, to obtain synthesized speech in which the target person reads the target text in the target dialect under the target style. Among them, the emotion label is equivalent to the style identifier in the above embodiment, the timbre label is equivalent to the character identifier in the above embodiment, and the dialect label is equivalent to the dialect label in the above embodiment. The target person is the person indicated by the timbre label, the target style is the emotion or style indicated by the emotion label, and the target dialect is the dialect indicated by the dialect label.

[0070] See also Figure 3 According to another aspect of the present application, in an exemplary embodiment, a method for training a speech synthesis model is provided, and the method may include:

[0071] S301: Acquire text features and voice requirement information of sample data; wherein the voice requirement information includes an identifier of a target dialect.

[0072] S302: Train a speech synthesis model based on text features and speech demand information.

[0073] S303: Update model parameters of the speech synthesis model based on the model loss.

[0074] Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information according to text features and speech demand information. The dialect discrimination module is used to generate decision information based on the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information based on the text features, speech demand information and decision information, and input the target prior information into the decoding module. The decoding module is used to generate synthesized speech based on the target prior information. The synthesized speech includes synthesized speech that reads sample data in the target dialect.

[0075] It should be noted that it is understandable that S101 to S102 in the above embodiment are processing procedures in the model inference process, and S301 to S302 in this embodiment are processing procedures in the model training process, and the similarities between the two are not repeated. In the model training process, the sample data used is the data collected in advance for training the speech synthesis model. In addition, in the model training scenario, it is necessary to use the model loss to reversely update the model parameters, which will not be described in detail here.

[0076] In the embodiment of the present application, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The dialect discrimination module can output judgment information to the encoding module, and then use the judgment information to correct the initial prior information output by the encoding module or improve the quality of the initial prior information to obtain target prior information with more dialect characteristics. The final decoding module generates synthesized speech based on the target prior information, making the dialect characteristics of the synthesized speech more significant and improving the effect of synthesizing the dialect. Therefore, the speech synthesis model trained by this embodiment has a better dialect speech synthesis effect.

[0077] In some embodiments of the present application, sample data can be collected in the following ways. Collect Mandarin data: by collecting voice data on the Internet, including data from sources such as radio stations, documentaries, and open source data sets. Collect dialect data: expand the dialect corpus through various means, such as cooperating with local cultural institutions to collect folk dialect voice materials; using crowdsourcing platforms to invite volunteers to record dialect voices, etc. For the collected voice data, score the voice data through MosNet, and filter out voice data with scores higher than a threshold. For example, filter out voice data with scores lower than 3.5. Use speaker checking technology to determine how many speakers there are in each sentence, and more than one person will be filtered out. For the filtered voice data, generate text corresponding to the voice through the language recognition system ASR.

[0078] In some embodiments, speaker clustering models can also be used to obtain speaker information data. In this way, it is possible to determine which speech data corresponds to the same speaker from multiple chaotic and unlabeled speech data. Then, for each speaker, the voiceprint features in the speech data can be extracted.

[0079] Collect a large amount of Mandarin text corpora of various styles, assign corresponding style labels to each text sample (the text converted by ASR), and establish a mapping relationship between text content and style labels. You can also use open source models to extract unsupervised Huber features for speech from the collected Mandarin data and dialect data, and use them for model pre-training.

[0080] In some embodiments of the present application, the speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate prosody coding of sample data in a target dialect according to text features and speech demand information;

[0081] The encoding module inputs the target prior information into the decoding module, including:

[0082] The encoding module concatenates the prosodic code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0083] It should be noted that the prosody code is the prosody information of the sample data in the target dialect. After the target prior information is combined with the prosody code, an intermediate feature that is more in line with the dialect characteristics can be obtained, namely the coding feature. Then, the synthesized speech generated using the coding feature has more obvious dialect characteristics.

[0084] In some embodiments, the prosody encoding module can be, for example, a large language model. By training the large language model separately, it can output prosody encoding based on the input text features and dialect identification. For example, a GPT model based on the Transformer architecture is used as the basic model. By pre-training on large-scale data, a wide range of language knowledge is learned, and then fine-tuning is performed to adapt to specific tasks to show excellent performance. The training process is as follows:

[0085] Phase 1: Baseline Mandarin model training.

[0086] The model is trained using Mandarin data so that the trained model can output prosodic coding according to the style label. In the label coding process, the voiceprint feature of the sample data can be encoded into a coding module of 192*1024 size; the style label of the sample data can be encoded into a coding module of 10*1024 size. In the word segmentation coding process: the pre-trained Tokenizer word segmenter can be used to segment and encode the input sequence and convert it into an input format acceptable to the model. The input sequence can be a phoneme sequence of the sample data or a phoneme sequence with dialect characteristics generated by the above-mentioned dialect synthesis front end. In the process of updating the model parameters based on the model loss, the model loss can be generated based on the prosodic coding generated by the pre-trained codec model. The codec model can generate synthetic speech in a specified dialect based on text data, and its intermediate features include prosodic coding.

[0087] Phase 2: Fine-tuning and training of high-quality Mandarin and dialect data models.

[0088] Using high-quality data from Mandarin data as samples, continue training based on the first-stage model. In the label encoding process, an additional encoding module for dialect labels is added, and its size can be 100*1024. The remaining training process is basically the same as the first stage, and will not be repeated here.

[0089] The third stage: fine-tuning and training of high-quality Mandarin data and dialect data models.

[0090] Using the high-quality data in the Mandarin data as samples, continue training based on the second-stage model. In the label encoding process, an additional encoding module for the timbre label is added, and its size can be 100*1024. The remaining training process is basically the same as the first stage, and will not be repeated here.

[0091] It is understandable that, based on different business needs or voice demand information, only the first and second stages of training can be performed during the training process. It is also possible to perform the first, second, and third stages during the training process. It is worth noting that the training data used in the training process includes Mandarin data, dialect data, timbre labels, style labels, dialect labels, voiceprint features, text features, etc., and they are different in each stage, so they will not be described in detail here.

[0092] In the embodiment of the present application, a prosody coding module dedicated to providing prosody coding is provided in the speech synthesis model. Through the processing of the prosody coding model, the dialect characteristics of the synthesized speech can be further enhanced.

[0093] In some embodiments of the present application, training a speech synthesis model according to text features and speech demand information includes:

[0094] The initial speech synthesis model is trained according to the Mandarin data to obtain an intermediate speech synthesis model, and the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin data;

[0095] The intermediate speech synthesis model is trained according to the text features and the speech demand information to obtain a trained speech synthesis model.

[0096] It should be noted that the speech synthesis model has the basic ability to generate synthesized speech for text. Since dialect data is difficult to organize and obtain. This embodiment will use easily accessible Mandarin data to train the speech synthesis model so that it has basic capabilities. How to train the model so that it has the ability to generate corresponding synthesized speech based on the input text will not be described in detail here. In this embodiment, a speech synthesis model with basic capabilities, that is, an intermediate speech synthesis model, is used as the basis. Then, sample data is used for training so that the trained speech synthesis model can generate dialect speech, that is, synthesized speech that reads text in dialect. In some embodiments, regarding the network architecture of the initial speech synthesis model, it can be the same as the network architecture of the SOVITS model, but is not limited to this.

[0097] In the embodiment of the present application, a multi-stage training method is adopted. First, the basic capabilities of the model are trained using easily available Mandarin data, and then dialect data is used as samples for training on this basis. This can improve the efficiency of model training and shorten the model training time.

[0098] In some embodiments of the present application, the initial speech synthesis model is trained according to Mandarin data to obtain an intermediate speech synthesis model, including:

[0099] The initial speech synthesis model is trained according to the Mandarin data and its style identifier to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to Mandarin and its style identifier. The speech data corresponding to Mandarin and its style identifier includes: speech data of reading the Mandarin data in the style indicated by the style identifier.

[0100] It should be noted that, regarding the style identifier, please refer to the same description of the above-mentioned embodiment of the speech synthesis method, which will not be repeated here. Add a style identifier to the synthesized speech that needs to be output by the speech synthesis model, even if the model has the ability to generate speech in a specified style. It is necessary to add a style identifier to instruct the model. During the model training process, due to the limited dialect data and the difficulty in obtaining it. Therefore, when using Mandarin data to train the intermediate speech synthesis model, the ability of the training model to generate speech in a specified style can be enhanced. The Mandarin data in the training process can provide Mandarin data for the above-mentioned embodiments, but is not limited to this.

[0101] In an embodiment of the present application, the ability of a model to generate speech in a specified style is trained on easily accessible Mandarin data and can be simultaneously applied to dialect data without style labels, thereby reducing the complexity of model training.

[0102] In some embodiments of the present application, the speech requirement information further includes a style identifier; training an intermediate speech synthesis model according to the text features and the speech requirement information to obtain a trained speech synthesis model includes:

[0103] An intermediate speech synthesis model is trained according to the text features, the identifier of the target dialect and the style identifier to obtain a trained speech synthesis model.

[0104] It should be noted that although the intermediate speech synthesis model has the ability to generate speech in a specified style, it does not involve dialect information. Therefore, it is necessary to add dialect information to enable it to generate speech in a specified dialect. During this stage of training, continuing to add style identifiers will greatly improve the model's ability to generate speech in a specified style.

[0105] In the embodiment of the present application, the ability of the model to generate speech in a specified style is improved in both training stages, thereby greatly improving the ability of the model to generate speech in a specified style.

[0106] In some embodiments of the present application, the initial speech synthesis model is trained according to Mandarin data to obtain an intermediate speech synthesis model, including:

[0107] The initial speech synthesis model is trained according to Mandarin data and voiceprint features to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to Mandarin and voiceprint features. The speech data corresponding to Mandarin and voiceprint features includes: speech data reading Mandarin data according to voiceprint features.

[0108] It should be noted that voiceprint features can be regarded as timbre information, and different voiceprint features can indicate different timbres. Therefore, speech data of Mandarin data read aloud according to voiceprint features can be regarded as speech data of a person with the timbre indicated by the voiceprint features reading the Mandarin data. In this way, the synthesized speech output by the model can be a dialect speech with a specified timbre.

[0109] When the speech synthesis model needs to be able to generate speech according to a specified timbre, it is necessary to add voiceprint features to instruct the model. During the model training process, since dialect data is limited and difficult to obtain, when using Mandarin data to train the intermediate speech synthesis model, the model can be trained to generate speech according to the specified timbre / voiceprint features. The Mandarin data in the training process can provide Mandarin data for the above-mentioned embodiment, but is not limited thereto.

[0110] In an embodiment of the present application, the ability of the model to generate speech according to a specified timbre is trained on easily accessible Mandarin data, and can be simultaneously applied to dialect data without timbre labels, thereby reducing the complexity of model training.

[0111] In some embodiments of the present application, the speech demand information further includes a character identification; training an intermediate speech synthesis model according to the text features and the speech demand information to obtain a trained speech synthesis model includes:

[0112] An intermediate speech synthesis model is trained according to text features, voiceprint features, identification of the target dialect and character identification to obtain a trained speech synthesis model.

[0113] It should be noted that although the intermediate speech synthesis model has the ability to generate speech according to the specified timbre, the control effect of the voiceprint on the timbre is usually average. Therefore, it is necessary to add a character identifier to the intermediate speech synthesis model. Regarding the character identifier, please refer to the same description of the above embodiment of the speech synthesis method, which will not be repeated here. During this stage of training, adding a character identifier to control the timbre will greatly improve the model's ability to generate speech according to the specified timbre.

[0114] In the embodiment of the present application, the ability of the model to generate speech according to the specified timbre will be improved in both training stages, thereby greatly improving the ability of the model to generate speech according to the timbre.

[0115] To facilitate understanding, the following example illustrates the model training process, and explains how the model acquires and improves the ability to generate speech according to timbre, style, and dialect.

[0116] The training process is as follows:

[0117] Phase 1: Mandarin basic model training.

[0118] First, select the network architecture of the model, for example, the network architecture of the SOVITS model can be selected. Then, based on the Mandarin data, voiceprint features, style tags and text features collected in the above embodiments, the model is trained. During the encoding process in the training process, the voiceprint features are encoded as a coding module of 192*1024 size; the style tags are encoded as a coding module of 10*1024 size, and converted into an input format acceptable to the model. For model loss, the audio adversarial loss makes the speech generated by the generator in the network architecture closer to the real speech, improving the quality and naturalness of the synthesized speech. The feedback of the discriminator in the network architecture can help the generator learn the characteristics of the real speech, thereby continuously improving the synthesis effect. The feature matching loss can prompt the generator to learn the feature distribution of the real speech at different levels, avoiding the generator from paying too much attention to the final output result and ignoring the learning of the intermediate process. This helps to improve the details and authenticity of the synthesized speech. Mel-Spectrogram Loss Mel-Spectrogram Loss calculates the difference between the Mel-spectrum of the synthesized speech and the real speech. The model can learn the frequency characteristics of the real speech and improve the timbre and sound quality of the synthesized speech. In some embodiments, the training data can be divided into small batches for iterative training to improve training efficiency and stability. In addition, when evaluating the performance of the model on the validation set, the similarity between the original recording and the generated speech can be measured to check the accuracy of the model-generated encoding.

[0119] Phase 2: Multi-dialect model training.

[0120] Based on the dialect data, voiceprint features, style labels, dialect labels and text features collected in the above embodiments, the model trained in the first stage is further trained. In this stage, a dialect discriminator is added to the original network architecture, thereby introducing coding adversarial loss, so that the code generated by the generator is closer to the coding sequence with dialect features. The structure of the dialect discriminator is basically similar to SOVITS, except that the multi-resolution discriminator is removed and the rest is retained. In this way, the original audio signal passes through a 6-layer 1D convolution module, is converted to 2D, and then passes through a 2D, 5-layer multi-cycle convolution module (the scale of the multi-cycle convolution is 2, 3, 5, 7, 11,), and the returned result can include two categories: the final generated judgment information and the output feature judgment results of each layer. In some embodiments, the training data is divided into small batches for iterative training to improve training efficiency and stability; when evaluating the performance of the model on the validation set, the similarity between the original recording and the generated speech is measured to check the accuracy of the model-generated coding. After training, specifying a style label can generate speech of the corresponding style; specifying a voiceprint feature can generate speech with the timbre characteristics of the voiceprint, and the style label and voiceprint feature can be combined to control the generated speech.

[0121] The third stage: fine-tuning training of multi-dialect models, using a small amount of high-quality target speaker data and some high-quality dialect data for fine-tuning training.

[0122] Based on the data collected in the aforementioned embodiments, a small amount of high-quality target speaker data and some high-quality dialect data, voiceprint features, timbre labels, dialect labels and text features, the model after the second stage training continues to be trained / fine-tuned. During the training process, the style coding module parameters are fixed; the timbre label coding module size is added to the initial model with a size of 100*1024. Since the speaker timbre control based on voiceprint is weak, in the fine-tuning stage, the speaker label is adjusted to better control the target speaker timbre. In some embodiments, the training data can be divided into small batches for iterative training to improve training efficiency and stability. When evaluating the performance of the model on the validation set, the similarity between the original recording and the generated speech is measured to check the accuracy of the model generation coding; the generated speech can be controlled in combination by style labels and timbre labels. In some embodiments, the trained model, in which the intermediate features include prosody coding, can be used to guide the training of the large language model in the above embodiments.

[0123] It should be noted that despite the continuous advancement of technology, there is still a gap between the naturalness of synthesized dialect speech and real human pronunciation. There may be problems such as mechanical feeling, unsmoothness or lack of emotional expression, which makes it difficult to fully simulate the rich changes of human speech. Especially when expressing complex emotions and subtle changes in tone, the synthesized speech often appears to be more stiff. In order to improve the accuracy and naturalness of multi-dialect speech synthesis, this application improves the key technology of speech synthesis. In terms of rhythm modeling, a modeling method based on dialect rhythm features is adopted to generate more natural dialect speech. In terms of emotion and pragmatic information modeling, dialect speech synthesis can express emotion and pragmatic information and enhance the expressiveness of speech. A multi-stage training method is adopted to learn the rhythmic characteristics of each dialect, and the model is pre-trained through massive data to obtain pre-trained model parameters, so that the basic natural pronunciation rhythm can be learned. Then a small amount of high-quality data with manual annotation is used to tune the model parameters to obtain an optimized model, so that the pronunciation and rhythmic characteristics of each dialect can be accurately learned.

[0124] In the embodiments of the present application, a multi-stage training scheme is adopted. The baseline Mandarin model can learn basic language information on large-scale data; the high-quality Mandarin and dialect data model adds dialect data and continues to train on higher-quality data to ensure model stability; the fine-tuning training with high-quality data can learn the pronunciation details of the target speaker more finely to ensure the synthesis effect. For the control of timbre, the timbre is first controlled based on the voiceprint, and then in the fine-tuning training model stage with high-quality data, the timbre is controlled using the speaker information, so that the timbre in the synthesized speech can be completely controlled, solving the problem of timbre control in big data training. In addition, adding information on dialects / dialect types at the coding level can ensure that the generated coding is more in line with the language characteristics of the dialect.

[0125] In some embodiments of the present application, the text features include: a target phoneme sequence of the sample data having characteristics of the target dialect.

[0126] It should be noted that after various data are collected in the manner described in the above embodiment, a customized front end, i.e., a dialect synthesis front end, can be created using the collected data. A dialect tag can be specified, and the front-end engine outputs text features that conform to the dialect according to the dialect tag, including phonemes, prosody, and tone information.

[0127] In some embodiments, a customized mapping dictionary can be constructed according to the phonemes, tones, and rhythms of different dialects. A small amount of dialect data can be collected to cover all the pronunciation characteristics of the dialect. Then, by using a combination of manual annotation and automatic annotation, accurate customized dictionaries of different dialects can be obtained at a low manual annotation cost. The front end of each dialect is created by using a specific dictionary for each dialect to obtain a dialect synthesis front end. For example, based on the dialect data, the pronunciation phonemes, pronunciation rhythm changes, and tone changes unique to each dialect are filtered and analyzed to determine the pronunciation rules, rhythm and tone control rules of the phonemes corresponding to each dialect. Based on the pronunciation rules and control rules corresponding to the dialect, a dialect synthesis front end is created, so that when a text is input into the dialect synthesis front end and the dialect is specified, a phoneme sequence with the characteristics of the dialect can be output. Taking a certain dialect as an example, the pronunciation rules of the phonemes can accurately convert the characters in the text into the corresponding phoneme sequence, and then combine them according to the specific rhythm and tone control rules to generate a phoneme sequence with the characteristics of the dialect.

[0128] In the embodiment of the present application, the text data also has dialect characteristics, which can further improve the effect of the model's final synthesis of dialect speech.

[0129] Exemplary Devices

[0130] Accordingly, the present application also provides a speech synthesis device, see Figure 4 As shown, the device comprises:

[0131] The first acquisition module 401 is used to acquire text features and speech requirement information of a target text; wherein the speech requirement information includes an identifier of a target dialect;

[0132] The speech module 402 is used to input text features and speech requirement information into a speech synthesis model to obtain a synthesized speech that reads the target text in a target dialect;

[0133] Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information according to text features and speech demand information. The dialect discrimination module is used to generate decision information according to the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information according to the text features, speech demand information and decision information, and input the target prior information into the decoding module. The decoding module is used to generate synthesized speech according to the target prior information.

[0134] In some embodiments, the speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate a prosody coding of the target text in the target dialect according to the text features and the speech requirement information;

[0135] The encoding module inputs the target prior information into the decoding module, including:

[0136] The encoding module concatenates the prosodic code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0137] In some embodiments, the text features include: a target phoneme sequence of the target text having characteristics of a target dialect.

[0138] In some embodiments, the first acquisition module 401 is specifically used to convert the target text into an initial phoneme sequence based on the phoneme pronunciation rules of the target dialect; based on the rhythm control rules and / or tone control rules of the target dialect, arrange and combine the phonemes in the initial phoneme sequence to generate a target phoneme sequence with the characteristics of the target dialect.

[0139] In some embodiments, the speech demand information also includes a character identification; the speech module 402 is specifically used to input text features and speech demand information into a speech synthesis model to obtain a synthesized speech of a target person reading a target text in a target dialect, wherein the target person includes the person indicated by the character identification.

[0140] In some embodiments, the speech requirement information also includes a style identifier; the speech module 402 is specifically used to input text features and speech requirement information into a speech synthesis model to obtain synthesized speech that reads the target text in a target dialect under a target style, wherein the target style includes the style indicated by the style identifier.

[0141] The speech synthesis device provided in this embodiment belongs to the same application concept as the speech synthesis method provided in the above embodiments of this application, can execute the speech synthesis method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not fully described in this embodiment, please refer to the specific processing content of the speech synthesis method provided in the above embodiments of this application, which will not be repeated here.

[0142] Accordingly, the present application also provides a training device for a speech synthesis model, see Figure 5 As shown, the device comprises:

[0143] The second acquisition module 501 is used to acquire text features and speech demand information of sample data; wherein the speech demand information includes an identifier of a target dialect;

[0144] A training module 502 is used to train a speech synthesis model according to text features and speech demand information;

[0145] An updating module 503, used for updating model parameters of the speech synthesis model based on the model loss;

[0146] Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module. The encoding module is used to generate initial prior information according to text features and speech demand information. The dialect discrimination module is used to generate decision information based on the initial prior information. The decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information based on the text features, speech demand information and decision information, and input the target prior information into the decoding module. The decoding module is used to generate synthesized speech based on the target prior information. The synthesized speech includes synthesized speech that reads sample data in the target dialect.

[0147] In some embodiments, the speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate prosody coding of the sample data in the target dialect according to the text features and the speech requirement information;

[0148] The encoding module inputs the target prior information into the decoding module, including:

[0149] The encoding module concatenates the prosodic code and the target prior information to obtain the encoding features, and inputs the encoding features into the decoding module.

[0150] In some embodiments, the training module 502 includes:

[0151] A first training unit is used to train an initial speech synthesis model according to Mandarin data to obtain an intermediate speech synthesis model, wherein the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin data;

[0152] The second training unit is used to train the intermediate speech synthesis model according to the text features and the speech demand information to obtain a trained speech synthesis model.

[0153] In some embodiments, the first training unit is specifically used to train an initial speech synthesis model based on Mandarin data and its style identifier to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to Mandarin and its style identifier. The speech data corresponding to Mandarin and its style identifier includes: speech data of reading Mandarin data in the style indicated by the style identifier.

[0154] In some embodiments, the speech requirement information also includes a style identifier; the second training unit is specifically used to train an intermediate speech synthesis model according to text features, an identifier of the target dialect and a style identifier to obtain a trained speech synthesis model.

[0155] In some embodiments, the first training unit is specifically used to train an initial speech synthesis model based on Mandarin data and voiceprint features to obtain an intermediate speech synthesis model. The intermediate speech synthesis model is used to generate speech data corresponding to Mandarin and voiceprint features. The speech data corresponding to Mandarin and voiceprint features includes: speech data of reading Mandarin data according to voiceprint features.

[0156] In some embodiments, the speech demand information also includes a character identification; the second training unit is specifically used to train an intermediate speech synthesis model according to text features, voiceprint features, identification of the target dialect and character identification to obtain a trained speech synthesis model.

[0157] In some embodiments, the text features include: a target phoneme sequence of the sample data having characteristics of the target dialect.

[0158] The training device for the speech synthesis model provided in this embodiment belongs to the same application concept as the training method for the speech synthesis model provided in the above embodiments of this application, and can execute the training method for the speech synthesis model provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not described in detail in this embodiment, please refer to the specific processing content of the training method for the speech synthesis model provided in the above embodiments of this application, and will not be repeated here.

[0159] It should be understood that the modules in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in part by a processor calling software, and the remaining part is implemented in the form of a hardware circuit.

[0160] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0161] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0162] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0163] Exemplary Electronic Devices

[0164] The present application embodiment provides an electronic device, see Figure 6 As shown, the device includes:

[0165] Memory 600 and processor 610;

[0166] The memory 600 is connected to the processor 610 and is used to store programs;

[0167] The processor 610 is used to implement the speech synthesis method or speech synthesis model training method disclosed in any of the above embodiments by running the program stored in the memory 600.

[0168] Specifically, the electronic device may further include: a bus, a communication interface 620 , an input device 630 and an output device 640 .

[0169] The processor 610, the memory 600, the communication interface 620, the input device 630 and the output device 640 are connected to each other via a bus.

[0170] A bus may include a pathway that transfers information between components of a computer system.

[0171] The processor 610 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0172] The processor 610 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0173] The memory 600 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 600 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0174] The input device 630 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0175] Output device 640 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0176] The communication interface 620 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0177] The processor 610 executes the program stored in the memory 600 and calls other devices, which can be used to implement each step of any speech synthesis method or speech synthesis model training method provided in the above embodiments of the present application.

[0178] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the speech synthesis method or the training method of a speech synthesis model introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the embodiment introduction of the above-mentioned speech synthesis method or the training method of a speech synthesis model.

[0179] Exemplary computer program products and storage media

[0180] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the speech synthesis method or the speech synthesis model training method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0181] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0182] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the speech synthesis method or the speech synthesis model training method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0183] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0184] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0186] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0187] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0188] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0190] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0191] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0192] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0193] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquire text features and speech requirement information of a target text; wherein the speech requirement information includes an identifier of a target dialect; Inputting the text features and the speech requirement information into a speech synthesis model to obtain a synthesized speech that reads the target text in the target dialect; Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module, the encoding module is used to generate initial prior information according to the text features and the speech requirement information, the dialect discrimination module is used to generate decision information according to the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information according to the text features, the speech requirement information and the decision information, and input the target prior information into the decoding module, and the decoding module is used to generate the synthesized speech according to the target prior information.

2. The method according to claim 1, characterized in that The speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate a prosody coding of the target text in the target dialect according to the text features and the speech requirement information; The encoding module inputs the target prior information into the decoding module, including: The encoding module concatenates the prosodic code and the target prior information to obtain encoding features, and inputs the encoding features into the decoding module.

3. The method according to claim 1, characterized in that The text features include: a target phoneme sequence of the target text having characteristics of a target dialect.

4. The method according to claim 3, characterized in that The acquiring of the text features of the target text includes: Based on the phoneme pronunciation rules of the target dialect, converting the target text into an initial phoneme sequence; Based on the prosody control rules and / or tone control rules of the target dialect, the phonemes in the initial phoneme sequence are arranged and combined to generate a target phoneme sequence with the characteristics of the target dialect.

5. The method according to claim 1, characterized in that: The speech requirement information also includes a character identifier; inputting the text features and the speech requirement information into a speech synthesis model to obtain a synthesized speech that reads the target text in the target dialect, including: The text features and the speech requirement information are input into a speech synthesis model to obtain a synthesized speech of a target person reading the target text in the target dialect, wherein the target person includes the person indicated by the person identifier.

6. The method according to claim 1, characterized in that The speech requirement information also includes a style identifier; inputting the text features and the speech requirement information into a speech synthesis model to obtain a synthesized speech that reads the target text in the target dialect, including: The text features and the speech requirement information are input into a speech synthesis model to obtain synthesized speech that reads the target text in the target dialect under a target style, wherein the target style includes the style indicated by the style identifier.

7. A method for training a speech synthesis model, characterized in that: The method comprises: Acquire text features and speech demand information of sample data; wherein the speech demand information includes an identifier of a target dialect; Training a speech synthesis model according to the text features and the speech demand information; Updating model parameters of the speech synthesis model based on the model loss; Among them, the speech synthesis model includes: an encoding module, a dialect discrimination module and a decoding module, the encoding module is used to generate initial prior information according to the text features and the speech demand information, the dialect discrimination module is used to generate decision information according to the initial prior information, and the decision information is used to correct the initial prior information. The encoding module is also used to generate target prior information according to the text features, the speech demand information and the decision information, and input the target prior information into the decoding module, and the decoding module is used to generate synthesized speech according to the target prior information, and the synthesized speech includes synthesized speech that reads the sample data in the target dialect.

8. The method according to claim 7, characterized in that The speech synthesis model further includes: a prosody coding module, wherein the prosody coding module is used to generate a prosody coding of the sample data in the target dialect according to the text features and the speech requirement information; The encoding module inputs the target prior information into the decoding module, including: The encoding module concatenates the prosodic code and the target prior information to obtain encoding features, and inputs the encoding features into the decoding module.

9. The method according to claim 7 or 8, characterized in that: Training a speech synthesis model according to the text features and the speech demand information includes: Training an initial speech synthesis model according to the Mandarin data to obtain an intermediate speech synthesis model, wherein the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin data; The intermediate speech synthesis model is trained according to the text features and the speech demand information to obtain a trained speech synthesis model.

10. The method according to claim 9, characterized in that The initial speech synthesis model is trained according to the Mandarin data to obtain the intermediate speech synthesis model, including: An initial speech synthesis model is trained according to Mandarin data and its style identifier to obtain an intermediate speech synthesis model, wherein the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin and its style identifier, wherein the speech data corresponding to the Mandarin and its style identifier includes: speech data of reading the Mandarin data in the style indicated by the style identifier.

11. The method according to claim 10, characterized in that The speech requirement information also includes a style identifier; training the intermediate speech synthesis model according to the text features and the speech requirement information to obtain a trained speech synthesis model, including: The intermediate speech synthesis model is trained according to the text features, the identifier of the target dialect and the style identifier to obtain a trained speech synthesis model.

12. The method according to claim 9, characterized in that The initial speech synthesis model is trained according to the Mandarin data to obtain the intermediate speech synthesis model, including: An initial speech synthesis model is trained according to Mandarin data and voiceprint features to obtain an intermediate speech synthesis model, wherein the intermediate speech synthesis model is used to generate speech data corresponding to the Mandarin and voiceprint features, wherein the speech data corresponding to the Mandarin and voiceprint features includes: speech data of reading the Mandarin data according to the voiceprint features.

13. The method according to claim 12, characterized in that The speech demand information also includes a character identifier; training the intermediate speech synthesis model according to the text features and the speech demand information to obtain a trained speech synthesis model, including: The intermediate speech synthesis model is trained according to the text features, the voiceprint features, the identification of the target dialect and the character identification to obtain a trained speech synthesis model.

14. The method according to claim 7, characterized in that The text features include: a target phoneme sequence of the sample data having characteristics of the target dialect.

15. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 14 by running the program in the memory.

16. A computer program product, characterized in that include: A computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images