Speech synthesis method and device, equipment and storage medium

Abstract

The invention discloses a speech synthesis method and device, equipment and a storage medium. The method comprises the steps of obtaining a phoneme sequence corresponding to a to-be-synthesized text; duration prediction is carried out based on duration prediction related information, target duration information corresponding to the to-be-synthesized text is obtained, and the duration prediction related information comprises reference duration information of the reference voice, the to-be-synthesized text and a phoneme sequence; performing voice representation prediction based on the target duration information to obtain target fine-grained voice representation; and synthesizing the target speech based on the target fine-grained speech representation. According to the scheme, the speech synthesis effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Art

[0002] Speech synthesis technology is one of the branches of artificial intelligence research. It mainly converts text information into audible sound information, that is, to make the machine speak like a human. For example, speech synthesis technology includes the reproduction of a sentence sound, which specifically refers to generating a synthesized speech with the speaker's speaking characteristics by referring only to a sentence of speech with a limited duration. The application prospects of a sentence sound are broad. It has application potential in the fields of voice interaction such as translation machines, mobile phones, and other intelligent interaction fields, novel reading, short videos and other fields.

[0003] The effect of synthesized speech is affected by many factors, and how to improve the effect of speech synthesis has become an urgent problem to be solved. Summary of the invention

[0004] The present application at least provides a speech synthesis method, apparatus, device and storage medium.

[0005] The present application provides a speech synthesis method, including: obtaining a phoneme sequence corresponding to a text to be synthesized; performing duration prediction based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized, wherein the duration prediction related information includes reference duration information of a reference speech, the text to be synthesized, and the phoneme sequence; performing speech representation prediction based on the target duration information to obtain a target fine-grained speech representation; and synthesizing a target speech based on the target fine-grained speech representation.

[0006] The present application provides a speech synthesis device, including an acquisition module, a duration prediction module, a representation prediction module and a synthesis module. The acquisition module is used to acquire a phoneme sequence corresponding to a text to be synthesized; the duration prediction module is used to perform duration prediction based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized, wherein the duration prediction related information includes reference duration information of a reference speech, the text to be synthesized and the phoneme sequence; the representation prediction module is used to perform speech representation prediction based on the target duration information to obtain a target fine-grained speech representation; the synthesis module is used to synthesize the target speech based on the target fine-grained speech representation.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement any of the above methods.

[0008] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, any of the above methods is implemented.

[0009] In the above scheme, duration prediction is performed based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized. This duration information is used to obtain target fine-grained speech representation to synthesize speech, so that the duration of the synthesized target speech can be more accurate, thereby improving the speech synthesis effect.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 It is a flowchart of an embodiment of the speech synthesis method of the present application; Figure 2 It is a schematic diagram of an embodiment of the first step of obtaining a phoneme sequence of the speech synthesis method of the present application; Figure 3 It is a schematic diagram of an embodiment of a duration prediction step 1 of the speech synthesis method of the present application; Figure 4 It is a schematic diagram of an embodiment of a coarse-grained representation prediction step 1 of the speech synthesis method of the present application; Figure 5 It is a schematic diagram of an embodiment of a fine-grained representation prediction step 1 of the speech synthesis method of the present application; Figure 6 is a schematic diagram of another embodiment of the speech synthesis method of the present application; Figure 7 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application; Figure 8 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0014] In the following description, for the purpose of explanation rather than limitation, specific details such as specific subsystem structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0015] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the speech synthesis method of the present application. It should be noted that the speech synthesis method of the present application can be used to synthesize speech, and the text content and speaker voice information of the synthesized speech can be specified. Specifically, the text content of the synthesized speech can be specified by providing the text to be synthesized, and the speaker voice information of the synthesized speech can be specified by providing a reference speech. Furthermore, the reference speech can be a sentence of speech. Compared with a long speech, the duration of a sentence of speech is relatively short, and the duration of the reference speech can be controlled by setting a duration threshold.

[0017] Specifically, the method may include: Step S110: Obtain the phoneme sequence corresponding to the text to be synthesized.

[0018] The text to be synthesized may be used as the text content of the target speech to be synthesized. In some embodiments, the text to be synthesized may include texts in multiple languages.

[0019] In some implementation scenarios, the text to be synthesized is segmented according to the punctuation marks contained therein to obtain multiple sentences. The number of languages ​​corresponding to all the sentences contained in the text to be synthesized is multiple. Exemplarily, the text to be synthesized includes one Chinese sentence and two English sentences.

[0020] Among them, the phoneme sequence can represent the phonemes corresponding to the text to be synthesized. Furthermore, the phoneme sequence can represent multi-dimensional information. It can not only represent the phonemes corresponding to the text to be synthesized, but also integrate prosodic markers (phoneme boundaries, subword boundaries, segment pauses, and long pause boundaries), intonation, stress, harmony, and silent segment markers, etc., considering the speech characteristics of multiple languages.

[0021] The multi-dimensional information represented by the phoneme sequence may include information associated with a specific language and information unrelated to a language. For example, intonation, stress, and harmony are information associated with a specific language. For information associated with a specific language, if the text to be synthesized contains sentences of a specific language, the phoneme sequence may contain corresponding information. Specifically, intonation is associated with Chinese, stress is associated with English, and harmony is associated with Japanese.

[0022] In some implementation scenarios, the phoneme sequence is a multi-dimensional information sequence of phoneme granularity, and the aforementioned multi-dimensional information includes phonemes, prosodic markers, intonation information, stress information, and silent segment markers.

[0023] Step S120: Perform duration prediction based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized.

[0024] The duration prediction related information may include reference duration information of the reference speech, the text to be synthesized, and the phoneme sequence. The reference duration information may characterize the speaking duration characteristics of the speaker of the reference speech, and may be used to predict the target duration information of the target speech. Thus, the target duration information may accurately characterize the duration information of the speaker when speaking the text to be synthesized.

[0025] Step S130: Predict the speech representation based on the target duration information to obtain the target fine-grained speech representation.

[0026] The target fine-grained speech representation may include information such as pronunciation content, rhythm, timbre, pronunciation details, and sound quality of the target speech. In some embodiments, the target fine-grained speech representation may be a mel spectrum.

[0027] Step S140: synthesizing the target speech based on the target fine-grained speech representation.

[0028] In the above scheme, duration prediction is performed based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized. This duration information is used to obtain target fine-grained speech representation to synthesize speech, so that the duration of the synthesized target speech can be more accurate, thereby improving the speech synthesis effect.

[0029] In some embodiments, obtaining the phoneme sequence corresponding to the text to be synthesized may include: determining the language of each sentence in the text to be synthesized to obtain language information of the text to be synthesized; and extracting phonemes of the corresponding language from the text to be synthesized based on the language information to obtain the phoneme sequence corresponding to the text to be synthesized.

[0030] In some embodiments, speech representation prediction may include two stages: coarse-grained prediction and fine-grained prediction. Specifically, the following steps may be included: based on the coarse-grained prediction related information, a coarse-grained speech representation prediction is performed to obtain a target coarse-grained speech representation, wherein the coarse-grained prediction related information may include a reference coarse-grained speech representation of a reference speech, a text to be synthesized, a phoneme sequence, and target duration information. Based on the reference fine-grained speech representation of a reference speech, a phoneme sequence, target duration information, and the target coarse-grained speech representation, a fine-grained speech representation prediction is performed to obtain a target fine-grained speech representation.

[0031] It should be noted that duration prediction can be implemented using a large model. In some embodiments, duration prediction may include: obtaining duration prediction related information, inputting duration prediction related information into the duration prediction large model, and obtaining target duration information. The duration prediction related information includes reference duration information of the reference speech, the text to be synthesized, and the phoneme sequence, and the reference duration information is used as prompt information (prompt) to prompt the duration prediction large model to predict the duration information.

[0032] In some embodiments, the reference duration information is extracted from reference information. Specifically, the reference duration information is encoded on the reference speech to obtain the reference duration information.

[0033] In some implementation scenarios, a duration information encoding module may be used to encode the reference speech to obtain reference duration information as prompt information.

[0034] In some embodiments, the duration prediction information further includes at least one of the language information of the text to be synthesized, the target style information, and the reference prosody representation of the reference speech. The target style information can represent the style of the target speech, thereby generating a target speech of the corresponding style. The target style information includes at least one type of style information.

[0035] In some embodiments, the target style information includes at least one type of style information among quality style, speaker number style, and expression style. One type of style information includes at least one sub-style information, and the target style information of the target speech includes a sub-style information of at least one type of style information.

[0036] In some implementation scenarios, the target style information may include at least one type of style information among quality style, number of speakers style, and expression style. The quality style may include two sub-style information of high audio quality and low audio quality. The number of speakers style may include two sub-style information of many speakers and few speakers. The expression style may include novel style. The above categories and the sub-style information under each category may be adjusted to generate a target speech of a corresponding style.

[0037] In some implementation scenarios, the training data is divided into low-quality data and high-quality data according to the data quality of different data sources, and divided into data with many speakers and data with few speakers according to the number of speakers. Different data obtains different data style information. Audios of different styles are learned to generate target speech with corresponding styles according to the target style information in the inference stage.

[0038] In a specific application scenario, style information can be used to distinguish between high-quality and low-quality data sources, and can also be used to distinguish between different data grid types to generate high-quality audio with more varied styles. Even when the amount of high-quality training data is small, high-quality synthetic speech can be generated by using the high-quality style definition in the target style information.

[0039] In some embodiments, the duration prediction related information may also include language information of the text to be synthesized, target style information, and reference prosodic features of the reference speech. The language information of the text to be synthesized may be determined based on the text to be synthesized. The target style information may be preset according to user needs or determined according to user selection operations. The reference prosodic features may characterize the prosodic features of the reference speech speaker and may be extracted based on the reference speech.

[0040] Specifically, obtaining information related to duration prediction may include the following steps: fusing the language information of the text to be synthesized, the target style information and the text to be synthesized to obtain a first fused representation, and extracting the prosodic representation of the reference speech to obtain a reference prosodic representation; and encoding the duration information of the reference speech to obtain reference duration information; using the phoneme sequence, the first fused representation, the reference prosodic representation and the reference duration information as information related to duration prediction.

[0041] In some implementation scenarios, reference duration information, text to be synthesized, phoneme sequence, first fusion representation, and reference prosodic representation are input into a large duration prediction model as duration prediction related information to obtain target duration information of the target speech.

[0042] In some embodiments, the duration prediction model is an autoregressive model, and duration prediction related information is input into the duration prediction model to obtain target duration information, including: inputting the duration prediction information and the target duration information predicted last by the duration prediction model into the duration prediction model to obtain the target duration information predicted this time.

[0043] refer to Figure 2 , Figure 2 It is a schematic diagram of an embodiment of the first step of obtaining a phoneme sequence of the speech synthesis method of the present application.

[0044] In this embodiment, sentence-level language judgment is constructed, and then data information is extracted and constructed for different languages ​​and different data sources.

[0045] The sentence-level language determination module can be used to determine the language of the synthesized text. Language determination can be used to determine the main language of the sentence.

[0046] The text to be synthesized is a multilingual text. After language determination, the language of each sentence is obtained. The corresponding sentences are processed using the front end of the corresponding language to obtain the phoneme sequence corresponding to each sentence. The phoneme sequences corresponding to each sentence are concatenated to obtain the phoneme sequence corresponding to the text to be synthesized.

[0047] In some implementation scenarios, in order to solve the problem that silent segments in data are difficult to predict, during the training phase, different silent segments are counted, and the silent segments are divided into different scales according to their duration, and the silent segments are classified and labeled. During the inference phase, the silent segment labels corresponding to the text to be synthesized are predicted as information of one dimension in the phoneme sequence.

[0048] The phoneme sequence can then be used as input to a large multilingual text-to-speech model to ultimately obtain synthesized speech.

[0049] refer to Figure 3 , Figure 3 It is a schematic diagram of an embodiment of a duration prediction step of the speech synthesis method of the present application.

[0050] In this embodiment, a causal transformer decoder is used as a large model for duration prediction.

[0051] The large model results are used to predict the duration of discrete tokens. This allows us to use massive amounts of data to learn the duration rhythmic characteristics of different people based on prompt information. At the same time, the fusion of information in different languages ​​and different data sources makes it easier to control the duration model.

[0052] The model structure is as follows Figure 3As shown in the figure, through the standard serialization structure [start, prompt(prompt_start, prompt, prompt_end), text (text_start, language, style, text, text_end), dur (dur_start, dur, dur_end), end], the prompt information and the text to be synthesized module use bidirectional attention to facilitate the model to learn the contextual information of the reference audio and the text to be synthesized before and after. The duration module combines the phoneme-level reference rhythm representation and adopts a unidirectional attention autoregressive structure to better integrate the prompt information and the language information and target style information in the text sequence to be synthesized. The phoneme sequence is integrated into the duration information sequence. The addition of the phoneme-level reference rhythm representation improves the length fluctuation and variability of the duration prediction, making the duration more natural. During generation, the language information and target style information can be used to control the change of the synthesis duration. The language information can be the distinction between high and low quality data sources, or the distinction between different data style types, to generate more varied and high-quality duration rhythm.

[0053] Integrating duration information into subsequent large-model representation predictions can further improve pronunciation stability. Even when there is less multilingual data, pronunciation stability can be guaranteed, reducing the problem of missing or extra words in large models. Extracting and constructing data information from different data sources in different languages ​​and incorporating it into the large-model duration prediction module can predict high-quality duration information in different languages.

[0054] It should be noted that the coarse-grained speech representation prediction can be implemented using a large model. In some embodiments, the coarse-grained speech representation prediction may include: obtaining coarse-grained prediction related information, inputting the coarse-grained prediction related information into the coarse-grained representation prediction large model, and obtaining a target coarse-grained speech representation.

[0055] In some implementation scenarios, the coarse-grained prediction related information may include a reference coarse-grained speech representation. The reference coarse-grained speech representation may represent a feature of a reference speech and may be extracted based on the reference speech. The reference coarse-grained speech representation may be used as prompt information to prompt the coarse-grained representation prediction model to predict the coarse-grained representation.

[0056] Obtaining information related to coarse-grained prediction may include: fusing the phoneme sequence and the target duration information to obtain a second fused representation; using the target duration information to extend the duration of the phoneme sequence to obtain an extended representation at the speech frame level; and extracting a coarse-grained representation of the reference speech to obtain a reference coarse-grained speech representation; using the second fused representation, the extended representation, and the reference coarse-grained speech representation as information related to coarse-grained prediction.

[0057] The phoneme sequence and the target duration information may be information at the phoneme level, and specifically, the target duration information may include duration information corresponding to each phoneme. The phoneme sequence may be expanded to a representation at the speech frame level using the target duration information.

[0058] In some implementation scenarios, the coarse-grained prediction related information may also include the text to be synthesized. The second fused representation, the extended representation, the reference coarse-grained speech representation, and the text to be synthesized may be used as the coarse-grained prediction related information.

[0059] In some embodiments, voiceprint information of the reference speech may be extracted as coarse-grained prediction related information. The granularity of the voiceprint information may be adjusted according to actual conditions. For example, it may be sentence-level voiceprint information, or the voiceprint information of the reference speech as a whole may be extracted.

[0060] In some embodiments, at least one of the language information and the target style information of the text to be synthesized may be used to obtain the coarse-grained prediction related information. Exemplarily, the language information and the other information are fused to serve as the coarse-grained prediction related information.

[0061] In some embodiments, obtaining coarse-grained prediction related information may also include: extracting sentence-level voiceprint information of the reference speech to obtain reference sentence-level voiceprint information, and, for the synthesized text, the second fusion representation and the extended representation, taking the above three as information to be fused respectively, fusing the information to be fused, the language information of the text to be synthesized and the target style information to obtain a fusion result; taking the reference sentence-level voiceprint information and the corresponding fusion results of the above three as coarse-grained prediction related information.

[0062] In some implementation scenarios, the fusion result corresponding to the text to be synthesized, the fusion result corresponding to the second fusion representation, the fusion result corresponding to the extended representation, the reference coarse-grained speech representation, and the reference sentence-level voiceprint information are used as coarse-grained prediction related information.

[0063] In some embodiments, the coarse-grained representation prediction large model is an autoregressive model, and the coarse-grained prediction related information is input into the coarse-grained representation prediction large model to obtain the target coarse-grained speech representation, including: inputting the coarse-grained prediction related information and the target coarse-grained speech representation predicted by the coarse-grained representation prediction large model last time into the coarse-grained representation prediction large model to obtain the target coarse-grained speech representation predicted this time.

[0064] refer to Figure 4 , Figure 4 It is a schematic diagram of an embodiment of a coarse-grained representation prediction step of the speech synthesis method of the present application.

[0065] In this embodiment, Transformer Decoder (a decoder) is used as a coarse-grained representation prediction model. The duration information is integrated into the autoregressive model to predict the coarse-grained speech representation, such as the sequence represented by C in the figure. BOS2 in the figure is used as the sign of the beginning of the sequence. The data information representation constructed by sentence-level representation and different languages ​​and different data sources is added to improve the controllability and naturalness of the speech synthesis model. Figure 4 Figures a and b show two different implementation scenarios.

[0066] like Figure 4 As shown in a, a coarse-grained representation is extracted from the reference speech as a reference representation, and the text to be synthesized, the phoneme sequence and the duration information are also provided as references. The fusion results of the prompt information, duration information and phoneme sequence are encoded using bidirectional attention. The phoneme sequence is then expanded to the coarse-grained representation level through duration, as shown in the sequence represented by P in the figure. The coarse-grained level representation is integrated into the unidirectional self-attention regression language of the coarse-grained representation. Through the multilingual semantic phoneme representation with duration extension, the duration of the synthesized coarse-grained representation is controlled, and a large model prediction model of coarse-grained representation with controllable duration is constructed. The use of massive data during training can improve the naturalness and similarity of the synthesis of coarse-grained representations.

[0067] like Figure 4 As shown in b, the extraction of clause-level and sentence-level representations is not limited to models such as VAE (Variational Autoencoder), AE (Autoencoder), GST (Global Style Token), and voiceprint extraction. The voiceprint model is used to extract sentence-level voiceprint information, which is added to the coarse-grained representation prediction representation and participates in the autoregressive prediction model. The language information and target style information are added to the coarse-grained representation prediction of the large model that incorporates the duration.

[0068] By adding sentence-level representations, clause-level and sentence-level representations can further extract the rhythmic similarity and timbre similarity in the coarse-grained representation, which can significantly improve the consistency of the sound reproduction effect of the large model, constrain the similarity between the predicted coarse-grained speech representation to be synthesized and the reference speech, thereby improving the sound reproduction effect.

[0069] The addition of language information and style information can help the model distinguish the characteristics of different languages, reduce the interference between languages ​​and the impact of data imbalance, and improve the accuracy and stability of pronunciation in each language. By controlling language information and style information, the impact of low-quality data on the sound quality and synthesis effect of the model can be reduced, and the richness and naturalness of the model synthesis can be expanded through multi-source data. By controlling different data styles, the synthesis rhythm and pronunciation can be controlled in multiple styles and emotions.

[0070] In some embodiments, performing fine-grained speech representation prediction may include: obtaining timbre information at the phoneme level based on a reference fine-grained speech representation and a phoneme sequence; using target duration information to extend the timbre information to obtain extended timbre information at the speech frame level; and obtaining a target fine-grained speech representation based on the extended timbre information and a target coarse-grained speech representation.

[0071] In some implementation scenarios, the extended timbre information may be fused with the target coarse-grained speech representation to obtain a target fusion result, and the target fusion result may be decoded to obtain a target fine-grained speech representation.

[0072] In some embodiments, fine-grained speech representation prediction may be implemented using an acoustic model.

[0073] Among them, the acoustic model can be a non-autoregressive model. The non-autoregressive prediction method can improve the system efficiency of sound reproduction. Moreover, since the fine-grained representation prediction work is simpler than the coarse-grained one, and there is no need to predict the duration and rhythmic information, the non-autoregressive network can more efficiently complete the prediction of fine-grained representation.

[0074] Before making a prediction, the acoustic model may be trained. Specifically, an acoustic model trained with the first sample data is obtained, and the second sample data is processed with the acoustic model to obtain a sample synthesis fine-grained representation; based on the sample synthesis fine-grained representation, at least one of the voiceprint similarity loss and the naturalness loss is calculated; based on at least one of the voiceprint similarity loss and the naturalness loss, a model parameter of the acoustic model is adjusted.

[0075] In some implementation scenarios, the acoustic model is first trained to basic convergence. Specifically, a certain stop condition can be set to control the end of the first stage of training. Then, the acoustic model is trained using the second sample data.

[0076] In some embodiments, the second sample data includes a sample text to be synthesized, a sample reference speech, and a real speech. The sample text to be synthesized and the sample reference speech are used to specify the synthesized text content and the speaker voice information, respectively. The real speech is the real speech corresponding to the sample synthesis fine-grained representation, that is, the speaker of the sample reference speech speaks the sample text to be synthesized. A real coarse-grained speech representation is extracted from the real speech, which is used to fuse with the sample extended timbre information to obtain the sample synthesis fine-grained representation. The sample extended timbre information can refer to the acquisition of the aforementioned extended timbre information.

[0077] Furthermore, calculating the voiceprint similarity loss may include: performing voiceprint extraction on the sample synthetic fine-grained representation and the reference speech respectively to obtain the sample synthetic voiceprint and the reference voiceprint, and obtaining the voiceprint similarity loss based on the difference between the sample synthetic voiceprint and the reference voiceprint.

[0078] Calculating the naturalness loss may include: using a naturalness scoring model to score the sample synthesis fine-grained representation to obtain a scoring result, and obtaining the naturalness loss based on the scoring result.

[0079] In some implementation scenarios, the naturalness loss can be directly obtained by directly using the scoring result of synthesizing fine-grained representation of the sample. In some implementation scenarios, the reference speech can also be scored using the naturalness scoring model to obtain the sample scoring result, and the naturalness loss can be obtained based on the difference between the two scoring results.

[0080] In some embodiments, other losses may be calculated and used together with at least one of the voiceprint similarity loss and the naturalness loss to adjust the model parameters. For example, a generative adversarial loss (GAN loss), a reconstruction loss, etc. may also be calculated. The generative adversarial loss may be obtained by discriminating the sample synthetic voiceprint.

[0081] refer to Figure 5 , Figure 5 It is a schematic diagram of an embodiment of a fine-grained representation prediction step of the speech synthesis method of the present application.

[0082] Based on the coarse-grained representation, fine-grained modeling is carried out, and a non-autoregressive model can be used to predict the fine-grained speech representation. The timbre similarity and naturalness can be improved through unsupervised negative feedback constraints on timbre similarity and naturalness.

[0083] Coarse-grained speech representation has determined the pronunciation content and rhythmic information of the synthesized speech, so coarse-grained speech representation mainly optimizes the rhythmic stability of the model and the naturalness of the synthesis. Taking Mel-spectrogram as an example, fine-grained speech representation mainly synthesizes the timbre, pronunciation details and sound quality of the reference speech to be synthesized based on the coarse-grained speech representation.

[0084] The structure of the acoustic model is shown in the figure, where the encoder can use the Conformer Encoder and the decoder can use the Streaming Conformer Decoder. In the training phase, the representation module and the vector quantization module can be used to process the real speech to obtain the corresponding real coarse-grained representation.

[0085] Encode the phoneme sequence into a phoneme representation; obtain the fine-grained timbre information at the phoneme level through the prompt information encoder and the cross-attention structure. The prompt information encoder module introduces a large amount of timbre data information, passes the duration information into the phoneme information, expands the duration of the fine-grained timbre information to the frame level, and fuses it with the coarse-grained representation, and predicts the Mel spectrum of the model through the decoder. Generative adversarial loss (GAN loss) is used to improve the synthesis effect, and the voiceprint loss negative feedback is used to further improve the synthesized timbre similarity. The score loss is used as a naturalness loss to further improve the synthesized sound quality and rhythmic naturalness. Among them, the voiceprint loss and naturalness loss can be obtained using the pre-trained voiceprint model and the Mean Opinion Score Network (MOS Net) model. MOS Net can be used to score the sound quality.

[0086] In the inference phase, the coarse-grained speech representation is directly input into the fully connected layer. In the training phase, the real speech is processed to obtain the real coarse-grained speech representation and then input into the fully connected layer.

[0087] refer to Figure 6 , Figure 6 It is a schematic diagram of another embodiment of the speech synthesis method of the present application.

[0088] This method mainly uses a sentence-level language discrimination module, a large model duration prediction module, a large model coarse-grained speech representation prediction module, and a fine-grained speech representation prediction module. Among them, the fine-grained speech representation prediction module is constructed using an acoustic model.

[0089] The sentence-level language judgment module automatically judges the language of the text to be synthesized through the text. With the help of the context understanding ability of the large model, it can more accurately predict the duration, pronunciation, and timbre, and ensure the diversity of the generated speech through sampling during reasoning, so as to achieve multi-language replication of a sentence. The multiple languages ​​involved can be adjusted as needed. For example, they include Chinese, English, Japanese and Korean. The language discrimination module can realize sentence-level language discrimination based on linguistic rules and language discrimination models. Complex language discrimination can use language discrimination models, simple languages ​​can use rules to distinguish binary codes of different languages, and complex languages ​​can also be judged by combining rules and models.

[0090] See also Figure 7 , Figure 7 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application.

[0091] In this embodiment, the speech synthesis device 70 includes an acquisition module 71, a duration prediction module 72, a representation prediction module 73 and a synthesis module 74, wherein the acquisition module 71 is used to acquire a phoneme sequence corresponding to the text to be synthesized; the duration prediction module 72 is used to perform duration prediction based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized, wherein the duration prediction related information includes reference duration information of the reference speech, the text to be synthesized and the phoneme sequence; the representation prediction module 73 is used to perform speech representation prediction based on the target duration information to obtain a target fine-grained speech representation; and the synthesis module 74 is used to synthesize the target speech based on the target fine-grained speech representation.

[0092] See also Figure 8 , Figure 8 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application.

[0093] The electronic device 80 includes a memory 81 and a processor 82, and the processor 82 is used to execute the program instructions stored in the memory 81 to implement the steps in any of the above-mentioned speech synthesis method embodiments. In a specific implementation scenario, the electronic device 80 may include, but is not limited to: computer equipment, electrical equipment, microcomputers, desktop computers, servers, and in addition, the electronic device 80 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0094] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps in any of the above-mentioned speech synthesis method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0095] See also Fig. 9 , Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0096] The computer-readable storage medium 90 provided in this embodiment stores program instructions 91 that can be executed by a processor. When the program instructions 91 are executed by the processor, they are used to implement the steps in any of the above-mentioned speech synthesis method embodiments.

[0097] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0098] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another subsystem, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0099] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0100] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Obtain the phoneme sequence corresponding to the text to be synthesized; Performing duration prediction based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized, wherein the duration prediction related information includes reference duration information of a reference speech, the text to be synthesized, and the phoneme sequence; Predicting speech representation based on the target duration information to obtain a target fine-grained speech representation; A target speech is synthesized based on the target fine-grained speech representation.

2. The method according to claim 1, characterized in that The duration prediction related information also includes at least one of the language information of the text to be synthesized, the target style information and the reference prosody representation of the reference speech, wherein the target style information represents the style of the target speech, and the target style information includes at least one type of style information.

3. The method according to claim 2, characterized in that The target style information includes information of at least one of quality style, speaker number style, and expression style. The quality style includes high audio quality and low audio quality. The speaker number style includes many speakers and few speakers. The expression style includes novel style.

4. The method according to claim 1, characterized in that: The performing of duration prediction based on duration prediction related information to obtain target duration information corresponding to the to-be-synthesized text includes: Obtaining information related to the duration prediction; The duration prediction related information is input into the duration prediction large model to obtain the target duration information.

5. The method according to claim 4, characterized in that The duration prediction related information also includes the language information of the text to be synthesized, the target style information and the reference prosody representation of the reference speech; and obtaining the duration prediction related information includes: Fusing the language information and target style information of the text to be synthesized with the text to be synthesized to obtain a first fused representation; and Extracting prosodic representation from the reference speech to obtain a reference prosodic representation; and, Encoding the duration information of the reference speech to obtain the reference duration information; Using the phoneme sequence, the first fusion representation, the reference prosody representation, and the reference duration information as the duration prediction related information; And / or, the duration prediction model is an autoregressive model, and the step of inputting the duration prediction related information into the duration prediction model to obtain the target duration information includes: The duration prediction related information and the target duration information predicted by the duration prediction big model last time are input into the duration prediction big model to obtain the target duration information predicted this time.

6. The method according to claim 1, characterized in that The step of obtaining the phoneme sequence corresponding to the text to be synthesized includes: Determining the language of each sentence in the text to be synthesized to obtain language information of the text to be synthesized; Based on the language information, phonemes of the language corresponding to the text to be synthesized are extracted to obtain a phoneme sequence corresponding to the text to be synthesized.

7. The method according to claim 1, characterized in that The performing speech representation prediction based on the target duration information to obtain the target fine-grained speech representation includes: Performing a coarse-grained speech representation prediction based on the coarse-grained prediction related information to obtain a target coarse-grained speech representation; the coarse-grained prediction related information includes a reference coarse-grained speech representation of a reference speech, the text to be synthesized, the phoneme sequence and the target duration information; Fine-grained speech representation prediction is performed based on the reference fine-grained speech representation of the reference speech, the phoneme sequence, the target duration information and the target coarse-grained speech representation to obtain the target fine-grained speech representation.

8. The method according to claim 7, characterized in that The step of performing coarse-grained speech representation prediction based on the coarse-grained prediction related information to obtain a target coarse-grained speech representation includes: Acquire the coarse-grained prediction related information; The coarse-grained prediction related information is input into a coarse-grained representation prediction large model to obtain the target coarse-grained speech representation.

9. The method according to claim 8, characterized in that The coarse-grained prediction related information includes a reference coarse-grained speech representation; and obtaining the coarse-grained prediction related information includes: Fusing the phoneme sequence and the target duration information to obtain a second fused representation; and Using the target duration information, the phoneme sequence is extended in duration to obtain an extended representation at the speech frame level; and Extracting a coarse-grained representation of the reference speech to obtain the reference coarse-grained speech representation; The second fused representation, the extended representation, and the reference coarse-grained speech representation are used as the coarse-grained prediction related information.

10. The method according to claim 9, characterized in that The obtaining of the coarse-grained prediction related information further includes: Extracting sentence-level voiceprint information from the reference speech to obtain reference sentence-level voiceprint information; and, The text to be synthesized, the second fused representation, and the extended representation are respectively used as information to be fused; the information to be fused, the language information of the text to be synthesized, and the target style information are fused to obtain a fusion result; The reference sentence-level voiceprint information and the fusion result are used as the coarse-grained prediction related information.

11. The method according to claim 7, characterized in that The step of predicting a fine-grained speech representation based on the reference fine-grained speech representation of the reference speech, the phoneme sequence, the target duration information, and the target coarse-grained speech representation to obtain the target fine-grained speech representation includes: Acquiring timbre information at the phoneme level based on the reference fine-grained speech representation and the phoneme sequence; Using the target duration information, the timbre information is extended in duration to obtain extended timbre information at the speech frame level; The target fine-grained speech representation is obtained based on the extended timbre information and the target coarse-grained speech representation.

12. The method according to claim 7, characterized in that The fine-grained speech representation prediction is implemented by using an acoustic model; the fine-grained speech representation prediction is performed based on the reference fine-grained speech representation of the reference speech, the phoneme sequence, the target duration information and the target coarse-grained speech representation, and before obtaining the target fine-grained speech representation, the method further includes: Acquiring the acoustic model trained with the first sample data; Processing the second sample data using the acoustic model to obtain a synthetic fine-grained representation of the sample; Based on the sample synthesized fine-grained representation, at least one of a voiceprint similarity loss and a naturalness loss is calculated; Based on at least one of the voiceprint similarity loss and the naturalness loss, a model parameter of the acoustic model is adjusted.

13. The method according to claim 12, characterized in that The step of synthesizing the fine-grained representation based on the sample and calculating at least one of the voiceprint similarity loss and the naturalness loss comprises at least one of the following steps: Performing voiceprint extraction on the sample synthetic fine-grained representation and the reference speech respectively to obtain a sample synthetic voiceprint and a reference voiceprint, and obtaining the voiceprint similarity loss based on the difference between the sample synthetic voiceprint and the reference voiceprint; The sample synthetic fine-grained representation is scored using a naturalness scoring model to obtain a scoring result, and the naturalness loss is obtained based on the scoring result.

14. A speech synthesis device, characterized in that: The device comprises: An acquisition module, used to acquire a phoneme sequence corresponding to the text to be synthesized; A duration prediction module, configured to perform duration prediction based on duration prediction related information to obtain target duration information corresponding to the text to be synthesized, wherein the duration prediction related information includes reference duration information of a reference speech, the text to be synthesized, and the phoneme sequence; A representation prediction module, used to predict speech representation based on the target duration information to obtain a target fine-grained speech representation; A synthesis module is used to synthesize a target speech based on the target fine-grained speech representation.

15. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images