A Multilingual Speech Synthesis Method and System

Abstract

This invention proposes a multilingual speech synthesis method and system, relating to the field of speech synthesis technology. The method constructs a multilingual speech synthesis model and a special pronunciation library. First, it synthesizes ordinary multilingual text and domain-specific pronunciation text separately, obtaining corresponding first and second spectra. Then, using the first spectrum as a reference, it performs timbre conversion on the second spectrum and concatenates the two spectra to ensure consistency between the two timbres. Finally, it decodes the synthesized spectrum using a vocoder to obtain the final synthesized speech, improving the speech synthesis effect when multiple languages ​​and special pronunciations are included in a specific domain.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and more specifically, to a multilingual speech synthesis method and system. Background Technology

[0002] Text-to-speech (TTS) has been developing for many years and has been widely applied in many fields. Generally speaking, TTS consists of two parts: an acoustic model and a vocoder. The acoustic model establishes the conversion relationship between phonemes and speech spectra, while the vocoder is responsible for converting the spectral features into speech that the human ear can hear. Currently, the common approach to TTS is an end-to-end deep neural network model, which is highly dependent on the amount of labeled data, typically requiring hundreds or even thousands of hours of speech data to achieve good synthesis results.

[0003] In general domains, speech synthesis technology is relatively mature, and many open-source datasets for mainstream languages ​​are available. However, in certain specific domains, there are situations where multiple languages ​​are used together, such as mixed reading of Chinese, English, and Japanese, or mixed reading of standard language and dialects. Therefore, multilingual speech synthesis is necessary. Although existing multilingual speech synthesis systems often have the ability to recognize different languages ​​in the text, during the speech synthesis stage, the system can only select the synthesis rules of one language to synthesize speech. This results in the synthesized speech corresponding to different language parts of the text being distorted, unnatural, or even noisy. In addition, some proprietary pronunciations lack corresponding data, making it difficult to quickly expand the domain when performing multilingual speech synthesis. Summary of the Invention

[0004] The purpose of this invention is to provide a multilingual speech synthesis method and system that can solve the problems of cross-language and speech synthesis with special pronunciations that may exist in specific fields, thereby improving the speech synthesis effect.

[0005] The embodiments of the present invention are implemented as follows:

[0006] In a first aspect, embodiments of this application provide a multilingual speech synthesis method, which includes:

[0007] The text to be processed is acquired, language is identified, and the text to be processed is segmented based on the identification results to obtain ordinary text and special word text;

[0008] The plain text is converted into vector input and then used in a pre-set multilingual speech synthesis model for spectrum synthesis to obtain the first spectrum;

[0009] The special word text is converted into vectors and input into a pre-set special pronunciation library for matching and recognition to obtain the second spectrum;

[0010] The first and second spectra are concatenated according to the order of the input text to be processed to obtain the composite spectrum;

[0011] Synthetic speech is obtained by decoding the synthesized spectrum using a vocoder.

[0012] Based on the first aspect, in some embodiments of the present invention, the above method further includes:

[0013] Enter the language types related to the domain, and construct a domain phoneme dictionary based on the corresponding words in the language;

[0014] Relevant linguistic data for this field is selected from open-source language datasets to form a training corpus;

[0015] Based on the aforementioned training corpus and the aforementioned domain phoneme dictionary, a deep neural network is trained to obtain the aforementioned multilingual speech synthesis model.

[0016] Based on the first aspect, in some embodiments of the present invention, the above method further includes:

[0017] Enter the language types related to the field, and construct a phoneme dictionary for that language based on the corresponding words;

[0018] Relevant corpus data for this field is selected from open-source language datasets to form training corpora for each language;

[0019] Based on the phoneme dictionary and training corpus corresponding to each language, a deep neural network is trained to obtain the speech synthesis model corresponding to each language.

[0020] By fusing the speech synthesis models corresponding to each language at the parameter level, the aforementioned multilingual speech synthesis model is obtained.

[0021] Based on the first aspect, in some embodiments of the present invention, the step of fusing the speech synthesis models corresponding to each language at the parameter level to obtain the multilingual speech synthesis model specifically includes:

[0022] According to the formula By weighted fusion of common phonemes from various languages, the embedding parameters corresponding to the above multilingual speech synthesis model are obtained, where... Representing phonemes The embedding parameter, for phonemes In language The embedding parameter in the text. for phonemes In language Weights in;

[0023] According to the formula The parameters of the speech synthesis model corresponding to each language By fusing the data, the parameters corresponding to the above multilingual speech synthesis model are obtained. ,in, Represents language type, L A collection of language types For language The parameters obtained during training, For language The corresponding language weight.

[0024] Based on the first aspect, in some embodiments of the present invention, the above method further includes: modifying the above multilingual speech synthesis model according to domain-specific audio data.

[0025] Based on the first aspect, in some embodiments of the present invention, the step of concatenating the first spectrum and the second spectrum according to the order of the input text to obtain the synthesized spectrum includes:

[0026] The timbre of the training speaker corresponding to the first spectrum is used as the target timbre, and the second spectrum is adaptively transformed according to the target timbre.

[0027] Speech synthesis is performed based on the first spectrum and the adaptively transformed second spectrum to obtain the synthesized spectrum.

[0028] Based on the first aspect, in some embodiments of the present invention, the above method further includes: smoothing the synthesized spectrum.

[0029] Secondly, embodiments of this application provide an electronic device including a memory for storing one or more programs; and a processor. When the one or more programs are executed by the processor, the methods described in any of the first aspects above are implemented.

[0030] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects above.

[0031] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0032] This application provides a multilingual speech synthesis method and system. First, the text to be processed is acquired and language is identified to identify different language characters and special word characters. Based on the identification results, the text to be processed is segmented using special words as boundaries to obtain multiple ordinary texts and multiple special word texts. Then, the ordinary texts are converted into vectors and input into a preset multilingual speech synthesis model for spectral synthesis to obtain multiple first spectra. Furthermore, the special word texts are converted into vectors and input into a preset special pronunciation library for matching and recognition to obtain multiple second spectra. Subsequently, the multiple first and second spectra are concatenated according to the input order of the text to be processed to obtain a synthesized spectrum. Finally, the synthesized spectrum is decoded by a vocoder to obtain the final synthesized speech. This method constructs a multilingual speech synthesis model and a special pronunciation library. First, it synthesizes ordinary multilingual text and domain-specific pronunciation text separately to obtain the corresponding first spectrum and second spectrum. Then, using the first spectrum as a reference, it converts the timbre of the second spectrum and concatenates the two to ensure the consistency of the two timbres. Finally, it decodes the synthesized spectrum through a vocoder to obtain the final synthesized speech, thus improving the speech synthesis effect when there are multiple languages ​​and special pronunciations in a specific domain. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a multilingual speech synthesis method provided by the present invention;

[0035] Figure 2 A flowchart of a multilingual speech synthesis method provided by the present invention;

[0036] Figure 3 This is a flowchart illustrating the construction of a multilingual speech synthesis model in a multilingual speech synthesis method provided by the present invention.

[0037] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention.

[0038] Icons: 1. Memory; 2. Processor; 3. Communication interface. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0040] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. Example

[0041] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0042] Please refer to Figure 1-3 This application provides a multilingual speech synthesis method, which includes the following steps:

[0043] Step S1: Obtain the text to be processed, identify the language, and segment the text to be processed according to the identification results to obtain ordinary text and special word text.

[0044] In a specific domain, multiple languages ​​may be used interchangeably. For example, in a subway broadcasting system, Chinese, English, and Japanese may be mixed, or in a local radio station's voice service, Mandarin may be mixed with certain dialectal words. Therefore, multilingual speech synthesis is necessary. In the above steps, a specific word library can be built for the target domain. This library is then used to match and recognize the text to be processed, extracting words with special pronunciations. Based on these words, the text is segmented into multiple segments of ordinary text and special word text. These segments are then statistically analyzed to form sets of ordinary text and special word text, facilitating subsequent synthesis using different methods and improving the accuracy of speech synthesis.

[0045] Step S2: Convert ordinary text into vector input and perform spectrum synthesis in a preset multilingual speech synthesis model to obtain the first spectrum.

[0046] The construction process of the pre-built multilingual speech synthesis model in the above steps is as follows: First, the relevant language types are input, and a domain phoneme dictionary is constructed based on the corresponding words. Then, relevant language data is selected from open-source language datasets to form a training corpus. Next, the deep neural network is trained based on the training corpus and the domain phoneme dictionary. The phoneme dictionary contains mappings from words to phonemes, which can be used to connect the language model and the acoustic model. Specifically, the training text and corresponding audio results from the training corpus are input into the initially built deep neural network model. Audio prediction is performed on the training text based on the domain phoneme dictionary to obtain the prediction results. Then, the prediction results are compared with the input audio results to adjust the relevant parameters of the model and improve the prediction accuracy. Afterward, through iterative training, the model parameters are continuously adjusted and updated until a preset learning accuracy or a preset number of iterations is reached, thus obtaining the aforementioned multilingual speech synthesis model.

[0047] The above process uses the same phoneme dictionary and model for end-to-end training until the model converges. The training process is simple and fast. However, because the data from different languages ​​are isolated during training, the resulting multilingual speech synthesis model may exhibit significant differences in timbre within a single sentence due to language changes, leading to an overall lack of harmony in the synthesized speech. Therefore, this application also proposes another method for constructing a multilingual speech synthesis model, as follows:

[0048] Step S2-1: Input the language types related to the field, and construct a phoneme dictionary for the corresponding language based on the words and phrases.

[0049] In the above steps, the relevant languages ​​involved in the target field are first analyzed, and the words involved in each language are counted, so as to construct the phoneme dictionary corresponding to the language and input it into the initial deep neural network model.

[0050] Step S2-2: Select relevant language data from open-source language datasets to form training corpora for each language;

[0051] In the above steps, although certain specific fields may involve multiple languages, the range of phonemes involved will be relatively narrow, belonging to a small subset of a certain language and basically fixed. Therefore, the synthesis of full phonemes of multiple languages ​​is not very necessary and will increase the training difficulty. So, based on the needs of the field or industry, the corpus data in the language dataset can be filtered to form the training corpus corresponding to each language, thereby reducing the training difficulty of the model and improving the training speed.

[0052] Steps S2-3: Train the deep neural network based on the phoneme dictionary and training corpus corresponding to the language to obtain the speech synthesis model corresponding to each language;

[0053] In the above steps, for each language, the initial deep neural network model is trained based on the phoneme dictionary and training corpus corresponding to that language to obtain the speech synthesis model corresponding to each language, so as to ensure the prediction accuracy for each language.

[0054] Step S2-4: Perform parameter-level fusion of the speech synthesis models corresponding to each language to obtain the above multilingual speech synthesis model.

[0055] In the above steps, the final multilingual speech synthesis model can be obtained by fusing the speech synthesis models corresponding to each language at the parameter level. Specifically, during the fusion process, the formula can be followed. By weighted fusion of common phonemes from various languages, the embedding parameters corresponding to the above multilingual speech synthesis model are obtained, where... Representing phonemes The embedding parameter, for phonemes In language The embedding parameter in the text. for phonemes In language The weight in the phoneme. If a word does not appear in a particular language, its corresponding weight is 0. Then, according to the formula... The parameters of the speech synthesis model corresponding to each language By fusing the data, the parameters corresponding to the final multilingual speech synthesis model described above are obtained. ,in, Represents language type, L A collection of language types For language The parameters obtained during training, For language The corresponding language weight.

[0056] The parameter-level fusion scheme described above can, to some extent, resolve the issue of inconsistent synthesized timbres, resulting in a more harmonious overall speech synthesis outcome. Furthermore, when performing domain transfer, such as from a national speech service domain to a local speech service domain, most of the data involved in the two domains is actually the same, but there will be some differences in the pronunciation of specific words. Therefore, the model can be fine-tuned based on a small amount of specific word data from the transferred domain, correcting the relevant parameters of the model (such as the parameters mentioned above). This enables the model to be transferred to other domains, making its use more flexible.

[0057] Step S3: Convert the special word text into vector inputs and perform matching and recognition in a preset special pronunciation library to obtain the second spectrum.

[0058] In the above steps, for domain-specific words or words with specific pronunciations, they can be pre-pronounced manually and a corresponding binary pronunciation library can be created, which is a mapping between special pronunciation words and their spectra, forming a special pronunciation library for use during synthesis. During speech synthesis, the special word text is directly converted into vectors and input into the pre-set special pronunciation library for matching and recognition, thus obtaining the second spectrum, ensuring accurate synthesis of special pronunciation words in specific domains.

[0059] Step S4: Concatenate the first spectrum and the second spectrum according to the order of the text input to be processed to obtain the synthesized spectrum.

[0060] In the above steps, when splicing multiple segments of the first and second spectra, the timbre of the training speaker corresponding to the first spectrum is first used as the target timbre. The second spectrum is then adaptively transformed based on the target timbre. Speech synthesis is then performed based on the first spectrum and the adaptively transformed second spectrum to obtain the synthesized spectrum. For example, One-shot Voice Conversion technology can be used to convert the timbre of the second spectrum and synthesize it, maintaining the consistency of the spectral timbre. Furthermore, due to the presence of noise and interference, the synthesized spectrum waveform may have jagged edges, affecting the spectral quality. In this case, the synthesized spectrum can be smoothed to filter out noise. For example, rectangular wave train smoothing algorithms, Gaussian smoothing algorithms, parabolic smoothing algorithms, etc., can be used to smooth the synthesized spectrum and improve the spectral quality.

[0061] Step S5: Decode the synthesized spectrum using a vocoder to obtain the synthesized speech.

[0062] In the above steps, the synthesized spectrum obtained by splicing can be decoded using a vocoder to obtain speech audio that can be detected by the human ear. The use of a vocoder for decoding is existing technology and will not be elaborated upon here.

[0063] In summary, this method constructs a multilingual speech synthesis model and a special pronunciation library. First, it synthesizes ordinary multilingual text and domain-specific pronunciation text separately to obtain the corresponding first and second spectra. Then, using the first spectrum as a reference, it converts the timbre of the second spectrum and concatenates the two spectra to ensure the consistency of the two timbres. Finally, it decodes the synthesized spectrum through a vocoder to obtain the final synthesized speech, thus improving the speech synthesis effect when multiple languages ​​and special pronunciations are included in a specific domain.

[0064] Please refer to Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device includes a memory 1, a processor 2, and a communication interface 3. The memory 1, processor 2, and communication interface 3 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory 1 can be used to store software programs and modules, such as the program instructions / modules corresponding to a multilingual speech synthesis system provided in an embodiment of this application. The processor 2 executes the software programs and modules stored in the memory 1 to perform various functional applications and data processing. The communication interface 3 can be used for signaling or data communication with other node devices.

[0065] Among them, memory 1 can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0066] Processor 2 can be an integrated circuit chip with signal processing capabilities. Processor 2 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0067] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 4The components shown can be implemented using hardware, software, or a combination thereof.

[0068] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0069] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0070] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0072] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A multi-lingual speech synthesis method, characterized by, The method comprises the following steps: acquiring a text to be processed to perform language recognition, and segmenting the text to be processed according to the recognition result to obtain common text and special word text; converting the common text into a vector to input a preset multilingual speech synthesis model to perform spectrum synthesis to obtain a first spectrum; converting the special word text into a vector to input a preset special pronunciation library to perform matching recognition to obtain a second spectrum; splicing the first spectrum and the second spectrum in the order of input of the text to be processed to obtain a synthesis spectrum; specifically, taking the timbre of a training speaker corresponding to the first spectrum as a target timbre, and performing adaptive transformation on the second spectrum according to the target timbre; performing speech synthesis based on the first spectrum and the second spectrum after the adaptive transformation to obtain the synthesis spectrum; decoding the synthesis spectrum through a vocoder to obtain synthesis speech; The method further comprises the following steps: recording a language type related to a field, and constructing a phoneme dictionary corresponding to the language according to words corresponding to the language; selecting corpus data related to the field from an open-source language data set to form a training corpus library corresponding to each language; training a deep neural network based on the phoneme dictionary corresponding to the language and the training corpus library to obtain a speech synthesis model corresponding to each language; performing parameter-level fusion on the speech synthesis models corresponding to each language to obtain the multilingual speech synthesis model; The step of performing parameter-level fusion on the speech synthesis models corresponding to each language to obtain the multilingual speech synthesis model specifically comprises the following steps: According to the formula The common phonemes of each language are weighted and fused to obtain the embedding parameters corresponding to the multi-lingual speech synthesis model, wherein The embedding parameters of the representative phonemes , The embedding parameters of the phonemes in the language , The weights of the phonemes in the language ; According to the formula The parameters corresponding to the speech synthesis model corresponding to each language are fused to obtain the parameters corresponding to the multi-language speech synthesis model , wherein The language type represents the language type, L The language type is a set of language types, The language type is a language type The parameters obtained by training, The language type is a language type The language type is a language type​ 2. The multi-lingual speech synthesis method of claim 1, wherein, The method further comprises correcting the multilingual speech synthesis model according to field-specific audio data.

3. The multi-lingual speech synthesis method of claim 1, wherein, The method further comprises performing smoothing processing on the synthesis spectrum.

4. An electronic device, comprising: The computer program is executed by the processor to implement the method according to any one of claims 1-3. The computer program is executed by the processor to implement the method according to any one of claims 1-3. ​ ​ 5. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images