A cross-language speech synthesis method, device, electronic device and storage medium

Abstract

The present application discloses a cross-language speech synthesis method, device, electronic device and storage medium, the method comprising: obtaining speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is in a first language; the speech data of the object to be synthesized is in a second language; the second language and the first language are different languages; obtaining a timbre feature matrix of the target object; and encoding the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized, and adding a set note at the phonemes with the same pronunciation or fuzzy pronunciation to obtain processed encoded data; decoding the timbre feature matrix and the encoded data to obtain synthesized speech data based on the second language with the timbre feature matrix. The present application can make the synthesized speech data have the timbre of the target object while retaining the pronunciation habits of the second language, so as to improve the transferability of the language, and then improve the cross-language speech synthesis effect.

Description

Technical Field

[0001] The present application relates to the technical field of speech synthesis, and in particular to a cross-language speech synthesis method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of electronic information processing technology, voice, as an important carrier for people to obtain information, has been widely used in daily life and work. The implementation scenarios involving voice usually include speech synthesis processing, which refers to synthesizing the specified text or audio into the target audio that meets the requirements.

[0003] Among them, speech conversion is a branch of speech synthesis, involving timbre conversion technology. The core issue of timbre conversion technology is how to decouple the data and timbre in speech, ensuring the integrity of speech data and ensuring that only the timbre information of the target object is retained in the speech data without mixing in the timbre information of the source speaker.

[0004] However, when converting timbre between different languages, existing speech synthesis methods suffer from unstable cross-language speech synthesis effects due to the habitual details of the source speaker's pronunciation and the problem of ambiguous or identical pronunciation. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a cross-language speech synthesis method, device, electronic device and storage medium, which can solve the problem of unstable cross-language speech synthesis effect existing in the prior art.

[0006] To solve the above technical problems, the first technical solution adopted in the present application is to provide a cross-language speech synthesis method, comprising: acquiring speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is speech data of a first language; the speech data of the object to be synthesized is speech data of a second language; the second language and the first language are languages ​​of different languages; acquiring a timbre feature matrix of the target object based on the speech data of the target object; and encoding the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized to obtain initial encoded data; in response to the initial encoded data including phonemes with the same pronunciation or ambiguously pronounced, adding a set note at the phonemes with the same pronunciation or ambiguously pronounced to obtain processed encoded data; decoding the timbre feature matrix of the target object and the processed encoded data of the object to be synthesized to obtain synthesized speech data based on the second language with the timbre feature matrix.

[0007] Among them, in response to the initial coding data including phonemes with the same pronunciation or ambiguous pronunciation, a set musical note is added to the phonemes with ambiguous pronunciation or the same pronunciation to obtain the processed coding data, which specifically includes: in response to the second language being English, multiple phonemes of English voice data are extracted from the voice data of the object to be synthesized based on the English dictionary standard; in response to some of the multiple phonemes having multiple combinations, and some of the phonemes have the same pronunciation in different combinations, set musical notes are added between some of the phonemes based on semantics to distinguish the phonemes with the same pronunciation, and then the coding data is obtained.

[0008] Among them, in response to the initial encoded data including phonemes with the same pronunciation or ambiguously pronounced, a set musical note is added to the phonemes with ambiguously pronounced or the same pronunciation to obtain the processed encoded data, and the step also includes: in response to the second language being Chinese, multiple phonemes of Chinese voice data are extracted from the voice data of the object to be synthesized based on the Chinese dictionary standard; in response to some of the multiple phonemes being ambiguously pronounced phonemes, set musical notes are added in front of some of the phonemes to make the pronunciation of some of the phonemes clear, and then the encoded data is obtained.

[0009] The set notes include spacers.

[0010] The step of acquiring the timbre feature matrix of the target object based on the speech data of the target object specifically includes: determining the mel spectrum of the speech data of the target object based on the speech data of the target object, and extracting the timbre feature matrix of the target object based on the mel spectrum.

[0011] Among them, the cross-language speech synthesis method is implemented by a speech synthesis model, and the speech synthesis model includes an encoder, an attention network and a decoder that are cascaded with each other; the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are encoded to obtain the initial encoded data, including: the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are encoded by the encoder in the speech synthesis model to obtain the initial encoded data; and in response to the initial encoded data including phonemes with the same pronunciation or ambiguity in pronunciation, a set note is added to the phonemes with the same pronunciation or ambiguity in pronunciation, and the step of obtaining the processed encoded data includes: in response to the initial encoded data including phonemes with the same pronunciation or ambiguity in pronunciation, a set note is added to the phonemes with the same pronunciation or ambiguity in pronunciation, and the processed encoded data is obtained; the timbre feature matrix of the target object and the processed encoded data of the object to be synthesized are decoded, and the step of obtaining the synthesized speech data based on the second language with the timbre feature matrix includes: the timbre feature matrix and the processed encoded data are decoded in sequence by the attention network and the decoder in the speech synthesis model to obtain the synthesized speech data.

[0012] Among them, the attention network includes an attention mechanism based on the forward position.

[0013] Among them, the step of obtaining a speech synthesis model specifically includes: obtaining a first training corpus based on a first language, and using the first training corpus to train the speech synthesis model to be trained to obtain a speech synthesis model after preliminary training; obtaining a second training corpus based on a second language, and using the second training corpus to train the speech synthesis model after preliminary training to obtain a speech synthesis model.

[0014] To solve the above technical problems, the second technical solution adopted in the present application is to provide a cross-language speech synthesis device, including: an acquisition module, used to acquire speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is speech data of a first language; the speech data of the object to be synthesized is speech data of a second language; the second language and the first language are languages ​​of different languages; a timbre feature matrix extraction module, used to acquire the timbre feature matrix of the target object based on the speech data of the target object; an encoding module, used to encode the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized to obtain initial encoded data; in response to the initial encoded data including phonemes with the same pronunciation or ambiguously pronounced, a set note is added to the phonemes with the same pronunciation or ambiguously pronounced to obtain processed encoded data; a decoding module, used to decode the timbre feature matrix of the target object and the processed encoded data of the object to be synthesized to obtain synthesized speech data based on the second language with a timbre feature matrix.

[0015] To solve the above technical problems, the third technical solution adopted in the present application is to provide an electronic device, comprising: a memory for storing program data, which, when executed, implements the steps in the cross-language speech synthesis method as described in any one of the above items; and a processor for executing program instructions stored in the memory to implement the steps in the cross-language speech synthesis method as described in any one of the above items.

[0016] In order to solve the above technical problems, the fourth technical solution adopted in the present application is to provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the cross-language speech synthesis method as described in any one of the above items are implemented.

[0017] The beneficial effects of the present application are as follows: Different from the prior art, the present application provides a cross-language speech synthesis method, system, device and computer-readable storage medium, which encodes the phonemes, pitches and phoneme duration of the speech data of the object to be synthesized to generate initial encoding data, and can refine the speech features on the basis of realizing speech synthesis, improve the precision and accuracy of speech synthesis, and thus improve the speech quality of the synthesized speech. Furthermore, by adding a set note to the phonemes with the same pronunciation or fuzzy pronunciation in the initial encoding data, the pronunciation content based on the second language can be made clearer, so that when cross-language speech synthesis is performed based on the timbre feature matrix of the target object in the subsequent step, the pronunciation habits of the second language are retained, the language transferability is improved, and the cross-language speech synthesis effect is then improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a flowchart of the first implementation method of the cross-language speech synthesis method of the present application;

[0020] Figure 2 It is a flowchart of the second implementation method of the cross-language speech synthesis method of the present application;

[0021] Figure 3 It is a flowchart of the third implementation method of the cross-language speech synthesis method of the present application;

[0022] Figure 4 It is a structural schematic diagram of an implementation method of a cross-language speech synthesis device of the present application;

[0023] Figure 5 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0024] Figure 6 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless otherwise clearly indicated above, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0027] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0028] It should be understood that the terms "include", "comprises" or any other variations used herein are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "includes..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0029] See also Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the cross-language speech synthesis method of the present application. Figure 1 As shown, in this embodiment, the method includes:

[0030] S11: Acquire speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is speech data in a first language; the speech data of the object to be synthesized is speech data in a second language; and the second language and the first language are different languages.

[0031] In this embodiment, the voice data of the target object may include voice audio input by the target object, or songs, operas recorded for the target object, or other texts input by the target object that include pronunciation data (such as broadcasts, recitations, etc.). The specific type of the voice data of the target object is not limited here, as long as it can reflect the voice characteristics of the target object as much as possible.

[0032] Among them, the target object is a specific natural person who can speak.

[0033] In this embodiment, the speech data of the object to be synthesized may include artificially emitted audio data or audio data output by a deep neural network, wherein the specific type of the object to be synthesized can be set based on actual conditions and is not limited here.

[0034] In this embodiment, as long as the languages ​​used in the speech data of the target object and the speech data of the object to be synthesized are different languages, there is no limitation on the specific languages ​​of the first language and the second language.

[0035] For example, in a specific implementation scenario, the first language is Chinese and the second language is English. In another specific implementation scenario, the first language may be English and the second language may be Chinese. In yet another specific implementation scenario, the first language may be Chinese and the second language may be other Western languages ​​other than English or Japanese, Korean, etc.

[0036] S12: Acquire a timbre feature matrix of the target object based on the speech data of the target object.

[0037] In this embodiment, the mel-spectrogram of the speech data of the target object is determined based on the speech data of the target object, and the timbre feature matrix of the target object is extracted based on the mel-spectrogram.

[0038] Among them, the Mel spectrum, also known as the Mel spectrogram, can reflect the audio characteristics.

[0039] In a specific implementation scenario, the mel spectrum can be obtained by first obtaining the line spectrum of the speech data of the target object, and then performing a weighted summation of the line spectrum on a mel scale. In another specific implementation scenario, the mel spectrum can also be obtained by first obtaining the line spectrum of the speech data of the target object, and then inputting the line spectrum into a mel filter group for filtering. The specific method for obtaining the mel spectrum is not limited here.

[0040] The features in the timbre feature matrix may include one or more features such as zero-crossing rate, spectrum centroid, spectrum attenuation, chromaticity frequency, etc., which are not specifically limited here.

[0041] In a specific implementation scenario, the timbre feature matrix of the object to be synthesized can be extracted based on the Mel spectrum by a three-level center clipping cross-correlation function method. In another specific implementation scenario, the timbre feature matrix of the object to be synthesized can be extracted based on the Mel spectrum by a Newton-Gaussian nonlinear estimation method. In yet another specific implementation scenario, the timbre feature matrix of the object to be synthesized can also be extracted based on the Mel spectrum by a deep learning network. Among them, the specific method for extracting the timbre feature matrix is ​​not limited here.

[0042] S13: Encode the phonemes, pitches and phoneme durations of the speech data to be synthesized to obtain initial encoded data.

[0043] In this implementation manner, firstly, the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are acquired based on the speech data of the object to be synthesized.

[0044] Among them, phoneme is the smallest unit of speech divided according to the natural properties of speech. Pitch refers to various sounds with different pitches, that is, the height of the sound, one of the basic characteristics of the sound. Phoneme duration refers to the pronunciation duration of the corresponding phoneme.

[0045] In a specific implementation scenario, the phoneme, pitch and phoneme duration of the voice data of the object to be synthesized can be obtained by inputting the voice data of the object to be synthesized into a pre-trained deep learning model for prediction. In another specific implementation scenario, the phoneme of the voice data of the object to be synthesized can be analyzed based on the dictionary corresponding to the second language, the pitch of the voice data of the object to be synthesized can be analyzed based on the international standard sound, and the voice data of the object to be synthesized can be predicted by the pre-trained deep learning model to obtain the phoneme duration of the voice data of the object to be synthesized.

[0046] Among them, the speech synthesis of this embodiment needs to improve the effect of speech synthesis. Therefore, the phonemes, pitches and phoneme durations of the object to be synthesized obtained in this step are indispensable, so as to refine the speech features of the object to be synthesized to the greatest extent, so that when performing subsequent speech synthesis, the feature accuracy of the cross-language synthesized speech can be improved based on the multiple features of the phonemes, pitches and phoneme durations of the object to be synthesized, thereby improving the accuracy of the cross-language synthesized speech.

[0047] In this embodiment, after the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are obtained, the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are encoded to obtain initial encoded data.

[0048] The execution order of this step and step S12 is not particular, and this step can be executed first and then step S12, or step S12 can be executed first and then this step, or this step and step S12 can be executed at the same time. Specifically, this step only needs to be executed before executing step S14, and step S12 only needs to be executed before executing step S15.

[0049] S14: In response to the initial coded data including phonemes with the same pronunciation or ambiguously pronounced, a set musical note is added to the phonemes with the same pronunciation or ambiguously pronounced to obtain processed coded data.

[0050] Specifically, since different languages ​​have different pronunciation habits and all have the problem of unclear expression of some phonemes, this embodiment needs to add a set note to the phonemes with the same pronunciation or ambiguous pronunciation to more accurately model the pronunciation phonemes and make the pronunciation content clearer. In the subsequent cross-language speech synthesis based on the timbre feature matrix of the target object, the pronunciation habits of the second language are retained, the language transferability is improved, and then the cross-language speech synthesis effect is improved.

[0051] S15: Decoding the timbre feature matrix of the target object and the processed coded data of the object to be synthesized to obtain synthesized speech data based on the second language having the timbre feature matrix.

[0052] In this embodiment, after the processed coded data is obtained, the timbre feature matrix and the processed coded data are decoded together to obtain synthesized speech data based on the second language having the timbre feature matrix.

[0053] Different from the prior art, this embodiment encodes the phonemes, pitches and phoneme duration of the speech data of the object to be synthesized to generate initial encoding data, which can refine the speech features on the basis of realizing speech synthesis, improve the precision and accuracy of speech synthesis, and thus improve the speech quality of the synthesized speech. Furthermore, by adding a set note to the phonemes with the same pronunciation or fuzzy pronunciation in the initial encoding data, the pronunciation content based on the second language can be made clearer, so that when cross-language speech synthesis is performed based on the timbre feature matrix of the target object in the subsequent process, the pronunciation habits of the second language are retained, the language transferability is improved, and the cross-language speech synthesis effect is then improved.

[0054] See also Figure 2 , Figure 2 It is a flow chart of the second embodiment of the cross-language speech synthesis method of the present application. In this embodiment, the cross-language speech synthesis method is implemented by a speech synthesis model. Specifically, the speech synthesis model is a model trained using training corpora of different languages. The steps of obtaining the speech synthesis model are as follows: obtaining a first training corpus based on a first language, and training a speech synthesis model to be trained using the first training corpus to obtain a speech synthesis model after preliminary training. The speech synthesis model after preliminary training is trained using a second training corpus to obtain a speech synthesis model.

[0055] It can be understood that the speech synthesis model after preliminary training can perform speech synthesis tasks on the first language, and the speech synthesis model after training can perform speech synthesis based on the first language and the second language.

[0056] like Figure 2 As shown, the cross-language speech synthesis method includes:

[0057] S21: Acquire speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is speech data in a first language; the speech data of the object to be synthesized is speech data in a second language; and the second language and the first language are different languages.

[0058] In this embodiment, the voice data of the target object may include voice audio input by the target object, or songs, operas recorded for the target object, or other texts input by the target object that include pronunciation data (such as broadcasts, recitations, etc.). The specific type of the voice data of the target object is not limited here, as long as it can reflect the voice characteristics of the target object as much as possible.

[0059] Among them, the target object is a specific natural person who can speak.

[0060] In this embodiment, the speech data of the object to be synthesized may include artificially emitted audio data or audio data output by a deep neural network, wherein the specific type of the object to be synthesized can be set based on actual conditions and is not limited here.

[0061] In this implementation, the second language is English, and the first language can be any language except English, which is not limited in this application.

[0062] S22: Acquire a timbre feature matrix of the target object based on the speech data of the target object.

[0063] In this embodiment, the speech data of the target object is first framed, windowed and Fourier transformed to obtain the linear spectrum of the speech data of the target object, and then the linear spectrum is input into a Mel filter bank for filtering to obtain a Mel spectrum, and the Mel spectrum is input into a deep learning network, and the timbre feature matrix of the object to be synthesized is extracted using the deep learning network. Among them, the Fourier transform includes a short-time Fourier transform (STFT) or other Fourier transforms. Specifically, after the speech data of the target object is framed, windowed and Fourier transformed, the time-frequency domain signal of the speech data of the target object can be obtained, and then the linear spectrum of the speech data of the target object is generated based on the time-frequency domain signal of the speech data of the target object.

[0064] In a specific implementation scenario, the speech data of the target object can be framed, windowed, and Fourier transformed to first obtain the time-frequency domain signal of the speech data of the target object, and then take the amplitude of the time-frequency domain signal to generate the linear spectrum of the speech data of the target object, so as to refine the features in the linear spectrum to improve the extraction speed, and then input the linear spectrum into a Mel filter group for filtering to obtain a Mel spectrum, and finally input the Mel spectrum into a deep learning network to extract the timbre information of the target object, thereby using the deep learning network to extract the timbre feature matrix of the target object.

[0065] Among them, the deep learning network of this step can include a combination network of a bidirectional gated recurrent unit (GRU) network + a fully connected network (Fully Connected Layers, FC) + a Bayesian network (Bayesian network, BN), a long short-term memory model, a convolutional neural network or a recurrent neural network and other deep learning networks. The specific type can be set based on the actual situation and is not limited here. And the deep learning network of this step is a network that has been pre-trained for extracting the timbre feature matrix based on the Mel spectrum and can be applied.

[0066] S23: Encode the phonemes, pitches and phoneme durations of the speech data to be synthesized to obtain initial encoded data.

[0067] In this embodiment, the cross-language speech synthesis method is implemented by a speech synthesis model, which includes an encoder, an attention network, and a decoder that are cascaded to each other.

[0068] Specifically, after obtaining the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized, the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are input into the speech synthesis model, and the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are encoded by the encoder in the speech synthesis model to obtain initial encoded data.

[0069] The speech synthesis model of this embodiment is based on the tacotron1 model and improved. Among them, since the feature extraction of phonemes, rhythms and pitches does not need to take into account contextual information, this embodiment removes the bidirectional GRU module in the tacotron1 model to simplify the initial model structure, reduce invalid model operations, and thus improve the speech synthesis efficiency of the initial model and the final speech synthesis model.

[0070] In a specific implementation scenario, the input of the speech synthesis model can be (iou C310), where iou is the phoneme information, C3 is the pitch information, and 10 is the phoneme duration corresponding to the phoneme information.

[0071] The execution order of this step and step S22 is not particular, and this step can be executed first and then step S22, or step S22 can be executed first and then this step, or this step and step S22 can be executed at the same time. Specifically, this step only needs to be executed before executing step S24, and step S22 only needs to be executed before executing step S26.

[0072] S24: In response to the second language being English, extracting a plurality of phonemes of English speech data from the speech data of the object to be synthesized based on an English dictionary standard.

[0073] In this embodiment, the speech synthesis model stores all English phonemes based on English dictionary standards, and can recognize and extract multiple phonemes in English speech data.

[0074] In a specific implementation scenario, the English dictionary includes the cmudict English dictionary compiled by Carnegie Mellon University.

[0075] S25: In response to the fact that some of the multiple phonemes have multiple combinations and some of the phonemes have the same pronunciation in different combinations, set notes are added between the some of the phonemes based on semantics to distinguish the phonemes with the same pronunciation, and then the encoded data is obtained.

[0076] In this embodiment, in response to the fact that some of the multiple phonemes have multiple combinations and some of the phonemes have the same pronunciation in different combinations, the encoder adds set notes between the partial phonemes based on semantics to distinguish the phonemes with the same pronunciation, and then obtains encoded data.

[0077] Specifically, since there are too many words in English and some of the phonemes have multiple combinations, it is necessary to determine whether the pronunciation of these phonemes is the same in multiple combinations. For example, the phoneme combination [g ih1 vb ae2 k] can form both giveback (noun, welfare return) and give back (verb, feedback), but the former is a noun and the latter is a verb phrase consisting of two words. In other words, although the pronunciation of the phoneme combination is the same in two different combinations, the meanings they express are different. Therefore, it is necessary to distinguish different combinations based on semantics.

[0078] In this embodiment, when some phonemes form a word, the input method of the phonemes remains unchanged; when some phonemes form a phrase, set notes are added between the words in the phrase to distinguish the phonemes with the same pronunciation, and then the encoded data is obtained. Taking the phoneme combination [g ih1 vb ae2 k] in the above text as an example, when the semantic meaning is the noun giveback, the input of the phoneme is still [g ih1 vb ae2 k], and when the phonetic meaning is the verb phrase give back, the input of the phoneme is changed to [gih1 v(sep)b ae2 k], so as to distinguish the phonemes with the same pronunciation.

[0079] Among them, (sep) is the separator.

[0080] It can be understood that by adding set notes between some phonemes that have multiple combinations and have the same pronunciation in different combinations, different combinations of phonemes can be better distinguished, thereby ensuring the accuracy of voice data and further improving the effect of cross-language speech synthesis.

[0081] S26: Decoding the timbre feature matrix of the target object and the processed coded data of the object to be synthesized to obtain synthesized speech data based on the second language having the timbre feature matrix.

[0082] In this embodiment, the timbre feature matrix of the target object and the processed coded data of the object to be synthesized are decoded to obtain English-based synthesized speech data having the timbre feature matrix.

[0083] In this embodiment, the extracted timbre feature matrix of the target object and the encoded data are taken as decoding objects and input into the speech synthesis model, so that the timbre feature matrix and the encoded data are decoded by the speech synthesis model to obtain synthesized speech data based on the second language with the timbre feature matrix.

[0084] Specifically, the timbre feature matrix and the encoded data are decoded in turn through the attention network and the decoder in the speech synthesis model to obtain the synthesized speech data.

[0085] Among them, the attention network includes a forward location-based attention mechanism (ForwardLocationBasedAttention).

[0086] In this implementation, the attention mechanism BahdanauAttention in the original tacotron1 model is replaced with a forward location based attention mechanism (Forward Location Based Attention).

[0087] It can be understood that the location-based attention mechanism (Location Based Attention) will ignore or reduce silence in speech data, which can reduce the ineffective analysis and consumption of silence in speech synthesis, thereby improving the efficiency and quality of speech synthesis. This embodiment adopts the forward position-based attention mechanism as the attention mechanism, which can further reduce the ineffective analysis and consumption of silence, thereby improving the efficiency and quality of speech synthesis.

[0088] In another specific application scenario, the speech synthesis model can also add a sentence segmentation model, which segments the decoded data into sentences, thereby separating the entire synthesized speech to make it conform to pronunciation rules, thereby improving the quality of speech synthesis.

[0089] In another specific application scenario, the decoded data can be further input into the sentence segmentation model in the speech synthesis model for segmentation to obtain synthesized speech. The sentence segmentation model may include a Stop Token network, which segments the decoded data through the trained sentence segmentation model to determine the breakpoints of the decoded data, thereby separating the entire synthesized speech to make it conform to the pronunciation rules, thereby improving the quality of speech synthesis.

[0090] In another specific application scenario, the sentence segmentation model can also be set in the decoder of the speech synthesis model. While the decoder is decoding, the sentence segmentation model is used to determine the decoding breakpoint when the decoding ends, thereby separating the decoded data segments. Then, the entire synthesized speech can be separated based on the separated decoded data segments, so that it conforms to the pronunciation rules, thereby improving the quality of speech synthesis.

[0091] In a specific application scenario, the segmentation rules of the target object can be manually input into the segmentation model so that the segmentation model can segment the decoded data or the data being decoded based on the segmentation rules of the target object. In another specific application scenario, the segmentation model can be trained in advance based on the segmentation of sample audio until it can segment based on the phonemes, pitches, and phoneme durations of the target object.

[0092] Therefore, through the above improvements, the speech synthesis efficiency and effect of the speech synthesis model of this embodiment are better than the traditional tacotron1 model.

[0093] In this embodiment, the timbre feature matrix is ​​input into the decoder of the speech synthesis model for decoding, and the timbre of the synthesized speech can be adjusted based on the timbre feature matrix, thereby controlling the timbre of the synthesized speech. That is, by modifying the timbre features of the Mel spectrum, a timbre feature matrix with different features can be obtained, so that the actual synthesized speech output has a different timbre.

[0094] The speech synthesis method of this embodiment can be applied to the cross-language work of karaoke software or similar software, so that users whose native language is Chinese or other languages ​​other than English can synthesize English songs or English voices based on the user's own timbre, and the synthesized voice finally output restores the user's own timbre, thereby realizing cross-language song synthesis or speech synthesis. Further, by adding a set note between some phonemes with the same pronunciation in the initial coding data, the pronunciation content based on English can be made clearer, so that when cross-language speech synthesis is performed based on the user's timbre feature matrix in the subsequent process, the pronunciation habits of English are retained, the language transferability is improved, and the cross-language speech synthesis effect is then improved. In addition, since all English phonemes based on the English dictionary standard are stored in the speech synthesis model, and the pitch and phoneme duration are the same for any language, when switching to English, it is only necessary to modify the corresponding phonemes in the first language to the corresponding phonemes in English, thereby improving the convenience of cross-language speech synthesis.

[0095] See also Figure 3 , Figure 3 is a flow chart of the third embodiment of the cross-language speech synthesis method of the present application. In this embodiment, the cross-language speech synthesis method is implemented by a speech synthesis model, such as Figure 3 As shown, the cross-language speech synthesis method includes:

[0096] S31: Acquire speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is speech data in a first language; the speech data of the object to be synthesized is speech data in a second language; and the second language and the first language are different languages.

[0097] In this embodiment, the voice data of the target object may include voice audio input by the target object, or songs, operas recorded for the target object, or other texts input by the target object that include pronunciation data (such as broadcasts, recitations, etc.). The specific type of the voice data of the target object is not limited here, as long as it can reflect the voice characteristics of the target object as much as possible.

[0098] Among them, the target object is a specific natural person who can speak.

[0099] In this embodiment, the speech data of the object to be synthesized may include artificially emitted audio data or audio data output by a deep neural network, wherein the specific type of the object to be synthesized can be set based on actual conditions and is not limited here.

[0100] In this implementation, the second language is Chinese, and the first language can be any language except Chinese, which is not limited in this application.

[0101] S32: Acquire a timbre feature matrix of the target object based on the speech data of the target object.

[0102] In this implementation, the specific method of obtaining the timbre feature matrix of the target object is the same as that in the above implementation. Please refer to the above text and will not be repeated here.

[0103] S33: Encode the phonemes, pitches and phoneme durations of the speech data to be synthesized to obtain initial encoded data.

[0104] In this embodiment, after obtaining the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized, the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are input into the speech synthesis model, and the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized are encoded by the encoder in the speech synthesis model to obtain initial encoded data.

[0105] Among them, the specific method for obtaining the initial encoded data and the speech synthesis model used are the same as those in the aforementioned implementation. Please refer to the previous text and will not be repeated here.

[0106] The execution order of this step and step S32 is not particular, and this step can be executed first and then step S32, or step S32 can be executed first and then this step, or this step and step S32 can be executed at the same time. Specifically, this step only needs to be executed before executing step S34, and step S32 only needs to be executed before executing step S36.

[0107] S34: In response to the second language being Chinese, extracting a plurality of phonemes of Chinese speech data from the speech data of the object to be synthesized based on a Chinese dictionary standard.

[0108] In this embodiment, the speech synthesis model stores all Chinese phonemes based on the Chinese dictionary standard, and can recognize and extract multiple phonemes in Chinese speech data.

[0109] Specifically, 422 pinyins in Chinese are obtained, and the 422 pinyins are divided into phonemes and stored in the speech synthesis model. For the input Chinese speech data, the pinyins corresponding to the speech data are divided into phonemes as the input of the speech synthesis model, so that the speech synthesis model can recognize the phonemes of the Chinese speech data.

[0110] In this implementation, since the speech synthesis model also stores all English phonemes based on the English dictionary standard, and there are some phonemes with the same pronunciation between Chinese and English phonemes, in order to save storage space, these phonemes with the same pronunciation are stored in the same set, and then the Chinese and English phonemes are stored separately.

[0111] S35: In response to some of the multiple phonemes being unclearly pronounced, set musical notes are added before the some of the phonemes to make the some of the phonemes pronounced clearly, thereby obtaining encoded data.

[0112] In this embodiment, in response to some of the multiple phonemes being ambiguously pronounced, the encoder adds set musical notes in front of the some of the phonemes to make the ambiguously pronounced phonemes clear, thereby obtaining encoded data.

[0113] Specifically, since there are pronunciations without initial consonants in Chinese, and phonemes without initial consonants have the problem of unclear pronunciation during speech synthesis, for example, the pinyin of "you" is "you", and the phonemes of "you" are [iou], which are phonemes without initial consonants. During synthesis, they are easily recognized as other Chinese characters due to their unclear pronunciation. Therefore, for phonemes without initial consonants, a set musical note is added in front of them. The set musical note is equivalent to a universal initial consonant, which can ensure that these phonemes without initial consonants are distinguishable, so that the pronunciation is clear and the meaning is clear.

[0114] The set notes include spacers.

[0115] It can be understood that by adding set musical notes before phonemes without initial consonants, the phonemes without initial consonants can be pronounced clearly, thereby improving the accuracy of speech data and further improving the effect of cross-language speech synthesis.

[0116] S36: Decoding the timbre feature matrix of the target object and the processed coded data of the object to be synthesized to obtain synthesized speech data based on the second language having the timbre feature matrix.

[0117] In this embodiment, the timbre feature matrix and the encoded data of the target object are decoded to obtain Chinese-based synthesized speech data having the timbre feature matrix.

[0118] In a specific implementation scenario, before decoding the timbre feature matrix, each operation step of the speech synthesis model is frozen respectively, and then the unfrozen operation steps are fine-tuned through small batch data to determine the impact of each operation step in the speech synthesis model on the timbre. Specifically, the following steps are included: (1) freezing the encoding step of the speech synthesis model, and then performing timbre training on the decoding step to determine the importance of the decoding step in the timbre synthesis of the entire model; (2) freezing the decoding step, and then performing timbre training on the encoding step to determine the importance of the encoding step in the timbre synthesis of the entire model. Finally, it is found that the decoding step has a greater impact on speech synthesis and timbre restoration.

[0119] It can be understood that in the present embodiment, since only the timbre feature matrix is ​​decoded without encoding and then decoding the timbre feature matrix or other additional operations, it is possible to reduce the ineffective operations of the model on the timbre feature matrix based on the specific impact of the model on the timbre, thereby improving the synthesis efficiency of the speech synthesis model while ensuring speech synthesis based on the timbre feature matrix.

[0120] In this embodiment, after the encoded data is obtained, the timbre feature matrix and the encoded data are decoded together to obtain synthesized speech data based on the second language having the timbre feature matrix.

[0121] In a specific implementation scenario, the output of the encoder can be used as the input of the decoder, and the input of the encoder can be used as the label of the decoder to train the corresponding decoder, so that the timbre feature matrix and the encoded data are decoded by the decoder to obtain the synthesized speech of the object to be synthesized.

[0122] In this step of other embodiments, one or more decoding methods such as a greedy decoding algorithm, a beam search decoding algorithm, and a sampling-based decoding algorithm may be used to decode the timbre feature matrix and the encoded data to obtain the synthesized speech of the object to be synthesized. The decoding step of this step may correspond to the encoding in step S33 to facilitate speech synthesis.

[0123] Among them, since not only the encoded data is decoded but also the timbre feature matrix is ​​decoded at the same time, the timbre of the synthesized speech data can be restored based on the timbre of the target object, and the phonemes, pitches and phoneme durations of the synthesized speech data can be matched with the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized, so that the finally generated synthesized speech data can restore the pronunciation habits of Chinese while realizing cross-language timbre conversion, thereby improving the portability of the language and then improving the cross-language speech synthesis effect.

[0124] Correspondingly, the present application provides a cross-language speech synthesis device.

[0125] See also Figure 4 , Figure 4 Schematic diagram of the structure of an embodiment of the cross-language speech synthesis device of the present application. Figure 4 As shown, in this embodiment, the cross-language speech synthesis device 40 includes an acquisition module 41 , a timbre feature matrix extraction module 42 , an encoding module 43 and a decoding module 44 .

[0126] The acquisition module 41 is used to acquire the speech data of the target object and the speech data of the object to be synthesized; wherein the speech data of the target object is speech data in a first language; the speech data of the object to be synthesized is speech data in a second language; and the second language and the first language are different languages.

[0127] The timbre feature matrix extraction module 42 is used to obtain the timbre feature matrix of the target object based on the speech data of the target object.

[0128] The encoding module 43 is used to encode the phonemes, pitches and phoneme durations of the speech data of the synthesis object to obtain initial encoded data; in response to the initial encoded data including phonemes with the same pronunciation or ambiguous pronunciation, a set note is added to the phonemes with the same pronunciation or ambiguous pronunciation to obtain processed encoded data.

[0129] The decoding module 44 is used to decode the timbre feature matrix of the target object and the processed coded data of the object to be synthesized to obtain synthesized speech data based on the second language with the timbre feature matrix.

[0130] For the specific synthesis process, please refer to the relevant text descriptions in steps S11 to S15, steps S21 to S26 and steps S31 to S36, which will not be repeated here.

[0131] Different from the prior art, this embodiment obtains the timbre feature matrix of the target object through the timbre feature matrix extraction module 42, and encodes the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized through the encoding module 43 to obtain initial encoding data. On the basis of realizing speech synthesis, it is possible to refine the speech features and improve the precision and accuracy of speech synthesis, thereby improving the speech quality of the synthesized speech; further, the encoding module 43 is used to add a set note to the phonemes with the same pronunciation or fuzzy pronunciation in the initial encoding data, which can make the pronunciation content based on the second language clearer. Finally, the decoding module 44 performs cross-language speech synthesis based on the timbre feature matrix of the target object, which can retain the pronunciation habits of the second language, improve the portability of the language, and then improve the cross-language speech synthesis effect.

[0132] Correspondingly, the present application provides an electronic device.

[0133] See also Figure 5 , Figure 5 Schematic diagram of the structure of an electronic device of the present application. Figure 5 As shown, in this embodiment, the electronic device 50 includes a memory 51 and a processor 52.

[0134] In this embodiment, the memory 51 is used to store program data, and when the program data is executed, the steps in the cross-language speech synthesis method as described in any one of the above items are implemented; the processor 52 is used to execute the program instructions stored in the memory 51 to implement the steps in the cross-language speech synthesis method as described in any one of the above items.

[0135] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps in any of the above-mentioned cross-language speech synthesis methods. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by multiple integrated circuit chips.

[0136] Different from the prior art, the present embodiment generates initial coding data by encoding the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized through the processor 52, which can refine the speech features and improve the precision and accuracy of speech synthesis on the basis of achieving speech synthesis, thereby improving the speech quality of the synthesized speech. Furthermore, by adding a set note to the phonemes with the same pronunciation or fuzzy pronunciation in the initial coding data, the pronunciation content based on the second language can be made clearer, so that when cross-language speech synthesis is performed based on the timbre feature matrix of the target object in the subsequent step, the pronunciation habits of the second language are retained, the language transferability is improved, and the cross-language speech synthesis effect is then improved.

[0137] Correspondingly, the present application provides a computer-readable storage medium.

[0138] See also Figure 6 , Figure 6 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application.

[0139] The computer-readable storage medium 60 includes a computer program 601 stored on the computer-readable storage medium 60 . When the computer program 601 is executed by the processor, the steps in any of the above-mentioned cross-language speech synthesis methods are implemented.

[0140] Specifically, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 60. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium 60, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 60 includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0145] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A cross-language speech synthesis method, characterized in that: include: Acquire speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is speech data in a first language; the speech data of the object to be synthesized is speech data in a second language; and the second language is a language different from the first language; Acquire a timbre feature matrix of the target object based on the speech data of the target object; and Encoding the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized to obtain initial encoded data; In response to the initial coded data including phonemes with the same pronunciation or ambiguously pronounced, adding a set musical note at the phonemes with the same pronunciation or ambiguously pronounced to obtain processed coded data, specifically comprising: in response to the second language being English, extracting multiple phonemes of English voice data from the voice data of the object to be synthesized based on an English dictionary standard; in response to some of the multiple phonemes having multiple combinations, and the some phonemes having the same pronunciation in different combinations, adding the set musical note between the some phonemes based on semantics to distinguish the phonemes with the same pronunciation, and then obtaining the coded data; or, In response to the second language being Chinese, extracting a plurality of phonemes of Chinese voice data from the voice data of the object to be synthesized based on a Chinese dictionary standard; in response to some of the phonemes among the plurality of phonemes being the phonemes with unclear pronunciation, adding the set musical notes before the some of the phonemes to make the some of the phonemes pronounced clearly, and then obtaining the encoded data; The timbre feature matrix of the target object and the processed coded data of the object to be synthesized are decoded to obtain synthesized speech data based on a second language having the timbre feature matrix.

2. The cross-language speech synthesis method according to claim 1, characterized in that: The set notes include spacers.

3. The cross-language speech synthesis method according to claim 1, characterized in that: The step of acquiring the timbre feature matrix of the target object based on the speech data of the target object specifically includes: The mel spectrum of the speech data of the target object is determined based on the speech data of the target object, and the timbre feature matrix of the target object is extracted based on the mel spectrum.

4. The cross-language speech synthesis method according to claim 1, characterized in that: The cross-language speech synthesis method is implemented by a speech synthesis model, which includes an encoder, an attention network and a decoder that are cascaded to each other; The step of encoding the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized to obtain initial encoded data comprises: The steps of encoding the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized by the encoder in the speech synthesis model to obtain the initial encoded data; and in response to the initial encoded data including phonemes with the same pronunciation or ambiguously pronounced, adding a set note at the phonemes with the same pronunciation or ambiguously pronounced to obtain the processed encoded data include: In response to the initial coded data including the phonemes with the same pronunciation or ambiguity in pronunciation, the encoder adds a set musical note at the phonemes with the same pronunciation or ambiguity in pronunciation to obtain the processed coded data; The step of decoding the timbre feature matrix of the target object and the processed coded data of the object to be synthesized to obtain synthesized speech data based on the second language having the timbre feature matrix comprises: The timbre feature matrix and the processed encoded data are decoded in turn by the attention network and the decoder in the speech synthesis model to obtain the synthesized speech data.

5. The cross-language speech synthesis method according to claim 4, characterized in that: The attention network includes a forward position based attention mechanism.

6. The cross-language speech synthesis method according to claim 4, characterized in that: The step of obtaining the speech synthesis model specifically includes: Acquire a first training corpus based on the first language, and use the first training corpus to train a speech synthesis model to be trained to obtain a preliminarily trained speech synthesis model; A second training corpus based on the second language is obtained, and the second training corpus is used to train the preliminarily trained speech synthesis model to obtain the speech synthesis model.

7. A cross-language speech synthesis device, characterized in that: include: An acquisition module, used to acquire speech data of a target object and speech data of an object to be synthesized; wherein the speech data of the target object is speech data in a first language; the speech data of the object to be synthesized is speech data in a second language; and the second language is a language different from the first language; A timbre feature matrix extraction module, used to obtain the timbre feature matrix of the target object based on the speech data of the target object; The encoding module is used to encode the phonemes, pitches and phoneme durations of the speech data of the object to be synthesized to obtain initial encoding data; in response to the initial encoding data including phonemes with the same pronunciation or ambiguously pronounced, add a set musical note at the phonemes with the same pronunciation or ambiguously pronounced to obtain processed encoding data, specifically comprising: in response to the second language being English, extracting multiple phonemes of English speech data from the speech data of the object to be synthesized based on English dictionary standards; in response to some of the multiple phonemes having multiple combinations, and the some phonemes having the same pronunciation in different combinations, adding the set musical note between the some phonemes based on semantics to distinguish the phonemes with the same pronunciation, and then obtaining the encoding data; or, In response to the second language being Chinese, extracting a plurality of phonemes of Chinese voice data from the voice data of the object to be synthesized based on a Chinese dictionary standard; in response to some of the phonemes among the plurality of phonemes being the phonemes with unclear pronunciation, adding the set musical notes before the some of the phonemes to make the some of the phonemes pronounced clearly, and then obtaining the encoded data; A decoding module is used to decode the timbre feature matrix of the target object and the processed encoded data of the object to be synthesized to obtain synthesized speech data based on a second language having the timbre feature matrix.

8. An electronic device, characterized in that: include: A memory for storing program data, wherein the program data, when executed, implements the steps of the cross-language speech synthesis method according to any one of claims 1 to 6; A processor, configured to execute the program instructions stored in the memory to implement the steps in the cross-language speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the cross-language speech synthesis method according to any one of claims 1 to 6 are implemented.

Images