Title core position prediction method and device, equipment and storage medium

Abstract

The invention discloses a tone core position prediction method and device, equipment and a storage medium. The tone core position prediction method comprises the following steps: obtaining a target phrase pause feature, a target phoneme feature and a target text feature corresponding to a target daily text; fusing the target phrase pause feature, the target phoneme feature and the target text feature to obtain a target fusion feature; and using the target fusion feature to carry out tone-kernel position prediction to obtain a tone-kernel position prediction result corresponding to the target daily text. In this way, the accuracy of the prediction result of the tone position of the target daily text can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a method, device, equipment and storage medium for predicting a modulation nucleus position. Background Art

[0002] Speech synthesis, also known as TTS (Text To Speech), includes front-end modules and back-end modules. The front-end module is used to extract linguistic and acoustic features related to pronunciation from the input text. The quality of the front-end module will directly affect the accuracy indicators of the synthesized audio, and thus affect the overall performance. Therefore, the front-end module is an important part of the entire speech synthesis process.

[0003] Japanese is a language with high pitch stress. The position of the intonation nucleus (i.e., the intonation nucleus or stress position) in Japanese not only determines the pitch changes of syllables in a sentence, but also reflects the semantic emphasis and emotional expression of the sentence. In the process of Japanese speech synthesis, the front-end module is usually required to predict the intonation nucleus position of the Japanese text. For Japanese speech synthesis, the accuracy of the intonation nucleus position prediction directly affects the naturalness and intelligibility of the synthesized speech.

[0004] Therefore, how to improve the accuracy of Japanese intonation nucleus position prediction has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a method, device, equipment and computer-readable storage medium for predicting the tonal core position, which can improve the accuracy of the tonal core position prediction results of the target Japanese text.

[0006] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a method for predicting the position of the intonation nucleus, the method comprising: obtaining the target phrase pause features, target phoneme features and target text features corresponding to the target Japanese text; fusing the target phrase pause features, target phoneme features and target text features to obtain the target fusion features; using the target fusion features to predict the position of the intonation nucleus, and obtaining the prediction result of the position of the intonation nucleus corresponding to the target Japanese text.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide a tone nucleus position prediction device, which includes: an acquisition module, used to obtain the target phrase pause features, target phoneme features and target text features corresponding to the target Japanese text; a fusion module, used to fuse the target phrase pause features, target phoneme features and target text features to obtain the target fusion features; a prediction module, used to predict the tone nucleus position using the target fusion features to obtain the tone nucleus position prediction result corresponding to the target Japanese text.

[0008] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above-mentioned core position prediction method.

[0009] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed by a processor to implement the above-mentioned tuning core position prediction method.

[0010] The above scheme obtains the target fusion feature by fusing the target phrase pause feature, target phoneme feature and target text feature corresponding to the target Japanese text, and uses the target fusion feature to predict the tone nucleus position, thereby obtaining the tone nucleus position prediction result corresponding to the target Japanese text. Since the text and pronunciation within the boundary of the Japanese tone nucleus position and the phrase pause are related, and the target phrase pause feature, target phoneme feature and target text feature can respectively reflect the phrase pause information, pronunciation information and text semantic information of the target Japanese text, therefore, using the target fusion feature to predict the tone nucleus position can integrate the information of the three different dimensions of phrase pause, pronunciation and text semantics of the target Japanese text, so that the predicted tone nucleus position prediction result has a higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of an embodiment of a method for predicting the core position provided by the present application;

[0012] Figure 2 It is a flowchart of another embodiment of the method for predicting the core position provided by the present application;

[0013] Figure 3 is a flow chart of an embodiment of determining pause features of a target phrase provided by the present application;

[0014] Figure 4 It is a flowchart of an embodiment of obtaining target phoneme features provided by the present application;

[0015] Figure 5 This is a flowchart of an embodiment of obtaining target fusion features provided by the present application;

[0016] Figure 6 It is a schematic diagram of the structure of the target prediction model provided by this application;

[0017] Figure 7 It is a schematic diagram of the framework of an embodiment of a device for predicting the position of a core provided by the present application;

[0018] Figure 8It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0019] Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0021] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0022] It should be noted that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The term "several" means at least one. In addition, the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0023] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of an embodiment of a method for predicting the position of the core provided in the present application. It should be noted that if there are substantially the same results, the method of the present application is not limited to Figure 1 The process sequence shown is limited. Figure 1 As shown, the method comprises the following steps:

[0024] S11: Obtain target phrase pause features, target phoneme features, and target text features corresponding to the target Japanese text.

[0025] The target Japanese text is the Japanese text to be subjected to speech synthesis. The target phrase pause feature, the target phoneme feature and the target text feature are all extracted from the target Japanese text. The target phrase pause feature includes phrase pause information corresponding to the target Japanese text, the target phoneme feature includes pronunciation information corresponding to the target Japanese text, and the target text feature includes text semantic information corresponding to the target Japanese text.

[0026] S12: Fusing the target phrase pause feature, the target phoneme feature, and the target text feature to obtain a target fusion feature.

[0027] The target phrase pause feature, target phoneme feature and target text feature may not be features of the same granularity. For example, the target phrase pause feature is a feature of character granularity, the target phoneme feature is a feature of phoneme granularity and the target text feature is a feature of word granularity. To facilitate feature fusion and subsequent prediction of the nucleus position, the target phrase pause feature, target phoneme feature and target text feature are first converted into features of syllable granularity, and then the target phrase pause feature of syllable granularity, the target phoneme feature of syllable granularity and the target text feature of syllable granularity are fused to obtain the target fusion feature of syllable granularity. The target fusion feature of syllable granularity is based on syllables, and each syllable contains the phrase pause information, pronunciation information and text semantic information corresponding to the syllable.

[0028] In one embodiment, the target phrase pause features at the syllable granularity, the target phoneme features at the syllable granularity, and the target text features at the syllable granularity may be directly concatenated by syllable to obtain the target fusion features at the syllable granularity.

[0029] In another embodiment, the target phrase pause feature and the target phoneme feature can only reflect the phrase pause information and pronunciation information of the target Japanese text respectively, and do not contain deep information. Therefore, in order to further improve the accuracy of subsequent tone nucleus position prediction, in this embodiment, the target phrase pause feature of syllable granularity and the target phoneme feature of syllable granularity are firstly extracted to obtain the first extracted feature of syllable granularity, and then the target phoneme feature of syllable granularity and the first extracted feature of syllable granularity are feature spliced ​​by syllable to obtain the target fusion feature.

[0030] S13: Utilize the target fusion feature to predict the position of the tone core, and obtain the prediction result of the tone core position corresponding to the target Japanese text.

[0031] In this embodiment, the target fusion feature is obtained by fusing the target phrase pause feature, the target phoneme feature and the target text feature corresponding to the target Japanese text, and the target fusion feature is used to predict the tone core position, so as to obtain the tone core position prediction result corresponding to the target Japanese text. Since the text and pronunciation within the boundary of the Japanese tone core position and the phrase pause are related, and the target phrase pause feature, the target phoneme feature and the target text feature can respectively reflect the phrase pause information, pronunciation information and text semantic information of the target Japanese text, therefore, the prediction of the tone core position using the target fusion feature can integrate the information of the three different dimensions of the phrase pause, pronunciation and text semantics of the target Japanese text, so that the predicted tone core position prediction result has a higher accuracy.

[0032] See also Figure 2 , Figure 2 FIG. 2 is a flow chart of another embodiment of the method for predicting the position of the core provided by the present application. Figure 2As shown, the method comprises the following steps:

[0033] S21: Process the target Japanese text using the pre-trained text processing network to obtain word-granularity text features corresponding to the target Japanese text.

[0034] The word-granularity text feature is based on characters, and includes sub-text features corresponding to each character of the target Japanese text, and the sub-text features of each character are multi-dimensional features.

[0035] In one embodiment, the pre-trained text processing network is word-granularity Bert (Bidirectional Encoder Representations from Transformers) to obtain word-granularity text features with context information. Exemplarily, word-granularity Bert is a standard 12-layer 768-node structure. The structure includes 12 layers of Transformer encoders, each layer is responsible for processing different aspects of the input data, and captures context information through a self-attention mechanism, and the hidden layer size of each layer is 768, so that each character of the target Japanese text is mapped to a 768-dimensional feature.

[0036] Optionally, the pre-trained text processing network is not limited to the word-granularity Bert. In other implementations, other similar text processing networks may also be used to obtain word-granularity text features of the target Japanese text.

[0037] Considering that Japanese contains a large number of Kanji and Kana, and word segmentation is relatively fuzzy, directly processing each character in the target Japanese text instead of converting it into a higher-level language unit (such as a word or phrase) and then processing it helps the pre-trained text processing network to more accurately capture the detailed information in the target Japanese text, thereby improving the accuracy of the target phrase pause features, target phoneme features, and target text features subsequently acquired.

[0038] S22: Determine target phrase pause features, target phoneme features, and target text features of the target Japanese text based on the word-granularity text features.

[0039] In one embodiment, the step of determining the target phrase pause feature is performed by a prosody prediction network. It should be noted that the prosody of the target Japanese text in this embodiment mainly includes phrase pauses (i.e., L1 pauses) and long pauses (i.e., L3 pauses), wherein the phrase pauses are pauses at the stress phrase level in the target Japanese text, and the long pauses are pauses at the sentence or paragraph level in the target Japanese text.

[0040] The prosody prediction network is a cascaded MLP (Multilayer Perceptron) structure. Specifically, the prosody prediction network includes a first feature extraction subnetwork, a second feature extraction subnetwork, a long pause prediction subnetwork and a phrase pause prediction subnetwork.

[0041] Figure 3 FIG. 1 is a flow chart of an embodiment of determining pause features of a target phrase provided by the present application. Figure 3 As shown, the method comprises the following steps:

[0042] S301: Using the first feature extraction sub-network to extract character-sized text features, and obtain character-sized long pause extraction features.

[0043] The first feature extraction subnetwork is used to further abstract and transform the word-granularity text features output by the pre-trained text processing network to extract features that are beneficial to long pause prediction, namely, word-granularity long pause extraction features. The word-granularity long pause extraction features are in units of characters, and the word-granularity long pause extraction features include long pause sub-extraction features corresponding to each character of the target Japanese text, and the long pause sub-extraction features corresponding to each character are all multi-dimensional features (such as 768-dimensional features). Exemplarily, the first feature extraction subnetwork is a fully connected layer of 768 nodes.

[0044] S302: Using the long pause prediction sub-network to predict the character-sized long pause extraction features, to obtain the character-sized target long pause features.

[0045] In a specific embodiment, the target long pause feature includes the first classification results corresponding to each character of the target Japanese text. The first classification results of each character include the probabilities that each character belongs to non-long pauses, long pauses, and padding. Furthermore, the first classification results of each character can also be represented by one-hot labels. For example, the first classification result of a character is [0.1, 0.6, 0.3], where 0.1, 0.6, and 0.3 are the probabilities that the character belongs to non-long pauses, long pauses, and padding, respectively, and the corresponding one-hot labels are represented as [0, 1, 0].

[0046] S303: Using the second feature extraction sub-network to extract the character-sized long pause extraction features, to obtain the character-sized phrase pause extraction features.

[0047] The second feature extraction subnetwork is used to further abstract and transform the word-granularity long pause features output by the first feature extraction subnetwork to extract features that are beneficial to phrase pause prediction, namely, word-granularity phrase pause extraction features. The word-granularity phrase pause extraction features are based on characters, and the word-granularity phrase pause extraction features include phrase pause sub-extraction features corresponding to each character of the target Japanese text, and the phrase pause sub-extraction features corresponding to each character are all multidimensional features. Exemplarily, the second feature extraction subnetwork is also a fully connected layer of 768 nodes.

[0048] S304: Using the phrase pause prediction sub-network to predict the character-level phrase pause extraction features, and obtain the character-level target phrase pause features.

[0049] In a specific embodiment, the target pause feature includes the second classification results corresponding to each character of the target Japanese text. The second classification results of each character include the probability that each character belongs to a non-phrase pause, a phrase pause, and padding. Furthermore, the second classification results of each character can also be represented using a one-hot label.

[0050] Exemplarily, both the long pause prediction subnetwork and the short pause prediction subnetwork are fully connected layers.

[0051] In steps S301 to S304, considering that the long pauses and phrase pauses in the target Japanese text have little to do with the pronunciation of words and mainly depend on the meaning of the text itself, the prosody prediction of the target Japanese text is closer to an NLP (Natural Language Processing) task, and the level of long pauses in Japanese grammar is higher than that of phrase pauses, that is, long pauses must be phrase pauses, but phrase pauses are not necessarily long pauses. Therefore, a two-layer cascaded prosody prediction network is used as a downstream task of the pre-trained text processing network to predict long pauses and phrase pauses layer by layer. Among them, the first layer of the prosody prediction network includes a first feature extraction subnetwork and a long pause prediction subnetwork, which are used to extract word-granularity long pause extraction features and predict word-granularity target long pause features; the second layer of the prosody prediction network includes a second feature extraction subnetwork and a phrase pause prediction subnetwork, which are used to extract word-granularity phrase pause extraction features and predict word-granularity target phrase pause features. Extracting word-granular long pause features first can provide more valuable contextual information for the prediction of target phrase pause features, help the network better utilize the hierarchical information of long pauses and phrase pauses, and improve the accuracy of target phrase pause feature prediction.

[0052] In one embodiment, the initial phoneme features of the target Japanese text and the polyphonetic word prediction results of the target Japanese text are first obtained, and then the initial phoneme features of the target Japanese text are updated based on the polyphonetic word prediction results of the target Japanese text to obtain the target phoneme features of the target Japanese text.

[0053] Figure 4 1 is a flow chart of an embodiment of obtaining target phoneme features provided by the present application. Figure 4 As shown, the method comprises the following steps:

[0054] S401: Acquire initial phoneme features of the target Japanese text.

[0055] Specifically, step S401 includes: performing word segmentation processing on the target Japanese text to obtain a number of divided Japanese words; determining the initial phoneme features of the target Japanese text. The initial phoneme features of the target Japanese text include initial phoneme sub-features of each Japanese word determined based on a Japanese dictionary (i.e., initial phoneme sequences of each Japanese word). After the target Japanese text is segmented, the initial phoneme sub-features of each Japanese word can be obtained based on a Japanese dictionary query.

[0056] Exemplarily, a Mecab word segmenter may be used to perform word segmentation processing on the target Japanese text, and the Japanese dictionary may be a unidic dictionary.

[0057] S402: Convert character-level phrase pause extraction features into word-level phrase pause extraction features.

[0058] The character-granularity phrase pause extraction feature includes phrase pause sub-extraction features corresponding to each character of the target Japanese text. Since each Japanese word includes a number of characters, each Japanese word includes a number of phrase pause sub-extraction features corresponding to the corresponding characters. The phrase pause sub-extraction features of each Japanese word can be combined to obtain a phrase pause extraction feature that can characterize each Japanese word, that is, a word-granularity phrase pause extraction feature.

[0059] Specifically, a plurality of phrase pause sub-extraction features corresponding to each Japanese word are obtained; and average pooling processing is performed on the plurality of phrase pause sub-extraction features of each Japanese word to obtain a word-granular phrase pause extraction feature. Among them, for each Japanese word, average pooling processing on the plurality of phrase pause sub-extraction features of the Japanese word means taking the average value of the plurality of phrase pause sub-extraction features of the Japanese word.

[0060] S403: Using a polyphone prediction network to predict the pause extraction features of word-level phrases, and obtaining polyphone prediction results of the target Japanese text.

[0061] The polyphone prediction results include the polyphone categories of each Japanese word. The polyphone prediction results can also be represented by one-hot tags. Polyphone tags with the category plus 1 can be designed according to the supported Japanese polyphone categories. For example, 37 polyphone tags can be designed for 15 words and 36 categories. 0 dimension represents non-polyphones, and the other dimensions correspond to the remaining 36 polyphone tags. Exemplarily, the polyphone prediction network is a fully connected layer.

[0062] In this embodiment, considering that most Japanese polyphones appear at word granularity, Japanese polyphone prediction is performed at word granularity. In addition, since the polyphone prediction task is mainly determined by the text semantic information of the target Japanese text, and the upstream pre-trained text processing network is at word granularity and the word granularity is lower than the phrase pause level, the polyphone prediction in this embodiment is performed after the phrase pause prediction, and the polyphone prediction is performed after converting the word granularity phrase pause extraction features into the word granularity phrase pause extraction features, so that the phrase pause information can be used to predict polyphones, thereby improving the accuracy of the polyphone prediction results.

[0063] S404: Based on the polyphone prediction result, determine polyphone words among the plurality of Japanese words in the target Japanese text.

[0064] After obtaining the polyphone prediction result of the target Japanese text, polyphone words among several Japanese words may be further determined based on the polyphone category of each Japanese word.

[0065] S405: Determine the target phoneme sub-features corresponding to each polyphonic word.

[0066] Specifically, for each polyphonic word, the target phoneme sub-feature of the polyphonic word (ie, the phoneme sequence corresponding to the polyphonic word) can be obtained based on the specific polyphonic word category query of the polyphonic word.

[0067] S406: updating the initial phoneme sub-features of each polyphonic word in the initial phoneme feature to the corresponding target phoneme sub-features to obtain the target phoneme feature.

[0068] In steps S401 to S406, considering that the initial phoneme sub-features of the Japanese words queried based on the Japanese dictionary may be the default pronunciation of the Japanese word, that is, the accuracy of the initial phoneme features of the target Japanese text is not high, therefore, the polyphone prediction results based on the target Japanese text can obtain the accurate pronunciation of the polyphone words in the target Japanese text, and then update the initial phoneme features of the target Japanese text to obtain the target phoneme features, which can improve the accuracy of the obtained target phoneme features.

[0069] In one embodiment, the step of determining the target text feature includes: obtaining any of the following features as the target text feature: character-granularity text feature, character-granularity long pause extraction feature, character-granularity phrase pause extraction feature, and word-granularity phrase pause extraction feature obtained by converting the character-granularity phrase pause extraction feature.

[0070] Since the character-granularity text features, and the character-granularity long pause extraction features, character-granularity phrase pause extraction features, and word-granularity phrase pause extraction features obtained based on the character-granularity text features can all reflect the text semantic information of the target Japanese text, therefore, the character-granularity text features, character-granularity long pause extraction features, character-granularity phrase pause extraction features, and word-granularity phrase pause extraction features can all be used as target text features of the target Japanese text.

[0071] Exemplarily, the word-granularity phrase pause extraction feature obtained by converting the character-granularity phrase pause extraction feature is used as the target text feature.

[0072] S23: Fusing the target phrase pause feature, the target phoneme feature and the target text feature to obtain a target fusion feature.

[0073] Figure 5 1 is a flow chart of an embodiment of obtaining target fusion features provided by the present application. Figure 5 As shown, the method comprises the following steps:

[0074] S501: Convert target phrase pause features, target phoneme features, and target text features into syllable-level features.

[0075] Since the tonal position of Japanese is a feature that controls the syllable stress at the pause boundary of the stressed phrase in Japanese, the determination of the tonal position of Japanese should be based on: locating the stressed phrase boundary; locating the syllable boundary; predicting the tonal position based on the syllable combined with the text. In other words, the tonal position needs to be predicted at the syllable granularity. Therefore, the acquired target phrase pause features, target phoneme features, and target text features need to be converted into features at the syllable granularity.

[0076] In a specific embodiment, the target phrase pause feature is a character-level feature, the target phoneme feature is a phoneme-level feature, and the target text feature is a word-level phrase pause extraction feature, that is, the target text feature is a word-level feature.

[0077] Since there is no direct, one-to-one mapping relationship between the Japanese writing system (including Hiragana, Katakana, Kanji, etc.) and syllables, especially when it comes to Kanji, a Kanji may correspond to multiple syllables, depending on its specific reading in the sentence. Therefore, there is no definite mapping relationship between single characters and syllables in Japanese. The only way to convert the character-granularity features into word-granularity features and then into syllable-granularity features is through the mapping relationship between character-word-syllable, at the cost of losing some character-granularity information.

[0078] Specifically, according to the pre-acquired character-word-syllable mapping relationship, the target phrase pause features at the character granularity may be mapped to the target phrase pause features at the syllable granularity.

[0079] Specifically, according to the pre-acquired word-syllable mapping relationship, the target text features that have been converted into word granularity (ie, word granularity phrase pause extraction features) can be further mapped into syllable granularity target text features.

[0080] For example, a number of syllables corresponding to each Japanese word can be determined based on the pre-acquired word-syllable mapping relationship; for each Japanese word, the phrase pause extraction feature of the Japanese word is averaged and de-pooled to obtain the phrase pause extraction features corresponding to each syllable of the Japanese word. The average de-pooling of the phrase pause extraction features of the Japanese word refers to copying the phrase pause extraction features of the Japanese word to each syllable of the Japanese word, that is, the phrase pause extraction features of each syllable of the Japanese word are the same as the phrase pause extraction features of the corresponding Japanese word.

[0081] Specifically, the target phoneme feature at the phoneme granularity may be mapped to the target phoneme feature at the syllable granularity according to the pre-acquired phoneme-syllable mapping relationship.

[0082] S502: Using the third feature extraction subnetwork of the tone nucleus position prediction network, extract the pause features of the target phrase at the syllable granularity and the target phoneme features at the syllable granularity to obtain the first extracted features.

[0083] The third feature extraction subnetwork includes a first embedding subnetwork, a second embedding subnetwork and a recurrent neural subnetwork. Specifically, step S502 includes the following sub-steps:

[0084] Sub-step 1: Process the pause features of the target phrase at the syllable granularity using the first embedding sub-network to obtain the first embedding features at the syllable granularity.

[0085] Sub-step two: using the second embedding sub-network to encode the target phoneme features at the syllable granularity to obtain the second embedding features at the syllable granularity.

[0086] Among them, the first embedding sub-network and the second embedding sub-network are both embedding sub-networks.

[0087] Exemplarily, the first embedding subnetwork and the second embedding subnetwork both have 256 nodes. The first embedding subnetwork is used to map the features of each syllable in the target phrase pause features at the syllable granularity into 256-dimensional features, and the second embedding subnetwork is used to map the features of each syllable in the target phoneme features at the syllable granularity into 256-dimensional features, so as to better capture the semantic and contextual relationships between the features of each syllable in the subsequent process.

[0088] Sub-step three: concatenate the first embedding feature at the syllable granularity and the second embedding feature at the syllable granularity.

[0089] Specifically, the first embedding feature of the syllable granularity and the second embedding feature of the syllable granularity are concatenated by syllable to obtain a concatenated embedding feature.

[0090] In sub-step four, a recurrent neural network is used to extract features from the concatenated embedded features to obtain the first extracted features at the syllable granularity.

[0091] Exemplarily, the recurrent neural sub-network is a bidirectional LSTM, which is used to extract context information on the syllable sequence from the concatenated embedded features, i.e., the first extracted features. The first extracted features obtained in this sub-step include a deep comprehensive representation of phrase pauses and phoneme information.

[0092] S503: performing feature concatenation on the target text feature at the syllable granularity and the first extracted feature at the syllable granularity to obtain a target fusion feature.

[0093] Specifically, the target text feature of the syllable granularity and the first extracted feature of the syllable granularity are concatenated by syllable to obtain the target fusion feature.

[0094] S24: Utilize the target fusion feature to predict the position of the tone core, and obtain the prediction result of the tone core position corresponding to the target Japanese text.

[0095] In one embodiment, step 24 includes: using the fourth feature extraction subnetwork of the tuning core position prediction network to extract the target fusion feature to obtain a second extracted feature; using the tuning core position prediction subnetwork of the tuning core position prediction network to predict the second extracted feature to obtain a tuning core position prediction result.

[0096] Among them, the second extracted features extracted by the fourth feature extraction sub-network include a deeper comprehensive representation corresponding to phrase pauses, phonemes and text semantic information, which is beneficial to the subsequent prediction of the key position.

[0097] The prediction result of the tone core position is also at the syllable granularity, specifically including the third classification result corresponding to each syllable, and the third classification result of each syllable includes the probability of each syllable being the tone core position and the probability of not being the tone core position. Furthermore, the third classification result of each syllable can also be represented by a one-hot label.

[0098] Exemplarily, the fourth feature extraction subnetwork includes a preset number of convolutional layers. For example, the preset number may be 2.

[0099] Exemplarily, the tuning core position prediction subnetwork is an LSTM autoregressive prediction subnetwork, and the LSTM autoregressive prediction subnetwork includes a bidirectional LSTM and a fully connected layer.

[0100] Furthermore, after obtaining the prediction result of the intonation core position of the target Japanese text, the prediction result of the intonation core position is input into the back-end speech synthesis model to assist the back-end speech synthesis.

[0101] Furthermore, in addition to inputting the acquired tone nucleus position prediction results into the back-end speech synthesis model, the aforementioned acquired target long pause features, target phrase pause features and polyphonetic word prediction results can also be input into the back-end speech synthesis model to assist the back-end speech synthesis.

[0102] In the related technology, the prediction of the nucleus position is performed based on the CRF algorithm. For longer and more complex target Japanese texts, the accuracy of the nucleus position predicted by this method will drop significantly, and the generalization ability is relatively general, which depends on a relatively complete data reserve.

[0103] In this embodiment, a pre-trained text processing network is used to obtain the character-granularity text features corresponding to the target Japanese text, and the target phrase pause features, target phoneme features and target text features used for the prediction of the tone core position are determined based on the character-granularity text features. Since the character-granularity text features obtained by the pre-trained text processing network contain rich contextual semantic information, the accuracy of the target phrase pause features, target phoneme features and target text features can be improved with certainty, thereby further improving the accuracy of the prediction of the tone core position. In addition, this embodiment integrates multi-task predictions, and can jointly predict the tone core position, rhythm (including long pauses and phrase pauses) and polyphonic word features required for the back-end speech synthesis, which can greatly improve the prediction performance of various front-end features, and the performance on complex long sentences is more prominent, with the advantages of high prediction accuracy and low deployment cost.

[0104] See also Figure 6 , Figure 6 It is a schematic diagram of the structure of the target prediction model provided by this application. Figure 6 As shown, the target prediction model includes a pre-trained text processing network, a rhythm prediction network, a polyphone prediction network, and a tone nucleus position prediction network.

[0105] Among them, the pre-trained text processing network is used to obtain the word-granular text features of the target Japanese text.

[0106] The prosody prediction network is used to extract character-level long pause extraction features and character-level phrase pause extraction features based on character-level text features, and predict character-level target long pause features and character-level target phrase pause features. The prosody prediction network specifically includes a first feature extraction subnetwork, a second feature extraction subnetwork, a long pause prediction subnetwork, and a phrase pause prediction subnetwork. The functions of each subnetwork of the prosody prediction network can be referred to Figure 3 The illustrated embodiments will not be described in detail here.

[0107] The polyphone prediction network is used to obtain the polyphone prediction results of the target Japanese text based on word-granular phrase pause extraction features.

[0108] The tone core position prediction network is used to predict the tone core position based on the pause feature of the target phrase, the target phoneme feature obtained according to the prediction result of the polyphone word, and the target text feature, and obtain the tone core position prediction result of the target Japanese text. The tone core position prediction network specifically includes a third feature extraction subnetwork, a fourth feature extraction subnetwork, and a tone core position prediction subnetwork. The third feature extraction subnetwork further includes a first embedding subnetwork, a second embedding subnetwork, and a recurrent neural subnetwork ( Figure 6 The functions of each sub-network of the core position prediction network can be referred to the related contents in the aforementioned steps S23 to S24, which will not be described in detail here.

[0109] In this embodiment, before applying the target prediction model, a sample data set is first obtained, and then the target prediction model is trained using the sample data set. The sample data set includes a number of sample Japanese texts, and each sample Japanese text is annotated with word segmentation information, phoneme information, sample long pause annotation features, sample phrase pause annotation features, sample polyphone annotation results, and sample tone position annotation results.

[0110] To ensure the reliability of the sample data set, before using the sample data set to train the target prediction model, the sample Japanese text in the sample data set may also be preprocessed, for example, written text regularization, processing of abnormal text symbols, etc.

[0111] In this embodiment, the target prediction model can be first trained, and the first training is used to train the tone core position prediction network in the target prediction model using the first sample Japanese text. The first training steps include: obtaining sample phrase pause features, sample phoneme features, and sample text features of the first sample Japanese text; using the tone core position prediction network to obtain a sample tone core position prediction result of the first sample Japanese text based on the sample phrase pause features, sample phoneme features, and sample text features of the first sample Japanese text; and adjusting the network parameters of the tone core position prediction network based on the sample tone core position prediction result of the first sample Japanese text.

[0112] Exemplarily, the sample phrase pause feature of the first sample Japanese text is the sample phrase pause annotation feature of the first sample Japanese text. The sample phoneme feature of the first sample Japanese text may be the sample phoneme annotation feature of the first sample Japanese text, or the sample phoneme feature of the first sample Japanese text may be a feature determined based on the sample polyphone annotation result of the first sample Japanese text. The sample text feature of the first sample Japanese text may be obtained based on the sample word granularity text feature of the first sample Japanese text, and the sample word granularity text feature of the first sample Japanese text may be obtained by processing the first sample Japanese text using a pre-trained text processing network. For example, the sample text feature of the first sample Japanese text is the sample word granularity text feature, or it may be the sample word granularity long pause extraction feature, the sample word granularity phrase pause extraction feature, or the word granularity phrase pause extraction feature obtained based on the sample word granularity text feature.

[0113] Specifically, during the first training process, a first loss can be calculated based on the difference between the first sample tuning core position prediction result of the first sample Japanese text and the sample tuning core position annotation result of the first sample Japanese text; and the network parameters of the tuning core position prediction network can be adjusted based on the first loss.

[0114] Before the target prediction model is first trained, the target prediction model is first trained for the second time, and the second training is used to train the pre-trained text processing network and the prosody prediction network in the target prediction model using the second sample Japanese text. The steps of the second training include: using the pre-trained text processing network to obtain the word-granularity text features of the second sample Japanese text; using the prosody prediction network to predict the sample long pause features and sample phrase pause features of the second sample Japanese text based on the word-granularity text features of the second sample Japanese text; and adjusting the network parameters of the pre-trained text processing network and the prosody prediction network based on the sample long pause features and sample phrase pause features of the second sample Japanese text.

[0115] Specifically, a second loss is calculated based on the difference between the sample long pause feature of the second sample Japanese text and the sample long pause annotation feature of the second sample Japanese text, and a third loss is calculated based on the difference between the sample phrase pause feature of the second sample Japanese text and the sample phrase pause annotation feature of the second sample Japanese text; based on the second loss and the third loss, network parameters of the pre-trained text processing network and the rhythm prediction network are adjusted.

[0116] After the target prediction model is subjected to the second training and the first training, the target prediction model can be further subjected to the third training, in which the sample polyphonic word annotation results of the third sample Japanese text are loaded to participate in the downstream tone nucleus position prediction. The steps of the third training include: using the pre-trained text processing network to obtain the word-granularity text features of the third sample Japanese text; using the rhythm prediction network to predict the sample long pause features and sample phrase pause features of the third sample Japanese text based on the word-granularity text features of the third sample Japanese text; determining the sample text features of the third sample Japanese text based on the word-granularity text features of the third sample Japanese text; determining the sample phoneme features of the third sample Japanese text based on the sample polyphonic word annotation results of the third sample Japanese text; using the tone nucleus position prediction network to obtain the sample tone nucleus position prediction results of the third sample Japanese text based on the sample phrase pause features, sample phoneme features and sample text features of the third sample Japanese text; adjusting the network parameters of the pre-trained text processing network, the rhythm prediction network and the tone nucleus position prediction network based on the sample long pause features, sample phrase pause features and sample tone nucleus position prediction results of the third sample Japanese text.

[0117] Specifically, based on the difference between the sample long pause feature of the third sample Japanese text and the sample long pause annotation feature of the third sample Japanese text, the difference between the sample phrase pause feature of the third sample Japanese text and the sample phrase pause annotation feature of the third sample Japanese text, and the difference between the sample tone core position prediction result of the third sample Japanese text and the sample tone core position annotation result of the third sample Japanese text, the fourth loss, the fifth loss and the sixth loss are calculated respectively; based on the sum of the losses corresponding to the fourth loss, the fifth loss and the sixth loss, the network parameters of the pre-trained text processing network, the rhythm prediction network and the tone core position prediction network are adjusted.

[0118] After the target prediction model is trained for the third time, the target prediction model may be further trained for the fourth time, in which a polyphone prediction task is added, but the sample polyphone annotation results of the fourth sample Japanese text are loaded to participate in the downstream nucleus position prediction. The steps of the fourth training include: using a pre-trained text processing network to obtain the word-granularity text features of the fourth sample Japanese text; using a rhythm prediction network to obtain the sample word-granularity long pause extraction features and the sample word-granularity phrase pause extraction features of the fourth sample Japanese text based on the word-granularity text features of the fourth sample Japanese text, and predicting the sample long pause features and the sample phrase pause features of the fourth sample Japanese text based on the sample word-granularity long pause extraction features and the sample word-granularity phrase pause extraction features of the fourth sample Japanese text respectively; determining the sample text features of the fourth sample Japanese text based on the word-granularity text features of the fourth sample Japanese text; determining the sample word-granularity phrase pause extraction features of the fourth sample Japanese text based on the sample word-granularity phrase pause extraction features of the fourth sample Japanese text; Extracting sample word-granularity phrase pause features of the sample Japanese text; obtaining sample polyphone prediction results of the fourth sample Japanese text based on the sample word-granularity phrase pause extraction features of the fourth sample Japanese text using a polyphone prediction network; determining sample phoneme features of the fourth sample Japanese text based on the sample polyphone annotation results of the fourth sample Japanese text; obtaining sample tone nucleus position prediction results of the fourth sample Japanese text based on sample phrase pause features, sample phoneme features and sample text features of the fourth sample Japanese text using a tone nucleus position prediction network; adjusting the overall network parameters of the target prediction model based on the sample long pause features, sample phrase pause features, sample polyphone prediction results and tone nucleus position prediction results of the fourth sample Japanese text.

[0119] Specifically, based on the difference between the sample long pause feature of the fourth sample Japanese text and the sample long pause annotation feature of the fourth sample Japanese text, the difference between the sample phrase pause feature of the fourth sample Japanese text and the sample phrase pause annotation feature of the fourth sample Japanese text, the difference between the sample polyphone prediction result of the fourth sample Japanese text and the sample polyphone annotation result of the fourth sample Japanese text, and the difference between the sample tone core position prediction result of the fourth sample Japanese text and the sample tone core position annotation result of the fourth sample Japanese text, the seventh loss, eighth loss, ninth loss and tenth loss are calculated respectively; based on the sum of losses corresponding to the seventh loss, the eighth loss, the ninth loss and the tenth loss, the overall network parameters of the target prediction model are adjusted.

[0120] After the target prediction model is trained for the fourth time, the target prediction model may be further trained for the fifth time, and any annotation information of the fifth sample Japanese text is not loaded in the fifth training to participate in the downstream nucleus position prediction. The steps of the fifth training include: using a pre-trained text processing network to obtain the word-granularity text features of the fifth sample Japanese text; using a rhythm prediction network to obtain the sample word-granularity long pause extraction features and sample word-granularity phrase pause extraction features of the fifth sample Japanese text based on the word-granularity text features of the fifth sample Japanese text, and predicting the sample long pause features and sample phrase pause features of the fifth sample Japanese text based on the sample word-granularity long pause extraction features and sample word-granularity phrase pause extraction features of the fifth sample Japanese text respectively; determining the sample text features of the fifth sample Japanese text based on the word-granularity text features of the fifth sample Japanese text; determining the sample word-granularity phrase pause extraction features of the fifth sample Japanese text based on the sample word-granularity phrase pause extraction features of the fifth sample Japanese text. Extracting features of sample word granularity phrase pauses of Japanese text; obtaining sample polyphone prediction results of the fifth sample Japanese text based on the sample word granularity phrase pause extraction features of the fifth sample Japanese text by using a polyphone prediction network; determining sample phoneme features of the fifth sample Japanese text based on the sample polyphone prediction results of the fifth sample Japanese text; obtaining sample tone nucleus position prediction results of the fifth sample Japanese text based on sample long pause features, sample phrase pause features and sample phoneme features of the fifth sample Japanese text by using a tone nucleus position prediction network; adjusting the overall network parameters of the target prediction model based on the sample long pause features, sample phrase pause features, sample polyphone prediction results and sample tone nucleus position prediction results of the fifth sample Japanese text.

[0121] Specifically, based on the difference between the sample long pause feature of the fifth sample Japanese text and the sample long pause annotation feature of the fifth sample Japanese text, the difference between the sample phrase pause feature of the fifth sample Japanese text and the sample phrase pause annotation feature of the fifth sample Japanese text, the difference between the sample polyphone prediction result of the fifth sample Japanese text and the sample polyphone annotation result of the fifth sample Japanese text, and the difference between the sample tone core position prediction result of the fifth sample Japanese text and the sample tone core position annotation result of the fifth sample Japanese text, the eleventh loss, the twelfth loss, the thirteenth loss and the fourteenth loss are calculated respectively; based on the sum of the losses corresponding to the eleventh loss, the twelfth loss, the thirteenth loss and the fourteenth loss, the overall network parameters of the target prediction model are adjusted.

[0122] Exemplarily, the first to fourteenth losses are all cross entropy losses. The first sample Japanese text, the second sample Japanese text, the third sample Japanese text, the fourth sample Japanese text, and the fifth sample Japanese text may be the same sample Japanese text or different sample Japanese texts.

[0123] In this embodiment, a multi-task teacher-force training strategy is used to gradually train the target prediction model, and the target prediction model is obtained through the second training, the first training, the third training, the fourth training and the fifth training in sequence. Through this training method, the target prediction model can simultaneously have the ability to predict rhythm, polyphonic words and tone nucleus position, and can reduce error propagation in the training process and accelerate the training process, thereby improving the prediction performance of the target prediction model.

[0124] See also Figure 7 , Figure 7 1 is a schematic diagram of a framework of an embodiment of a modulation core position prediction device provided by the present application. In this embodiment, the modulation core position prediction device 70 includes: an acquisition module 71 , a fusion module 72 and a prediction module 73 .

[0125] Among them, the acquisition module 71 is used to obtain the target phrase pause features, target phoneme features and target text features corresponding to the target Japanese text; the fusion module 72 is used to fuse the target phrase pause features, target phoneme features and target text features to obtain the target fusion features; the prediction module 73 is used to use the target fusion features to predict the tone nucleus position to obtain the tone nucleus position prediction result corresponding to the target Japanese text.

[0126] Optionally, the acquisition module 71 is used to process the target Japanese text using a pre-trained text processing network to obtain word-granularity text features corresponding to the target Japanese text; based on the word-granularity text features, determine the target phrase pause features, target phoneme features and target text features.

[0127] Optionally, the step of determining the target phrase pause feature is performed by a prosody prediction network, which includes a first feature extraction subnetwork, a second feature extraction subnetwork, and a phrase pause prediction subnetwork. The acquisition module 71 is used to extract the character-granularity text feature using the first feature extraction subnetwork to obtain the character-granularity long pause extraction feature; to extract the character-granularity long pause extraction feature using the second feature extraction subnetwork to obtain the character-granularity phrase pause extraction feature; and to predict the character-granularity phrase pause extraction feature using the phrase pause prediction subnetwork to obtain the character-granularity target phrase pause feature.

[0128] Optionally, the prosody prediction network also includes a long pause prediction subnetwork, and the acquisition module 71 is also used to use the long pause prediction subnetwork to predict the word-granularity long pause extraction features to obtain the word-granularity target long pause features; wherein, the tone nucleus position prediction results, the target long pause features and the target phrase pause features are all used to input the back-end speech synthesis model.

[0129] Optionally, before determining the target phoneme feature, the acquisition module 71 is also used to perform word segmentation processing on the target Japanese text to obtain a number of divided Japanese words; determine the initial phoneme feature of the target Japanese text; wherein the initial phoneme feature includes the initial phoneme sub-feature of each Japanese word determined based on the Japanese dictionary. The acquisition module 71 is used to convert the character-granularity phrase pause extraction feature into the word-granularity phrase pause extraction feature; use the polyphone prediction network to predict the word-granularity phrase pause extraction feature to obtain the polyphone prediction result of the target Japanese text; wherein the polyphone prediction result includes the polyphone category of each Japanese word; based on the polyphone prediction result, determine the polyphone words in a number of Japanese words; respectively determine the target phoneme sub-features corresponding to each polyphone word; respectively update the initial phoneme sub-features of each polyphone word in the initial phoneme feature to the corresponding target phoneme sub-features to obtain the target phoneme feature.

[0130] Optionally, the character-granularity phrase pause extraction feature includes phrase pause sub-extraction features corresponding to each character of the target Japanese text, and the acquisition module 71 is used to obtain several phrase pause sub-extraction features corresponding to each Japanese word; perform average pooling processing on several phrase pause sub-extraction features of each Japanese word to obtain word-granularity phrase pause extraction features; and / or, the polyphonetic word prediction result is used to input the back-end speech synthesis model.

[0131] Optionally, the acquisition module 71 is used to acquire any of the following features as target text features: character-granularity text features, character-granularity long pause extraction features, character-granularity phrase pause extraction features, and word-granularity phrase pause extraction features obtained by converting character-granularity phrase pause extraction features.

[0132] Optionally, the fusion module 72 is used to convert the target phrase pause features, target phoneme features and target text features into syllable-granularity features; use the third feature extraction subnetwork of the tone nucleus position prediction network to extract the syllable-granularity target phrase pause features and the syllable-granularity target phoneme features to obtain a first extracted feature; and perform feature concatenation on the syllable-granularity target text features and the first extracted features to obtain a target fusion feature.

[0133] Optionally, the third feature extraction subnetwork includes a first embedding subnetwork, a second embedding subnetwork and a recurrent neural subnetwork. The fusion module 72 is used to process the target phrase pause feature at the syllable granularity using the first embedding subnetwork to obtain the first embedding feature at the syllable granularity; encode the target phoneme feature at the syllable granularity using the second embedding subnetwork to obtain the second embedding feature at the syllable granularity; perform feature splicing on the first embedding feature and the second embedding feature; and perform feature extraction on the spliced ​​embedded feature using the recurrent neural subnetwork to obtain the first extracted feature.

[0134] Optionally, the prediction module 73 is used to use the fourth feature extraction subnetwork of the tuning core position prediction network to extract the target fusion feature to obtain the second extracted feature; and use the tuning core position prediction subnetwork of the tuning core position prediction network to predict the second extracted feature to obtain the tuning core position prediction result.

[0135] Optionally, the target prediction model includes a pre-trained text processing network, a rhythm prediction network, a polyphone prediction network and a tone core position prediction network. The tone core position prediction device 70 also includes a training module 74. The training module 74 is used to perform first training, second training, third training, fourth training and fifth training on the target prediction model in sequence. Among them, the first training step includes: using the pre-trained text processing network and the rhythm prediction network to obtain the first sample long pause prediction feature and the first sample phrase pause prediction feature of the first sample Japanese text; based on the first sample long pause prediction feature and the first sample phrase pause prediction feature, adjusting the network parameters of the pre-trained text processing network and the rhythm prediction network; the second training step includes: determining the first sample phoneme feature of the second sample Japanese text based on the sample polyphone annotation result of the second sample Japanese text; using the tone core position prediction network to obtain the first sample long pause annotation feature, the sample phrase pause annotation feature and the first sample phoneme feature of the second sample Japanese text; The method comprises the following steps: obtaining a first sample tone core position prediction result of a second sample Japanese text by using a pre-trained text processing network and a rhythm prediction network; adjusting a network parameter of a tone core position prediction network based on the first sample tone core position prediction result; the third training step comprises: obtaining a second sample long pause prediction feature and a second sample phrase pause prediction feature of a third sample Japanese text by using a pre-trained text processing network and a rhythm prediction network; determining a second sample phoneme feature of the third sample Japanese text based on a sample polyphone tagging result of the third sample Japanese text; obtaining a second sample long pause prediction feature, a second sample phrase pause prediction feature, and a second sample phoneme feature of the third sample Japanese text by using a tone core position prediction network. The sample tone core position prediction result; based on the second sample long pause prediction feature, the second sample phrase pause prediction feature and the second sample tone core position prediction result, adjusting the network parameters of the pre-trained text processing network, the rhythm prediction network and the tone core position prediction network; the fourth training step includes: using the pre-trained text processing network and the rhythm prediction network to obtain the third sample long pause prediction feature and the third sample phrase pause prediction feature of the fourth sample Japanese text; using the polyphonic word prediction network to obtain the first sample polyphonic word prediction result of the fourth sample Japanese text based on the third sample phrase pause prediction feature; determining the first polyphonic word annotation result of the fourth sample Japanese text based on the sample polyphonic word annotation result of the fourth sample Japanese text The third sample phoneme feature of the fourth sample Japanese text; using the tone core position prediction network to obtain the third sample tone core position prediction result of the fourth sample Japanese text based on the third sample long pause prediction feature, the third sample phrase pause prediction feature and the third sample phoneme feature; adjusting the overall network parameters of the target prediction model based on the third sample long pause prediction feature, the third sample phrase pause prediction feature, the first sample polyphone prediction result and the third sample tone core position prediction result; the fifth training step includes: using the pre-trained text processing network and the rhythm prediction network to obtain the fourth sample long pause prediction feature and the fourth sample phrase pause prediction feature of the fifth sample Japanese text;Using a polyphone prediction network to obtain a second sample polyphone prediction result of a fifth sample Japanese text based on a fourth sample phrase pause prediction feature; determining a fourth sample phoneme feature of the fifth sample Japanese text based on the second sample polyphone prediction result; using a tone core position prediction network to obtain a fourth sample tone core position prediction result of the fifth sample Japanese text based on a fourth sample long pause prediction feature, a fourth sample phrase pause prediction feature, and a fourth sample phoneme feature; adjusting the overall network parameters of the target prediction model based on the fourth sample long pause prediction feature, the fourth sample phrase pause prediction feature, the second sample polyphone prediction result, and the fourth sample tone core position prediction result. ;

[0136] It should be noted that the device of this embodiment can execute the steps in the above method. For detailed description of the relevant content, please refer to the above method part, which will not be repeated here.

[0137] See also Figure 8 , Figure 8 8 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 80 includes a memory 81 and a processor 82 .

[0138] The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. A general-purpose processor may be a microprocessor or the processor 82 may also be any conventional processor 82, etc.

[0139] The memory 81 in the electronic device 80 is used to store program instructions required for the processor 82 to run.

[0140] The processor 82 is used to execute program instructions to implement the method for predicting the core position in the present application.

[0141] See also Fig. 9 , Fig. 9It is a framework diagram of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 90 of the embodiment of the present application stores program instructions 91, and when the program instructions 91 are executed, the method for predicting the adjustment core position provided by the present application is implemented. Among them, the program instructions 91 can form a program file and be stored in the above-mentioned computer-readable storage medium 90 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a sub-network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 90 includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0142] The above scheme obtains the target fusion feature by fusing the target phrase pause feature, target phoneme feature and target text feature corresponding to the target Japanese text, and uses the target fusion feature to predict the tone nucleus position, thereby obtaining the tone nucleus position prediction result corresponding to the target Japanese text. Since the text and pronunciation within the boundary of the Japanese tone nucleus position and the phrase pause are related, and the target phrase pause feature, target phoneme feature and target text feature can respectively reflect the phrase pause information, pronunciation information and text semantic information of the target Japanese text, therefore, using the target fusion feature to predict the tone nucleus position can integrate the information of the three different dimensions of phrase pause, pronunciation and text semantics of the target Japanese text, so that the predicted tone nucleus position prediction result has a higher accuracy.

[0143] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0144] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple sub-network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a sub-network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0149] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for predicting a tuning core position, characterized in that: The method comprises: Obtaining target phrase pause features, target phoneme features, and target text features corresponding to the target Japanese text; Fusing the target phrase pause feature, the target phoneme feature, and the target text feature to obtain a target fusion feature; The target fusion feature is used to predict the position of the tone core, and a prediction result of the tone core position corresponding to the target Japanese text is obtained.

2. The method according to claim 1, characterized in that The step of obtaining target phrase pause features, target phoneme features, and target text features corresponding to the target Japanese text includes: Processing the target Japanese text using a pre-trained text processing network to obtain word-granularity text features corresponding to the target Japanese text; Based on the word-granularity text feature, the target phrase pause feature, the target phoneme feature and the target text feature are determined.

3. The method according to claim 2, characterized in that The step of determining the pause feature of the target phrase is performed by a prosody prediction network, wherein the prosody prediction network includes a first feature extraction subnetwork, a second feature extraction subnetwork, and a phrase pause prediction subnetwork; The step of determining the pause feature of the target phrase comprises: Using the first feature extraction sub-network to extract the character-granularity text features, to obtain character-granularity long pause extraction features; Using the second feature extraction sub-network to extract the word-granularity long pause extraction feature, to obtain the word-granularity phrase pause extraction feature; The phrase pause prediction subnetwork is used to predict the character-scale phrase pause extraction features to obtain the character-scale target phrase pause features.

4. The method according to claim 3, characterized in that: The prosody prediction network also includes a long pause prediction subnetwork, and the method further includes: Using the long pause prediction subnetwork to predict the word-sized long pause extraction features, to obtain the word-sized target long pause features; The tone core position prediction result, the target long pause feature and the target phrase pause feature are all used to input into a backend speech synthesis model.

5. The method according to claim 3, characterized in that: Before determining the target phoneme feature, the method further includes: Performing word segmentation processing on the target Japanese text to obtain a plurality of segmented Japanese words; Determining initial phoneme features of the target Japanese text; wherein the initial phoneme features include initial phoneme sub-features of each of the Japanese words determined based on a Japanese dictionary; The step of determining the target phoneme feature comprises: Converting the character-granularity phrase pause extraction feature into a word-granularity phrase pause extraction feature; Using a polyphone prediction network to predict the word-granular phrase pause extraction features, to obtain a polyphone prediction result of the target Japanese text; wherein the polyphone prediction result includes the polyphone category of each of the Japanese words; Based on the polyphonic word prediction result, determining the polyphonic words among the plurality of Japanese words; Determining the target phoneme sub-features corresponding to each of the polyphonic words respectively; The initial phoneme sub-features of each of the polyphonic words in the initial phoneme feature are respectively updated to the corresponding target phoneme sub-features to obtain the target phoneme feature.

6. The method according to claim 5, characterized in that The character-level phrase pause extraction feature includes phrase pause sub-extraction features corresponding to each character of the target Japanese text, and the step of converting the character-level phrase pause extraction feature into a word-level phrase pause extraction feature includes: Obtaining a plurality of the phrase pause sub-extraction features corresponding to each of the Japanese words; Performing average pooling processing on the plurality of phrase pause sub-extraction features of each of the Japanese words respectively to obtain the word-granularity phrase pause extraction feature; And / or, the polyphonetic word prediction result is used to input into a backend speech synthesis model.

7. The method according to claim 3, characterized in that The step of determining the target text features comprises: Any one of the following features is obtained as the target text feature: the character-granularity text feature, the character-granularity long pause extraction feature, the character-granularity phrase pause extraction feature, and the word-granularity phrase pause extraction feature obtained by converting the character-granularity phrase pause extraction feature.

8. The method according to claim 1, characterized in that The step of fusing the target phrase pause feature, the target phoneme feature and the target text feature to obtain a target fusion feature includes: Converting the target phrase pause feature, the target phoneme feature, and the target text feature into features at the syllable granularity; Using a third feature extraction subnetwork of the tone nucleus position prediction network, feature extraction is performed on the target phrase pause feature at the syllable granularity and the target phoneme feature at the syllable granularity to obtain a first extracted feature; The target text feature at the syllable granularity and the first extracted feature are concatenated to obtain the target fusion feature.

9. The method according to claim 8, characterized in that The third feature extraction subnetwork includes a first embedding subnetwork, a second embedding subnetwork and a recurrent neural subnetwork; the third feature extraction subnetwork is used to extract the target phrase pause feature at the syllable granularity and the target phoneme feature at the syllable granularity to obtain the first extracted feature, including: Using the first embedding sub-network to process the pause feature of the target phrase at a syllable granularity, to obtain a first embedding feature at a syllable granularity; Encoding the target phoneme feature at the syllable granularity by using the second embedding subnetwork to obtain a second embedding feature at the syllable granularity; Performing feature splicing on the first embedded feature and the second embedded feature; The recurrent neural sub-network is used to perform feature extraction on the concatenated embedded features to obtain the first extracted features.

10. The method according to claim 1, characterized in that The step of using the target fusion feature to predict the position of the tone core to obtain the prediction result of the tone core position corresponding to the target Japanese text includes: Using the fourth feature extraction subnetwork of the tuning core position prediction network to extract features from the target fusion features to obtain second extracted features; The second extracted feature is predicted using the tuning core position prediction subnetwork of the tuning core position prediction network to obtain the tuning core position prediction result.

11. The method according to claim 1, characterized in that: The steps of obtaining the target fusion feature and using the target fusion feature to predict the tuning core position are both performed by the tuning core position prediction network. The training steps of the tuning core position prediction network include: Obtaining a sample phrase pause feature, a sample phoneme feature, and a sample text feature of a first sample Japanese text; Using the tone core position prediction network, based on the pause feature of the sample phrase, the sample phoneme feature and the sample text feature, to obtain a sample tone core position prediction result of the first sample Japanese text; Based on the sample core position prediction result, the network parameters of the core position prediction network are adjusted.

12. A device for predicting the position of a tuning core, characterized in that: The device comprises: An acquisition module, used for acquiring target phrase pause features, target phoneme features and target text features corresponding to the target Japanese text; A fusion module, used for fusing the target phrase pause feature, the target phoneme feature and the target text feature to obtain a target fusion feature; The prediction module is used to predict the position of the key point using the target fusion feature to obtain the prediction result of the position of the key point corresponding to the target Japanese text.

13. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed by a processor to implement the method according to any one of claims 1 to 11.

Images