A time length determination method, apparatus, device, and storage medium

Abstract

A time length determination method, device, equipment and storage medium are disclosed. The method comprises: obtaining a text sample and the time length of each phoneme in the text sample; performing synonym replacement on at least one word sample in the text sample to obtain a target text; and determining the time length of each phoneme in the target text according to the text sample, the time length of each phoneme in the text sample and the target text. According to the technical scheme of the present application, the text-time length data can be expanded for the time length prediction model of speech synthesis, the scale of the text-time length data is greatly increased, and the generalization ability of the time length prediction model is enhanced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech synthesis technology, and in particular to a method, apparatus, device and storage medium for determining duration. Background Technology

[0002] In text-to-speech (TTS) synthesis, popular acoustic modeling solutions (such as DurIAN and FastSpeech) require duration prediction models (which take a sequence of phonemes as input and predict the duration of each phoneme). These duration models also require training data based on text-duration information. Typically, a high-quality TTS library contains around 10,000 text-duration data entries, or even fewer. Compared to the arbitrary text input used in later TTS synthesis, this amount of text-duration data is extremely small, resulting in insufficient generalization ability for the trained duration prediction models. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and storage medium for determining duration, thereby enhancing the generalization ability of duration prediction models.

[0004] According to one aspect of the present invention, a method for determining duration is provided, comprising:

[0005] Obtain a text sample and the duration of each phoneme in the text sample;

[0006] At least one word sample in the text sample is replaced with a synonym to obtain the target text;

[0007] The duration of each phoneme in the target text is determined based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0008] According to another aspect of the present invention, a duration determination device is provided, the device comprising:

[0009] The acquisition module is used to acquire text samples and the duration of each phoneme in the text samples;

[0010] The synonym replacement module is used to replace at least one word sample in the text sample with a synonym to obtain the target text.

[0011] The determination module is used to determine the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the duration determination method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the duration determination method according to any embodiment of the present invention.

[0017] This invention provides an embodiment that obtains a text sample and the duration of each phoneme in the text sample, performs synonym replacement on at least one word sample in the text sample to obtain the target text, and determines the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text. Through the technical solution of this invention, the duration prediction model for speech synthesis can be expanded by increasing the text-duration data, significantly increasing the scale of the text-duration data, thereby enhancing the generalization ability of the duration prediction model.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a duration determination method according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the structure of a duration determination device according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the duration determination method in the embodiments of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Example 1

[0026] Figure 1 This is a flowchart of a duration determination method according to an embodiment of the present invention. This embodiment is applicable to situations where duration is determined. The method can be executed by the duration determination device in this embodiment of the present invention, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0027] S110. Obtain the text sample and the duration of each phoneme in the text sample.

[0028] It should be noted that the text samples can be text data samples from the original speech library. For example, the original speech library can be a high-quality speech synthesis library, which includes approximately 10,000 text-duration data entries.

[0029] It is known that a phoneme is the smallest unit of speech that is defined based on the natural properties of speech.

[0030] The duration is measured in audio frames (the exact number of milliseconds represented by one frame can be preset, but this embodiment does not limit this; it is generally related to the frame shift information in audio processing).

[0031] Specifically, text samples and the duration of each phoneme in the text samples are obtained from the original speech database. For example, the phonemes corresponding to the text sample "strong #1 policy" are: sil q iang2 iou3 l i4 d e5#1 zh eng4 ce4 sil, where the duration of each phoneme is: 40 14 20 19 5 13 5 12 5 24 15 23 33 (in milliseconds). Here, #1 represents a special pause marker with no duration information; the phoneme here is a general term, and the example uses the initials and finals of Chinese with tones. 'sil' indicates silence, and the phoneme "iou3" represents the pinyin for "you" in Chinese.

[0032] S102. Replace at least one word sample in the text sample with a synonym to obtain the target text.

[0033] It should be explained that word samples can be words obtained after segmenting the text sample. For example, after segmenting the text sample "strong policy", three word samples are obtained: strong, of, and policy.

[0034] It should be noted that synonym replacement can replace word samples in the text sample with words that are synonyms of word samples in the text sample. For example, synonyms for "policy" can be "strategy," "guideline," or "method," so the synonym for "policy" in the text sample "strong policy" can be replaced with "strategy," "guideline," or "method."

[0035] Here, the target text refers to the text obtained by replacing at least one word sample in the text sample with a synonym. For example, replacing the word sample "policy" in the text sample "strong policy" with a synonym can yield the target text "strong strategy", "strong approach", or "strong method", etc.

[0036] Of course, synonym replacement can also be performed on multiple word samples in the text sample. For example, synonyms for "powerful" can be "powerful," "elite," or "robust," and synonyms for "policy" can be "strategy," "guideline," or "method." By performing synonym replacement on both the word samples "powerful" and "policy" in the text sample "powerful policy," we can obtain the target text "powerful strategy," "elite guideline," or "robust method," etc.

[0037] S103. Determine the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0038] Specifically, after obtaining the target text, the duration prediction model also needs the duration information of each phoneme (pronunciation unit) in the target text. Therefore, for the obtained target text, it is also necessary to determine the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0039] This invention provides an embodiment that obtains a text sample and the duration of each phoneme in the text sample, performs synonym replacement on at least one word sample in the text sample to obtain the target text, and determines the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text. Through the technical solution of this invention, the duration prediction model for speech synthesis can be expanded by increasing the text-duration data, significantly increasing the scale of the text-duration data, thereby enhancing the generalization ability of the duration prediction model.

[0040] Optionally, the duration of each phoneme in the target text is determined based on the text sample, the duration of each phoneme in the text sample, and the target text, including:

[0041] For the first word that is the same in the target text and the text sample, the duration of each phoneme in the first word sample that is the same as the first word in the text sample is determined as the duration of each phoneme in the first word in the target text.

[0042] It should be explained that the first word can be the same word in both the target text and the text sample. For example, if the text sample is "strong policy" and the target text is "strong strategy", then the first word would be the same word in both the target text and the text sample: "strong" and "of".

[0043] The first word sample is the word sample in the text sample that corresponds to the first word. For example, if the text sample is "strong policy" and the target text is "strong strategy", then the first word sample is the words "strong" and "of" in the text sample.

[0044] Specifically, for example, if the text sample is "strong policy" and the target text is "strong strategy", and the words that are the same in the target text and the text sample are "strong" and "of", then the duration of the first word sample "strong" and the first word sample "of" in the text sample can be determined as the duration of the first word "strong" and the first word "of" in the target text.

[0045] For a second word in the target text that differs from the text sample, if the number of characters in the second word is different from the number of characters in the second word sample, then the duration of each phoneme in the second word sample is determined based on the duration of each phoneme in the second word sample, the number of phonemes in the second word sample, the number of phonemes in the second word, and the ratio of phoneme durations in the second word.

[0046] It should be explained that the second word can be a synonym that is different in the target text and the text sample. For example, if the text sample is "strong policy" and the target text is "powerful policy", then the second word is the word "powerful" that is different in the target text and the text sample.

[0047] The second word sample is the word sample in the text sample that corresponds to the second word. For example, if the text sample is "strong policy" and the target text is "powerful policy", then the second word sample is the word "powerful" in the text sample.

[0048] Specifically, for example, if the text sample is "strong policy", the target text is "powerful policy", the second word is "powerful", and the second word sample is "strong and powerful", and the number of characters in the second word is different from the number of characters in the second word sample, then the duration of each phoneme in the second word is determined based on the duration of each phoneme in the second word sample, the number of phonemes in the second word sample, the number of phonemes in the second word, and the ratio of the phoneme durations in the second word.

[0049] For a second word in the target text that differs from the text sample, if the number of characters in the second word is the same as the number of characters in the second word sample, then the duration of each phoneme in the second word is determined based on the duration of each phoneme in the second word sample.

[0050] Specifically, for example, if the text sample is "strong policy", the target text is "strong strategy", the second word is "strategy", the second word sample is "policy", and the number of characters in the second word is the same as the number of characters in the second word sample, then the duration of each phoneme in the second word is determined based on the duration of each phoneme in the second word sample.

[0051] Optionally, the duration of each phoneme in the second word is determined based on the duration of each phoneme in the second word sample, including:

[0052] The duration of each phoneme in the second word sample is sequentially increased by a preset value or decreased by a preset value to obtain the adjusted duration of each phoneme.

[0053] The preset value can be a value that slightly alters the phoneme duration based on actual conditions. Specifically, the preset value can be obtained by random sampling between [-f, +f], where f can be a value preset based on actual conditions. This embodiment of the invention does not limit the magnitude of the value f.

[0054] For example, the text sample is "strong policy", and the corresponding phonemes are: sil q iang2 iou3 li4 d e5#1 zh eng4 c e4 sil, where the duration of each phoneme is: 40 14 20 19 5 13 5 12 5 24 15 23 (unit: millisecond); the target text is "strong strategy", the second word is "strategy", the second word sample is "policy", and the number of characters of the second word is the same as the number of characters of the second word sample. Then, the duration of each phoneme in the second word sample "policy" can be sequentially added with a preset value or subtracted by a preset value to obtain the adjusted duration of each phoneme, that is, the duration "5 24 15 23 (unit: millisecond)" of each phoneme in the phonemes "zh eng4 c e4" is added with a preset value or subtracted by a preset value, so as to obtain the adjusted durations "6", "20", "17" and "20" (unit: millisecond) of the phonemes "c", "e4", "l" and "ve4".

[0055] The adjusted duration of each phoneme is determined as the duration of each phoneme in the second word.

[0056] For example, the adjusted durations "6", "20", "17" and "20" (unit: millisecond) of the phonemes "c", "e4", "l" and "ve4" are determined as the duration of each phoneme in the second word "strategy".

[0057] Optionally, determining the duration of each phoneme in the second word according to the duration of each phoneme in the second word sample, the number of phonemes in the second word sample, the number of phonemes in the second word, and the phoneme duration ratio in the second word includes:

[0058] For a first character sample that is the same in the second word sample and the second word, the duration of each phoneme in the character in the second word that is the same as the first character sample is determined according to the duration of each phoneme in the first character sample.

[0059] It should be noted that the first character sample may be a character that is the same in the second word sample and the second word. For example, if the text sample is "strong policy" and the target text is "powerful policy", then the second word sample is "strong and powerful", the second word is "powerful", and the first character sample is exactly "strong" in the second word sample.

[0060] Specifically, for the first character sample that is the same in the second word sample and the second word, the duration of each phoneme of the first character sample in the second word sample may be added with a preset value or subtracted by a preset value to obtain the adjusted duration of each phoneme of the first character sample, and the adjusted duration of each phoneme of the first character sample is determined as the duration of each phoneme in the character in the second word that is the same as the first character sample.

[0061] For example, if the text sample is "powerful policy" and the target text is "strong policy", the second word sample is "powerful", the second word is "strong", the first character sample is exactly the character "pow" in the second word sample, the durations "14" and "20" (in milliseconds) of each phoneme "q" and "iang2" of the first character sample "pow" in the second word sample are added with a preset value or reduced by a preset value, to obtain the durations "15" and "22" (in milliseconds) of each phoneme "q" and "iang2" of the adjusted first character sample "pow", and the durations "15" and "22" (in milliseconds) of each phoneme "q" and "iang2" of the adjusted first character sample "pow" are determined as the durations of each phoneme "q" and "iang2" in the character "pow" that is the same as the first character sample in the second word.

[0062] For the second character sample set in the second word sample that is different from that in the second word, the duration of each phoneme in the character corresponding to the second character sample set in the second word is determined according to the number of phonemes in the second character sample set, the duration information of the second character sample set, the number of phonemes of the character corresponding to the second character sample set in the second word, and the phoneme duration ratio in the character corresponding to the second character sample set in the second word.

[0063] It should be explained that the second character sample set may be a set composed of at least one character that is different in the second word sample and the second word. For example, if the text sample is "powerful policy" and the target text is "strong policy", the second word sample is "powerful", the second word is "strong", and the second character sample set includes "wer" and "ful" in the second word sample.

[0064] Optionally, determining the duration of each phoneme in the character corresponding to the second character sample set in the second word according to the number of phonemes in the second character sample set, the duration information of the second character sample set, the number of phonemes of the character corresponding to the second character sample set in the second word, and the phoneme duration ratio in the character corresponding to the second character sample set in the second word includes:

[0065] The ratio of the total phoneme duration of the second character sample set to the number of phonemes in the second character sample set is determined as the average phoneme duration.

[0066] It should be noted that the total phoneme duration may be the result of summing the durations corresponding to each phoneme in the second character sample set. For example, if the text sample is "qiang you li de zheng ce" (powerful policy) and the target text is "qiang da de zheng ce" (strong policy), the second word sample is "qiang you li", the second word is "qiang da", and the second character sample set is "you" and "li", then the total phoneme duration of the second character sample set is the sum of "19", "5" and "13" corresponding to the phonemes "iou3", "l" and "i4", which is 37 (unit: millisecond).

[0067] Wherein, the average duration may be the average value of the durations corresponding to each phoneme in the second character sample set.

[0068] Specifically, the total phoneme duration of the second character sample set is divided by the number of phonemes in the second character sample set, so as to obtain the average duration of each phoneme in the second character sample set. For example, if the text sample is "qiang you li de zheng ce" and the target text is "qiang da de zheng ce", the second word sample is "qiang you li", the second word is "qiang da", the second character sample set is "you" and "li", the total phoneme duration of the second character sample set is 37 (unit: millisecond), which is the sum of the durations "19", "5" and "13" corresponding to the phonemes "iou3", "l" and "i4", and the total phoneme duration 37 (unit: millisecond) is divided by the number 3 of phonemes in the second character sample set, so as to obtain the average duration of each phoneme in the second character sample set (approximately equal to 12) (unit: millisecond).

[0069] The product of the average phoneme duration and the number of phonemes of the character corresponding to the second character sample set in the second word is determined as the total phoneme duration of the character corresponding to the second character sample set in the second word.

[0070] Specifically, the average phoneme duration is multiplied by the number of phonemes of the character corresponding to the second character sample set, so as to obtain the total phoneme duration of the character corresponding to the second character sample set in the second word. For example, if the text sample is "qiang you li de zheng ce" and the target text is "qiang da de zheng ce", the second word sample is "qiang you li", the second word is "qiang da", the second character sample set is "you" and "li", and the total phoneme duration of the second character sample set is 37 (unit: millisecond), which is the sum of the durations "19", "5" and "13" corresponding to the phonemes "iou3", "l" and "i4". The phonemes of the character "da" corresponding to the second character sample set are "d" and "a4", and the number of phonemes is 2. The total phoneme duration 37 (unit: millisecond) is divided by the number 3 of phonemes in the second character sample set to obtain the average duration of each phoneme in the second character sample set (approximately equal to 12) (unit: millisecond), and the average phoneme duration 12 (unit: millisecond) is multiplied by the number 2 of phonemes of the character corresponding to the second character sample set, so as to obtain the total phoneme duration 24 (unit: millisecond) of the character corresponding to the second character sample set in the second word.

[0071] The duration of each phoneme in the character corresponding to the second character sample set in the second word is determined according to the phoneme duration ratio of the character corresponding to the second character sample set in the second word and the total phoneme duration in the character corresponding to the second character sample set in the second word.

[0072] Specifically, the phoneme duration ratio of the character corresponding to the second character sample set in the second word may be determined based on the duration of each phoneme of the text sample in the original speech library, for example, the ratio of the average pronunciation duration of "d" and "a4" (such as 1:2), and the duration of each phoneme in the character corresponding to the second character sample set in the second word is determined according to the phoneme duration ratio of the character corresponding to the second character sample set in the second word and the total phoneme duration in the character corresponding to the second character sample set in the second word. For example, the total phoneme duration of 24 (in milliseconds) in the character corresponding to the second character sample set in the second word obtained above is divided according to the ratio of 1:2 of the average duration of phonemes "d" and "a4", then the durations of phonemes "d" and "a4" of the character "da" are "8" and "16" (in milliseconds).

[0073] Optionally, performing synonym replacement on at least one word sample in the text sample to obtain the target text includes:

[0074] Word segmentation processing is performed on the text sample to obtain at least one word sample.

[0075] It can be known that word segmentation, also called tokenization, refers to the process of converting an original character stream into individual tokens. In this embodiment, the word segmentation processing may be splitting the text sample into individual separate words (i.e., word samples). Specifically, Chinese word segmentation algorithms are mainly divided into thesaurus-based word segmentation algorithms, statistical model-based word segmentation algorithms, and sequence labeling-based word segmentation algorithms, wherein the thesaurus-based word segmentation algorithms may include a forward maximum matching method, a reverse maximum matching method, and a bidirectional maximum matching method, the statistical model-based word segmentation algorithms may be a word segmentation method based on N-gram (a statistical language model) language model, and the sequence labeling-based word segmentation algorithms may include a word segmentation method based on HMM (Hidden Markov Model), a word segmentation method based on MEMM (Maximum Entropy Markov Model), a word segmentation method based on CRF (Conditional Random Field, a machine learning technology), and a deep learning-based end-to-end word segmentation method. It should be noted that this embodiment does not limit the specific word segmentation method used.

[0076] Specifically, a Chinese word segmentation algorithm may be used to perform word segmentation processing on a text sample to obtain at least one word sample. For example, if the text sample is "powerful policy", performing word segmentation processing on the text sample may yield 3 word samples: powerful, of, and policy.

[0077] Perform synonym replacement on the at least one word sample to obtain a target text.

[0078] Specifically, a synonym table may be prepared in advance, and lookup and replacement may be performed directly according to the word samples. For example, the word table may have the word "powerful" corresponding to synonyms such as "mighty", "strong" and "robust", and the word "policy" corresponding to synonyms such as "strategy", "guideline" and "method". Synonym replacement is performed on the at least one word sample to obtain a target text. For example, synonym replacement may be performed on the word sample "powerful" and / or the word sample "policy" in the text sample "powerful policy", so as to obtain target texts such as "mighty policy", "strong policy", "robust policy", "powerful strategy", "powerful guideline", "powerful method", "mighty strategy", "mighty guideline", "mighty method", "strong strategy", "strong guideline", "strong method", "robust strategy", "robust guideline" and "robust method".

[0079] Optionally, determining the duration of each phoneme in the target text according to the text sample, the duration of each phoneme in the text sample, and the target text includes:

[0080] Obtain the perplexity indicator of the target text and the perplexity indicator of the text sample.

[0081] It can be known that perplexity is a common evaluation indicator for the quality of a language model, and the perplexity indicator can directly evaluate the language model. In simple terms, a language model calculates the probability value of a sentence, and sentences in a test set are usually recognized as normal sentences in the actual production environment of the model. For a language model well-trained on a training dataset, the probability values of normal sentences in the test set are calculated. The formula for calculating the perplexity of the target text / text sample is as follows:

[0082]

[0083] where PPL represents the perplexity of the target text / text sample, N represents the number of words in the sentence of the target text / text sample, and p(ω i ) represents the probability of the i-th word in the target text / text sample, and p(ω i |ω1ω2…ω i-1 ) represents the probability of the i-th word calculated based on the first i-1 words.

[0084] As can be seen from the above formula, the lower the perplexity obtained on the test set, the better the language model is; that is, the higher the probability value of a normal sentence, the better the language model is.

[0085] If the ratio of the target difference to the perplexity index of the text sample is less than the ratio threshold, then the duration of each phoneme in the target text is determined based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0086] The target difference is the difference between the perplexity index of the target text and the perplexity index of the text sample.

[0087] The ratio threshold can be a pre-set threshold based on actual conditions. If the ratio of the difference between the perplexity index of the target text and the perplexity index of the text sample to the perplexity index of the text sample is less than this threshold, it indicates that the target text conforms to conventional language. Specifically, the size of the ratio threshold is not limited in this embodiment of the invention.

[0088] Specifically, the generated target text may not conform to conventional language. Therefore, it can be evaluated using a speech model (such as N-gram), and target text with excessively high perplexity scores can be deleted. N-gram needs to be trained independently. The higher the perplexity score of a sentence, the less like the sentence is to conventional language. The deletion rules are as follows:

[0089]

[0090] Among them, PPL new PPL is a perplexity metric representing the target text. ori The perplexity index represents the text sample, and T is the ratio threshold.

[0091] Specifically, if the ratio of the target difference to the perplexity index of the text sample is greater than or equal to the ratio threshold, the target text is deleted; if the ratio of the target difference to the perplexity index of the text sample is less than the ratio threshold, the duration of each phoneme in the target text is determined based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0092] In addition, the difference between the perplexity index of the target text and the perplexity index of the text sample can also be used for judgment. For example, a difference threshold can be set. If the difference between the perplexity index of the target text and the perplexity index of the text sample is less than the difference threshold, then the duration of each phoneme in the target text can be determined based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0093] Optionally, after generating target data based on the target text and its duration information, the following may also be included:

[0094] The first phoneme sequence sample is determined based on the text sample.

[0095] It should be noted that the first phoneme sequence sample may be the phoneme sequence corresponding to the text sample in the original speech library. For example, the phoneme sequence corresponding to the text sample "powerful policy" in the original speech library is: sil q iang2iou3 l i4 d e5 zh eng4 c e4 sil.

[0096] Specifically, the phonemes of all characters in the text sample in the original speech library are determined as the first phoneme sequence sample.

[0097] A second phoneme sequence sample is determined according to the target text.

[0098] It should be explained that the second phoneme sequence sample may be the phoneme sequence corresponding to the target text obtained by performing at least one synonym replacement on the text sample in the original speech library. For example, the phoneme sequence corresponding to the target sample "strong policy" obtained by performing synonym replacement on the text sample "powerful policy" in the original speech library is: silq iang2 d a4 d e5 zh eng4 c e4 sil.

[0099] Specifically, the phonemes of all characters in the target text obtained by performing at least one synonym replacement on the text sample in the original speech library are determined as the second phoneme sequence sample.

[0100] A target sample set is generated according to the first phoneme sequence sample, the duration of each phoneme in the first phoneme sequence sample, the second phoneme sequence sample, and the duration of each phoneme in the second phoneme sequence sample.

[0101] The target sample set may consist of at least one first phoneme sequence sample corresponding to the text sample in the original speech library and the duration of each phoneme in the first phoneme sequence sample, as well as at least one second phoneme sequence sample corresponding to the target text obtained by performing at least one synonym replacement on the text sample in the original speech library and the duration of each phoneme in the second phoneme sequence sample.

[0102] Specifically, the target sample set is composed of the first phoneme sequence sample corresponding to the text sample in the original speech library and the duration of each phoneme in the first phoneme sequence sample, as well as the second phoneme sequence sample corresponding to the target text obtained by performing at least one synonym replacement on the text sample in the original speech library and the duration of each phoneme in the second phoneme sequence sample.

[0103] The phoneme sequence samples in the target sample set are input into a pre-established neural network model, and the predicted duration of each phoneme in the phoneme sequence samples is obtained.

[0104] It should be noted that a phoneme sequence sample can be a phoneme sequence in the target sample set composed of a first phoneme sequence sample corresponding to a text sample in the original speech database and the duration of each phoneme in the first phoneme sequence sample, a second phoneme sequence sample corresponding to the target text obtained after replacing the text sample in the original speech database with at least one synonym, and the duration of each phoneme in the second phoneme sequence sample.

[0105] The phoneme sequence samples include either the first phoneme sequence sample or the second phoneme sequence sample.

[0106] In this embodiment, the neural network model can be the neural network model corresponding to the duration prediction model.

[0107] The predicted duration can be the predicted duration of each phoneme in the phoneme sequence sample of the target sample set output by the neural network model. The phoneme sequence sample can include at least one first phoneme sequence sample corresponding to a text sample in the original speech library and the duration of each phoneme in the first phoneme sequence sample, as well as at least one second phoneme sequence sample corresponding to the target text obtained by performing at least one synonym replacement on the text sample in the original speech library and the duration of each phoneme in the second phoneme sequence sample.

[0108] Specifically, the first or second phoneme sequence sample from the target sample set is input into a pre-established neural network model. That is, the target text obtained by simultaneously replacing the text sample in the original speech database with at least one synonym is trained to obtain the predicted duration of each phoneme in the phoneme sequence sample output by the neural network model.

[0109] The parameters of the neural network model are trained based on the objective function formed by the predicted duration of each phoneme in the phoneme sequence sample and the duration of each phoneme in the phoneme sequence sample.

[0110] The objective function can be the activation function of the neural network model.

[0111] Specifically, the parameters of the neural network model are trained based on the objective function formed by the predicted duration of each phoneme in the phoneme sequence sample and the duration of each phoneme in the phoneme sequence sample, until a well-trained neural network model is obtained.

[0112] This invention proposes a text data expansion scheme for speech synthesis duration prediction models, which can enhance the generalization ability of the trained duration prediction model.

[0113] Example 2

[0114] Figure 2This is a schematic diagram of a duration determination device according to an embodiment of the present invention. This embodiment is applicable to situations requiring duration determination. The device can be implemented using software and / or hardware, and can be integrated into any device that provides duration determination functionality, such as… Figure 2 As shown, the duration determination device specifically includes: an acquisition module 201, a synonym replacement module 202, and a determination module 203.

[0115] The acquisition module 201 is used to acquire text samples and the duration of each phoneme in the text samples.

[0116] The synonym replacement module 202 is used to replace at least one word sample in the text sample with a synonym to obtain the target text.

[0117] The determining module 203 is used to determine the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0118] Optionally, the determining module 203 includes:

[0119] The first determining unit is configured to, for the first word that is the same in the target text and the text sample, determine the duration of each phoneme in the first word sample that is the same as the first word in the text sample as the duration of each phoneme in the first word in the target text, wherein the first word sample is the word sample in the text sample that corresponds to the first word;

[0120] The second determining unit is used to determine the duration of each phoneme in the second word based on the duration of each phoneme in the second word sample, the number of phonemes in the second word sample, the number of phonemes in the second word sample, and the phoneme duration ratio in the second word for a second word that is different from the text sample in the target text. The second word sample is the word sample in the text sample that corresponds to the second word.

[0121] The third determining unit is used to determine the duration of each phoneme in the second word based on the duration of each phoneme in the second word sample if the number of characters in the second word is the same as the number of characters in the second word sample.

[0122] Optionally, the third determining unit includes:

[0123] The adjustment subunit is used to sequentially add a preset value to or reduce the duration of each phoneme in the second word sample to obtain the adjusted duration of each phoneme.

[0124] The first determining subunit is used to determine the duration of each adjusted phoneme as the duration of each phoneme in the second word.

[0125] Optionally, the second determining unit includes:

[0126] The second determining subunit is used to determine the duration of each phoneme in the second word that is the same as the first character sample in the second word sample, based on the duration of each phoneme in the first character sample.

[0127] The third determining subunit is used to determine the duration of each phoneme in the second word corresponding to the second character sample set, based on the number of phonemes in the second character sample set, the duration information of the second character sample set, the number of phonemes in the second word corresponding to the second character sample set, and the phoneme duration ratio in the second word corresponding to the second character sample set, for the second character sample set that is different from the second word sample set.

[0128] Optionally, the third determining subunit is specifically used for:

[0129] The ratio of the total phoneme duration in the second character sample set to the number of phonemes in the second character sample set is determined as the average phoneme duration.

[0130] The total duration of phonemes in the second word corresponding to the second character sample set is determined by multiplying the average duration of the phonemes by the number of phonemes in the second word corresponding to the second character sample set.

[0131] The duration of each phoneme in the second word corresponding to the second character sample set is determined based on the ratio of phoneme durations of the characters in the second word corresponding to the second character sample set and the total duration of phonemes in the characters in the second word corresponding to the second character sample set.

[0132] Optionally, the synonym replacement module 202 includes:

[0133] The word segmentation processing unit is used to segment the text sample to obtain at least one word sample;

[0134] The synonym replacement unit is used to perform synonym replacement on the at least one word sample to obtain the target text.

[0135] Optionally, the determining module 203 includes:

[0136] An acquisition unit is used to acquire the perplexity index of the target text and the perplexity index of the text sample;

[0137] The fourth determining unit is used to determine the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text if the ratio of the target difference to the perplexity index of the text sample is less than the ratio threshold.

[0138] Optionally, the device further includes:

[0139] The fifth determining unit is used to determine the first phoneme sequence sample based on the text sample after generating target data according to the target text and the duration information of the target text;

[0140] The sixth determining unit is used to determine a second phoneme sequence sample based on the target text after generating target data based on the target text and the duration information of the target text;

[0141] The generation unit is configured to generate a target sample set based on the first phoneme sequence sample, the duration of each phoneme in the first phoneme sequence sample, the second phoneme sequence sample, and the duration of each phoneme in the second phoneme sequence sample after generating target data according to the target text and the duration of the target text.

[0142] An input unit is configured to, after generating target data based on the target text and the duration information of the target text, input phoneme sequence samples from the target sample set into a pre-established neural network model to obtain the predicted duration of each phoneme in the phoneme sequence samples, wherein the phoneme sequence samples include: a first phoneme sequence sample or a second phoneme sequence sample;

[0143] The forming unit is used to train the parameters of the neural network model based on the predicted duration of each phoneme in the phoneme sequence sample and the duration of each phoneme in the phoneme sequence sample after generating target data according to the target text and the duration of the target text.

[0144] The above-mentioned product can execute the duration determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0145] Example 3

[0146] Figure 3A schematic diagram of an electronic device 30 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0147] like Figure 3 As shown, the electronic device 30 includes at least one processor 31 and a memory, such as a read-only memory (ROM) 32 or a random access memory (RAM) 33, communicatively connected to the at least one processor 31. The memory stores computer programs executable by the at least one processor. The processor 31 can perform various appropriate actions and processes based on the computer program stored in the ROM 32 or loaded from storage unit 38 into the RAM 33. The RAM 33 can also store various programs and data required for the operation of the electronic device 30. The processor 31, ROM 32, and RAM 33 are interconnected via a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0148] Multiple components in electronic device 30 are connected to I / O interface 35, including: input unit 36, such as keyboard, mouse, etc.; output unit 37, such as various types of monitors, speakers, etc.; storage unit 38, such as disk, optical disk, etc.; and communication unit 39, such as network card, modem, wireless transceiver, etc. Communication unit 39 allows electronic device 30 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0149] Processor 31 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 31 performs the various methods and processes described above, such as the duration determination method:

[0150] Obtain a text sample and the duration of each phoneme in the text sample;

[0151] At least one word sample in the text sample is replaced with a synonym to obtain the target text;

[0152] The duration of each phoneme in the target text is determined based on the text sample, the duration of each phoneme in the text sample, and the target text.

[0153] In some embodiments, the duration determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 38. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 30 via ROM 32 and / or communication unit 39. When the computer program is loaded into RAM 33 and executed by processor 31, one or more steps of the duration determination method described above may be performed. Alternatively, in other embodiments, processor 31 may be configured to perform the duration determination method by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0159] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0160] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining duration, characterized in that, include: Obtain a text sample and the duration of each phoneme in the text sample; At least one word sample in the text sample is replaced with a synonym to obtain the target text; The duration of each phoneme in the target text is determined based on the text sample, the duration of each phoneme in the text sample, and the target text. The determination of the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text includes: For the first word that is the same in the target text and the text sample, the duration of each phoneme in the first word sample that is the same as the first word in the text sample is determined as the duration of each phoneme in the first word in the target text, wherein the first word sample is the word sample in the text sample that corresponds to the first word; For a second word in the target text that is different from the text sample, if the number of characters in the second word is different from the number of characters in the second word sample, then the duration of each phoneme in the second word sample is determined according to the duration of each phoneme in the second word sample, the number of phonemes in the second word sample, the number of phonemes in the second word, and the ratio of phoneme durations in the second word, wherein the second word sample is the word sample in the text sample that corresponds to the second word; For a second word in the target text that is different from the text sample, if the number of characters in the second word is the same as the number of characters in the second word sample, then the duration of each phoneme in the second word is determined based on the duration of each phoneme in the second word sample.

2. The method according to claim 1, characterized in that, Determining the duration of each phoneme in the second word based on the duration of each phoneme in the second word sample includes: The duration of each phoneme in the second word sample is sequentially increased by a preset value or decreased by a preset value to obtain the adjusted duration of each phoneme. The duration of each adjusted phoneme is determined to be the duration of each phoneme in the second word.

3. The method according to claim 1, characterized in that, The duration of each phoneme in the second word is determined based on the duration of each phoneme in the second word sample, the number of phonemes in the second word sample, the number of phonemes in the second word, and the phoneme duration ratio in the second word, including: For the first character sample that is the same as the first character sample in the second word sample, the duration of each phoneme in the second word that is the same as the first character sample is determined according to the duration of each phoneme in the first character sample; For the second character sample set that is different from the second word sample set, the duration of each phoneme in the second word corresponding to the second character sample set is determined based on the number of phonemes in the second character sample set, the duration information of the second character sample set, the number of phonemes in the second word corresponding to the second character sample set, and the phoneme duration ratio in the second word corresponding to the second character sample set.

4. The method according to claim 3, characterized in that, The duration of each phoneme in the character corresponding to the second character sample set in the second word is determined based on the number of phonemes in the second character sample set, the duration information of the second character sample set, the number of phonemes in the character corresponding to the second character sample set in the second word, and the ratio of phoneme durations in the character corresponding to the second character sample set in the second word. This includes: The ratio of the total phoneme duration in the second character sample set to the number of phonemes in the second character sample set is determined as the average phoneme duration. The total duration of phonemes in the second word corresponding to the second character sample set is determined by multiplying the average duration of the phonemes by the number of phonemes in the second word corresponding to the second character sample set. The duration of each phoneme in the second word corresponding to the second character sample set is determined based on the ratio of phoneme durations of the characters in the second word corresponding to the second character sample set and the total duration of phonemes in the characters in the second word corresponding to the second character sample set.

5. The method according to claim 1, characterized in that, At least one word sample in the text sample is replaced with a synonym to obtain the target text, including: The text sample is segmented to obtain at least one word sample; The target text is obtained by performing synonym replacement on at least one word sample.

6. The method according to claim 1, characterized in that, Determining the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text includes: Obtain the perplexity index of the target text and the perplexity index of the text sample; If the ratio of the target difference to the perplexity index of the text sample is less than the ratio threshold, then the duration of each phoneme in the target text is determined based on the text sample, the duration of each phoneme in the text sample, and the target text, wherein the target difference is the difference between the perplexity index of the target text and the perplexity index of the text sample.

7. The method according to claim 1, characterized in that, After determining the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text, the method further includes: Determine the first phoneme sequence sample based on the text sample; Determine a second phoneme sequence sample based on the target text; A target sample set is generated based on the first phoneme sequence sample, the duration of each phoneme in the first phoneme sequence sample, the second phoneme sequence sample, and the duration of each phoneme in the second phoneme sequence sample. The phoneme sequence samples in the target sample set are input into a pre-established neural network model to obtain the predicted duration of each phoneme in the phoneme sequence samples. The phoneme sequence samples include: a first phoneme sequence sample or a second phoneme sequence sample. The parameters of the neural network model are trained based on the objective function formed by the predicted duration of each phoneme in the phoneme sequence sample and the duration of each phoneme in the phoneme sequence sample.

8. A duration determination device, characterized in that, include: The acquisition module is used to acquire text samples and the duration of each phoneme in the text samples; The synonym replacement module is used to replace at least one word sample in the text sample with a synonym to obtain the target text. The determination module is used to determine the duration of each phoneme in the target text based on the text sample, the duration of each phoneme in the text sample, and the target text. The determining module includes: The first determining unit is configured to, for the first word that is the same in the target text and the text sample, determine the duration of each phoneme in the first word sample that is the same as the first word in the text sample as the duration of each phoneme in the first word in the target text, wherein the first word sample is the word sample in the text sample that corresponds to the first word; The second determining unit is used to determine the duration of each phoneme in the second word based on the duration of each phoneme in the second word sample, the number of phonemes in the second word sample, the number of phonemes in the second word sample, and the phoneme duration ratio in the second word for a second word that is different from the text sample in the target text. The second word sample is the word sample in the text sample that corresponds to the second word. The third determining unit is used to determine the duration of each phoneme in the second word based on the duration of each phoneme in the second word sample if the number of characters in the second word is the same as the number of characters in the second word sample.

9. A terminal device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images