Method for determining the prosodic structure of Chinese text in synthesizing speech and computer-readable storage medium
By extracting the semantic features and modeling contextual relationships of Chinese texts, and using dynamic programming algorithms to find the optimal rhythm structure tree, the problem of ignoring hierarchical relationships and relying on additional features in the existing technology is solved, and a more accurate Chinese text synthesized pronunciation is achieved. Rhythmic structure prediction.
Patent Information
- Application Number
- CN202111525128.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-14
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-14
AI Technical Summary
The prior art ignores the hierarchical relationship of the pronunciation structure when synthesizing pronunciation in Chinese text, resulting in a high prediction error rate of pronunciation structure and requires additional word participle and part-of-speech characteristics as input. Complex feature engineering is not conducive to practical application.
The semantic representation vector is extracted using a pre-trained word-level semantic feature extractor, and the contextual relationship is modeled by the encoder. The decoder uses a span representation method to model the prosthetic structure tree, uses a dynamic programming algorithm to find the optimal prosthetic structure tree, and The final sequence of rhythmic structure tags is obtained through regular transformation.
Unify the hierarchical relationship between pronunciation words, pronunciation phrases and tone phrases, improve prediction performance, avoid error accumulation and complex feature engineering, and simplify the input and output process.
Smart Images

Figure CN114242039B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text-to-speech conversion, and in particular to a method for determining a prosodic structure when synthesizing speech from Chinese text and a computer storage medium. Background Art
[0002] Text-to-speech (TTS) technology has been widely used in voice assistants, smart navigation, e-books and other products. For ideographic languages, such as Chinese, front-end text analysis is very necessary before synthesizing speech, that is, converting the input text into corresponding linguistic features. Among them, prosodic structure prediction is an indispensable part, and the quality of the prediction results has a great impact on the prosodic pause effect of the final synthesized speech.
[0003] The prosodic structure of Chinese is as follows Figure 1 As shown in the figure, it is generally divided into three levels, namely prosodic words, prosodic phrases and intonation phrases, which are three prosodic levels between sentences and Chinese characters. Prosodic words are composed of multiple Chinese characters, which are larger than dictionary words in syntax. They are a group of characters that are closely related and often pronounced together in actual speech flow; prosodic phrases are composed of multiple prosodic words, which are medium rhythm blocks between prosodic words and intonation phrases. The several prosodic words that make up the prosodic phrases sound like they share a common rhythm group; intonation phrases are composed of multiple prosodic phrases, which are connected according to a certain sentence intonation pattern, corresponding to a large pause; sentences are composed of multiple intonation phrases.
[0004] The current prosodic structure prediction schemes mainly have the following problems: the prosodic structure is a tree-like hierarchical relationship, and most of the current schemes ignore this hierarchical relationship and instead convert the prediction of the prosodic structure into three levels of sub-tasks, that is, they are split into three binary classification tasks for predicting prosodic words, prosodic phrases, and intonation phrases, respectively. Each sub-task uses an equal-length sequence-to-sequence method to predict a label of 0 or 1 for each Chinese character or word segmentation; the sequence-to-sequence method used in the prior art schemes requires additional features such as word segmentation and part of speech as input, and error accumulation will affect the accuracy of prosodic structure prediction. Complex feature engineering is also not conducive to practical applications.
[0005] The prior art lacks an effective method for determining the prosodic structure when synthesizing speech from Chinese text.
[0006] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention
[0007] In order to solve the existing problems, the present invention provides a method for determining the prosodic structure when synthesizing speech from Chinese text and a computer storage medium.
[0008] In order to solve the above problems, the technical solution adopted by the present invention is as follows:
[0009] A method for determining the prosodic structure when synthesizing speech from Chinese text comprises the following steps: S1: inputting each sentence of Chinese text to be synthesized into a pre-trained word-level semantic feature extractor to obtain a semantic representation vector of each word; a sentence of the Chinese text to be synthesized is a sequence of Chinese characters containing punctuation marks;
[0010] S2: Input the semantic representation vector of each word into the encoder to obtain the hidden state vector of each word containing the contextual relationship; S3: Input the hidden state vector into the decoder, the decoder represents each span through the difference of the hidden state vectors at different positions, models the rhythm structure tree based on the span representation method, and uses the dynamic programming algorithm to find the optimal rhythm structure tree; S4: Convert the optimal rhythm structure tree into a final complete character sequence with rhythm structure labels through a rule-based format.
[0011] Preferably, the pre-trained word-level semantic feature extractor is a semantic feature extractor BERT based on an attention mechanism and a fully connected network.
[0012] Preferably, the encoder is composed of multiple Transformer blocks, each of which contains two sub-layers: one is a multi-head attention mechanism layer and the other is a fully connected layer; each sub-layer has a residual connection and layer normalization; the encoder is based on the input position i and the character x at position j i and x j The semantic representation vectors of the corresponding positions are obtained by i and h j .
[0013] Preferably, a rhythm structure tree is represented as follows:
[0014] T:={(i t ,j t ,l t ):t=1,...,|T|}
[0015] The triple (i, j, l) represents a labeled span, where i and j refer to the starting and ending positions of the span (i, j) respectively, and l refers to the label corresponding to the span, l∈L, where L refers to the prosodic structure label set in a broad sense; for unary chains or nested prosodic components, a nested label will be given; the data representation of span (i, j) is defined as the difference between the corresponding hidden state vectors: h ij =h i -h j .
[0016] Preferably, each of the spans is characterized, and a multi-layer feedforward neural network is used to define each span span(i,j) to specify a score s(i,j,l) corresponding to a label l:
[0017] s(i,j,l)=[W2g(W1h ij +z1)+z2] l
[0018] Among them, g is the ReLU activation function, W1, W2, z1, z2 represent the parameters of the feedforward neural network;
[0019] Assign a corresponding score s(T) to all possible rhythm structure trees T:
[0020]
[0021] The rhythm structure tree with the highest score is determined as the optimal rhythm structure tree.
[0022] Preferably, a CKY-style bottom-up dynamic programming algorithm is used to find the optimal rhythm structure tree, specifically including: constructing an n-ary rhythm structure tree by continuously performing binary division; nodes that appear in the binary division process but are not rhythm structure components themselves will be given an empty label And it satisfies the following properties: the scores of all spans with empty labels are zero; define s best (i, j) represents the best score of the prosody structure tree of span (i, j). For a general span, a label l∈L is assigned independently each time and a best separation point k (i<k<j) is selected. Then, the above recursive process is repeated for the left and right subspans after separation. The formula is as follows:
[0023]
[0024] For a span of length 1, the best score depends only on the choice of label l:
[0025]
[0026] Calculate the best score s for each sentence of length n best (0,n), which is the score of the optimal tree for the entire sentence;
[0027] By backtracking the entire process of finding the separation point and specifying the label, the corresponding rhythm structure tree is reconstructed; during the reconstruction process, the spans with the specified empty label are merged to obtain a complete n-ary tree;
[0028] The complexity of the entire algorithm is O(n 3 +Dn 2 ), where D is the size of the prosodic structure label set L.
[0029] Preferably, for a given truth tree T * , so that all possible predicted rhythm structure trees T satisfy the following interval constraints:
[0030] s(T * )≥s(T)+Δ(T,T * )
[0031] The above constraints are implemented by minimizing the following chain losses:
[0032]
[0033] where Δ is the Hamming loss for the labeled span.
[0034] Preferably, the chain loss is achieved by making a minor modification to the dynamic programming algorithm in the decoding stage, namely, replacing s(i,j,l) with in, is the span span(i,j) in the truth tree T * The tags in .
[0035] Preferably, the optimal rhythm structure tree is converted into a rule-based format to obtain the final complete character sequence with rhythm structure tags, including: S41: moving each rhythm structure tag of the optimal rhythm structure tree to the back of the corresponding rhythm level component to obtain a sequence containing multiple layers of rhythm structure tags; S42: merging the multiple layers of rhythm tags after each rhythmic word, and only retaining the rhythm structure tag at the highest level to obtain the final complete character sequence with rhythm structure tags.
[0036] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0037] The beneficial effects of the present invention are: providing a method and computer storage medium for determining the prosodic structure when synthesizing speech from Chinese text, extracting semantic features through a feature extractor, further modeling contextual relationships through an encoder, characterizing the prosodic structure tree using spans through a decoder, and finding the optimal prosodic structure tree through a dynamic programming algorithm. Rule-based post-processing converts the prosodic structure tree, an intermediate representation form, into the form of a final prosodic structure label sequence according to certain rules; predicting prosodic structure labels in a unified end-to-end method, accepting a character sequence containing punctuation as input, and outputting a complete character sequence with a prosodic structure label, which is more concise and convenient.
[0038] Furthermore, a span representation-based method is used to model the prosody structure tree, and a dynamic programming algorithm is used to find the optimal prosody structure tree. Relying on the tree structure, the three subtasks of prosodic words, prosodic phrases and intonation phrases are unified, and the inclusion relationship between the three prosodic structure levels is better considered, which can greatly improve the prediction performance.
[0039] Furthermore, the present invention does not require information such as word segmentation and part of speech as additional input, thus avoiding the impact of error accumulation and complex feature engineering, which is more conducive to practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 Schematic diagram of the Chinese prosodic structure hierarchy in an embodiment of the present invention.
[0041] Figure 2 Schematic diagram of an end-to-end prosodic structure prediction solution in an embodiment of the present invention.
[0042] Figure 3 It is a schematic diagram of a method for determining the prosodic structure when synthesizing speech from Chinese text in an embodiment of the present invention.
[0043] Figure 4 This is a practical application scenario of a method for determining the prosodic structure when synthesizing speech from Chinese text in an embodiment of the present invention.
[0044] Figure 5 It is a structural diagram of an end-to-end rhythmic structure prediction model in an embodiment of the present invention.
[0045] Figure 6 It is a schematic diagram of obtaining the final complete character sequence with prosodic structure labels in an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0047] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and circuit connection.
[0048] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0049] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0050] TTS: Text-to-speech, also known as speech synthesis, refers to the conversion of text information into standard and fluent pronunciation.
[0051] BLSTM: Bi-directional Long Short-Term Memory, a bidirectional long short-term memory network. It is an implementation of the recurrent neural network (RNN).
[0052] CRF: Conditional Random Fields, a discriminative probability model, is a type of random field.
[0053] Transformer: A model based on self-attention mechanism.
[0054] BERT: Bidirectional Encoder Representation from Transformers is a pre-trained language representation model.
[0055] MTL: Multi-Task Learning, multi-task learning.
[0056] CKY: Cocke–Kasami–Younger algorithm is an algorithm used to perform grammatical analysis on context-free grammars.
[0057] Most of the existing methods are not unified end-to-end models, and cannot predict three levels of prosodic structure labels at the same time, so it is impossible to directly obtain an output text sequence with prosodic structure labels from an input text sequence; the prosodic structure is a tree-like hierarchical structure, and most of the current methods ignore this hierarchical relationship and instead convert the prediction of the prosodic structure into three-level subtasks, that is, split into three binary classification tasks for predicting prosodic words, prosodic phrases, and intonation phrases respectively; existing solutions all use an equal-length sequence-to-sequence method to predict a label of 0 or 1 for each Chinese character or word segmentation, which will result in serious data label imbalance problems; this sequence-to-sequence method requires additional features such as word segmentation and part of speech as input, and error accumulation will have a great impact on the performance of the prosodic structure prediction task, and complex feature engineering is not conducive to practical applications.
[0058] In view of the above problems, the present invention proposes an end-to-end prosodic structure prediction solution based on a neural network.
[0059] like Figure 2 As shown, the solution of the present invention consists of four parts: a semantic feature extractor pre-trained based on a large amount of unsupervised data, an encoder that further models contextual relationships, a decoder based on span representation and dynamic programming algorithm, and rule-based post-processing. Its input is a Chinese character sequence containing punctuation (a sentence), and its output is the original character sequence with prosodic structure labels.
[0060] like Figure 3 As shown, the present application proposes a method for determining the prosodic structure of Chinese text when synthesizing speech, characterized in that it includes the following steps:
[0061] S1: inputting each sentence of Chinese text to be synthesized into a pre-trained word-level semantic feature extractor to obtain a semantic representation vector of each word; a sentence of the Chinese text to be synthesized is a sequence of Chinese characters including punctuation marks;
[0062] In the present invention, a character sequence containing punctuation (ie, a sentence) is accepted as input without any additional feature extraction. An end-to-end prosodic structure prediction model is established.
[0063] Furthermore, in the present invention, pre-training is performed with the help of a large amount of unlabeled data, which promotes the semantic feature extractor to extract effective semantic features and improves the performance of the semantic feature extractor.
[0064] S2: Input the semantic representation vector of each word into the encoder to obtain a hidden state vector of each word containing the context relationship;
[0065] S3: inputting the hidden state vector into a decoder, wherein the decoder characterizes each span by the difference of the hidden state vectors at different positions, models the rhythm structure tree based on the span characterization method, and uses a dynamic programming algorithm to find the optimal rhythm structure tree;
[0066] In the present invention, the prosody structure tree is characterized by span, the three tasks of prosodic words, prosodic phrases and intonation phrases are unified by tree structure, and the optimal prosody structure tree is found by dynamic programming decoding. The tree relationship between the three prosodic levels is fully considered, the influence of serious error accumulation on performance is avoided, and the prediction effect is greatly improved.
[0067] S4: converting the optimal prosody structure tree into a final complete character sequence with prosody structure tags through a rule-based format.
[0068] The present invention is of great value to the human-computer interaction industry and can be used in a variety of application scenarios such as speech synthesis, speech interaction, voice assistants, and audio novel generation. Compared with the prior art, it can more accurately obtain the rhythmic structure label sequence of the target text to be synthesized.
[0069] In the present invention, the semantic feature extractor is pre-trained on a large amount of unsupervised data; the encoder uses a network structure similar to the semantic feature extractor to further model the contextual relationship; the decoder uses span to represent the rhythm structure tree and finds the optimal rhythm structure tree through a dynamic programming algorithm; the rule-based post-processing converts the rhythm structure tree, an intermediate representation form, into the final rhythm structure label sequence according to certain rules. Compared with the existing methods, it has at least the following advantages:
[0070] The prosodic structure labels are predicted in a unified end-to-end method, which accepts character sequences containing punctuation as input and outputs complete character sequences with prosodic structure labels, which is more concise and convenient.
[0071] Furthermore, a span representation-based method is used to model the prosody structure tree, and a dynamic programming algorithm is used to find the optimal prosody structure tree. Relying on the tree structure, the three subtasks of prosodic words, prosodic phrases and intonation phrases are unified, and the inclusion relationship between the three prosodic structure levels is better considered, which can greatly improve the prediction performance.
[0072] Furthermore, the present invention does not require information such as word segmentation and part of speech as additional input, thus avoiding the impact of error accumulation and complex feature engineering, which is more conducive to practical applications.
[0073] In one embodiment of the present invention, the key to modeling the rhythm structure tree lies in the representation of span and the representation of scoring function. The specific implementation of span representation can be to make a difference in hidden state vectors, or to consider other hidden state vectors in the middle, etc.; the specific implementation of scoring function representation can be to use multi-layer full connection, or a recurrent neural network or a convolutional neural network, etc.
[0074] In another embodiment of the present invention, the key to optimizing the rhythm structure tree is to use an effective dynamic programming algorithm, which can be implemented from bottom to top or from top to bottom.
[0075] like Figure 4 As shown, it is a practical application scenario of a method of determining the prosodic structure of Chinese text when synthesizing speech of the present invention. In the method of the present invention, the user only needs to provide a character sequence without a prosodic structure label as input. The end-to-end prosodic structure prediction model directly accepts a character sequence as input and outputs a character sequence with a prosodic structure label. For example, if the user wants to predict the prosodic structure of the sentence "The monkey swings on the swing with its tail.", just input the original character sequence (a sentence), and the prosodic structure sequence of the sentence will be automatically returned.
[0076] like Figure 5 As shown, a structural diagram of an end-to-end rhythmic structure prediction model implementing the method of the present invention.
[0077] In one embodiment of the present invention, the semantic feature extractor is first pre-trained on a large amount of Chinese data to extract effective semantic features. The pre-trained word-level semantic feature extractor is a semantic feature extractor BERT based on an attention mechanism and a fully connected network.
[0078] The encoder further fuses the output of the pre-trained semantic feature extractor to better model the contextual relationship and finally outputs the context-related hidden state vector representation. The encoder consists of multiple Transformer blocks, each of which contains two sub-layers: one is a multi-head attention mechanism layer and the other is a fully connected layer; each sub-layer has a residual connection and layer normalization;
[0079] The encoder calculates the character x at position i and position j according to the input i and x j The semantic representation vectors of the corresponding positions are obtained by i and h j .
[0080] The decoder receives the latent state representation of each character obtained by the encoder, and then decodes them to output a rhythm structure tree. Before defining the model of the rhythm structure tree, it is necessary to explain the span. A labeled span can be represented by a triple (i, j, l), where i and j refer to the starting position and the ending position of span (i, j) respectively, and l refers to the label corresponding to the span, l∈L, where L refers to the rhythm structure label set in a broad sense. One point that needs to be emphasized is that for unary chains or nested rhythm components, that is, a span corresponds to multiple rhythm components at the same time, a nested label will be given. For example, span (i, j) is both a rhythm word and a rhythm phrase, then a label "#2-#1" will be given. Therefore, a rhythm structure tree is represented as follows:
[0081] T:={(i t ,j t ,l t ):t=1,...,|T|}
[0082] The data representation of span(i,j) is defined as the difference of the corresponding hidden state vector: h ij =h i -h j .
[0083] After all span representations are obtained, each span is represented, and a multi-layer feedforward neural network is used to define each span (i, j) to specify a score s (i, j, l) corresponding to a label l:
[0084] s(i,j,l)=[W2g(W1h ij +z1)+z2] l
[0085] Among them, g is the ReLU activation function, W1, W2, z1, z2 represent the parameters of the feedforward neural network;
[0086] Assign a corresponding score s(T) to all possible prosody structure trees T:
[0087]
[0088] Then, the rhythm structure tree having the highest score is determined as the optimal rhythm structure tree.
[0089] After defining the network structure of the entire decoder, a CKY-style bottom-up dynamic programming algorithm is used to find the optimal rhythm structure tree.
[0090] The present invention uses a CKY-style dynamic programming algorithm to find the optimal rhythm structure tree.
[0091] The rhythm structure tree is usually an n-ary tree, and this CKY-style algorithm builds the n-ary tree by continuously performing binary splits. Nodes that appear during the binary splitting process but are not components of the rhythm structure are given an empty label. And satisfy the following properties:
[0092] All spans with empty labels have a score of zero;
[0093] Definitions best (i, j) represents the best score of the prosody structure tree of span (i, j). For a general span, a label l∈L is assigned independently each time and a best separation point k (i<k<j) is selected. Then, the above recursive process is repeated for the left and right subspans after separation. The formula is as follows:
[0094]
[0095] For a span of length 1, the best score depends only on the choice of label l:
[0096]
[0097] Calculate the best score s for each sentence of length n best (0,n), which is the score of the optimal tree for the entire sentence;
[0098] By backtracking the entire process of finding the separation point and specifying the label, the corresponding rhythm structure tree is reconstructed; during the reconstruction process, the spans with the specified empty label are merged to obtain a complete n-ary tree;
[0099] The complexity of the entire algorithm is O(n 3 +Dn 2), where D is the size of the prosodic structure label set L.
[0100] In the training phase, based on the above algorithm, for a given truth tree T * , so that all possible predicted rhythm structure trees T satisfy the following interval constraints:
[0101] s(T * )≥s(T)+Δ(T,T * )
[0102] It can be understood that the truth tree refers to the rhythm structure tree corresponding to the original labeled data, that is, the target. The input data of the prediction process is unlabeled, and a rhythm structure tree will be predicted, so that the predicted rhythm structure tree can eventually approach the truth tree. If all predicted trees are the same as the truth tree, the accuracy is 100%.
[0103] The above constraints are performed by minimizing the following hinge loss:
[0104]
[0105] Where Δ is the Hamming loss for the labeled span.
[0106] In one embodiment, the chain loss is implemented by making a minor modification to the dynamic programming algorithm in the decoding stage, namely, replacing s(i,j,l) with
[0107] in, is the span span(i,j) in the truth tree T * The tags in .
[0108] After the span-based decoder, there is a simple but important rule-based data format conversion module, which converts the intermediate representation in tree form into the form of a prosodic structure label sequence.
[0109] like Figure 6 As shown, the optimal prosody structure tree is converted into a final complete character sequence with a prosody structure tag through a rule-based format, including:
[0110] S41: moving each rhythm structure label of the optimal rhythm structure tree to the back of the corresponding rhythm level component to obtain a sequence containing multiple layers of rhythm structure labels;
[0111] S42: The multiple layers of prosodic tags behind each prosodic word are merged, and only the prosodic structure tag at the highest level is retained to obtain the final complete character sequence with the prosodic structure tag.
[0112] This forced approach ensures that the correct prosodic structure label sequence can be obtained even in the case of local label confusion due to insufficient training.
[0113] The present invention finally obtains a character sequence having only the highest level of prosodic structure labels at each place.
[0114] The present invention predicts the prosodic structure label in a unified end-to-end method, accepts a character sequence containing punctuation as input, and outputs a complete character sequence with a prosodic structure label, which is more concise and convenient;
[0115] Furthermore, a span representation-based method is used to model the prosody structure tree, and a dynamic programming algorithm is used to find the optimal prosody structure tree. Relying on the tree structure, the three subtasks of prosodic words, prosodic phrases and intonation phrases are unified, and the inclusion relationship between the three prosodic structure levels is better considered, which can greatly improve the prediction performance.
[0116] Furthermore, the present invention does not require information such as word segmentation and part of speech as additional input, thus avoiding the impact of error accumulation and complex feature engineering, which is more conducive to practical applications.
[0117] A specific example is provided below to further illustrate the method of the present invention.
[0118] 1. Build a dataset
[0119] There is no unified and universal public dataset for prosodic structure prediction tasks. We selected a public dataset of Biaobei, in which the prosodic structure labels in all sentences were manually annotated by professional annotators by reading transcripts and listening to speech. It was divided into training set, validation set and test set according to 8:1:1.
[0120] 2. Experimental Setup
[0121] We selected several of the most representative methods in recent years for comparative experiments. They are all sequence-to-sequence methods, which treat prosodic words, prosodic phrases, and intonation phrases as three subtasks and predict labels respectively. All models use pre-trained word-level BERT as the vector embedding for each word, and the parameters of BERT are frozen. The specific experimental benchmark comparison is as follows:
[0122] BLSTM-CRF: The BLSTM-CRF based model achieved extensive and acceptable experimental results, which adopts BLSTM as encoder and CRF as decoder.
[0123] Transformer-Softmax: It uses the Transformer structure as the encoder, followed by a 2D normalized exponential function Softmax as the decoder, in which prosodic words, prosodic phrases, and intonation phrases are treated as three independent subtasks to predict labels separately.
[0124] Transformer-Softmax-MTL: It is based on the above-mentioned Transformer-Softmax. It connects the three subtasks into a unified model by adding multi-task learning (MTL). The output of the previous task is spliced into the input of the next task, and finally global optimization training is performed.
[0125] Transformer-CRF: Considering the completeness of the experiment, Transformer is used as the encoder and a CRF is used as the decoder to perform the label prediction task.
[0126] Transformer-Tree: The method proposed in this invention uses Transformer as an encoder, followed by a span-based decoder to find the optimal prosody structure tree.
[0127] For all the above models involving Transformer, their structures remain consistent with those in Transformer-Tree.
[0128] 3. Experimental results and analysis
[0129] In order to compare the performance of the above methods, commonly used precision, recall and F1 score are used as measurement criteria. The experimental results are shown in Table 1 below.
[0130] Table 1 Results on the Biaobei dataset
[0131]
[0132] By comparing the experimental results, it can be found that the span-based method Transformer-Tree proposed in the present invention significantly surpasses the results of all benchmark methods in all performance indicators, especially in the prediction results of rhythmic phrases. Compared with three independent binary classification tasks, multi-task learning (MTL) simply combines these three subtasks. It is still a sequence-to-sequence method. The wrong result of the previous task will be accumulated to the next task, resulting in the model being difficult to train. It can also be seen from the experimental results that the performance improvement it brings is very small. The end-to-end method proposed by the present invention uses the rhythmic structure tree model to unify the three subtasks into one task. It can predict all labels at the same time and can realize an end-to-end method that directly uses Chinese characters as input.
[0133] An embodiment of the present application also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor is used to execute at least the method described above when executing the computer program.
[0134] An embodiment of the present application also provides a storage medium for storing a computer program, which at least performs the method described above when executed.
[0135] An embodiment of the present application further provides a processor, which executes a computer program and at least performs the method described above.
[0136] The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAMEnhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.
[0137] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0138] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0139] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0140] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.
[0141] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0142] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0143] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0144] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0145] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A method for determining the prosodic structure of Chinese text when synthesizing speech, characterized in that: The steps include: S1: inputting each sentence of Chinese text to be synthesized into a pre-trained word-level semantic feature extractor to obtain a semantic representation vector of each word; a sentence of the Chinese text to be synthesized is a sequence of Chinese characters including punctuation marks; S2: Input the semantic representation vector of each word into the encoder to obtain a hidden state vector of each word containing the context relationship; S3: inputting the hidden state vector into a decoder, wherein the decoder characterizes each span by the difference of the hidden state vectors at different positions, models the rhythm structure tree based on the span characterization method, and uses a dynamic programming algorithm to find the optimal rhythm structure tree; S4: converting the optimal prosody structure tree into a final complete character sequence with prosody structure tags through a rule-based format.
2. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 1, characterized in that: The pre-trained word-level semantic feature extractor is a semantic feature extractor BERT based on an attention mechanism and a fully connected network.
3. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 2, characterized in that: The encoder consists of multiple Transformer blocks, each of which contains two sublayers: a multi-head attention mechanism layer and a fully connected layer; each sublayer has a residual connection and layer normalization; The encoder is based on the input position i and the character x at position j. i and x j The semantic representation vectors of the corresponding positions are obtained by i and h j .
4. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 3, characterized in that: A rhythm structure tree is represented as follows: T:={(i t ,j t ,l t ):t=1,...,|T|} The triple (i, j, l) represents a labeled span, where i and j refer to the starting and ending positions of the span (i, j) respectively, and l refers to the label corresponding to the span, l∈L, where L refers to the prosodic structure label set in a broad sense; for unary chains or nested prosodic components, a nested label will be given; the data representation of span (i, j) is defined as the difference between the corresponding hidden state vectors: h ij =h i -h j .
5. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 4, characterized in that: Each of the spans is characterized, and a multi-layer feedforward neural network is used to define each span span(i,j) to specify a score s(i,j,l) corresponding to a label l: s(i,j,l)=[W2g(W1h ij +z1)+z2] l Among them, g is the ReLU activation function, W1, W2, z1, z2 represent the parameters of the feedforward neural network; Assign a corresponding score s(T) to all possible rhythm structure trees T: The rhythm structure tree with the highest score is determined as the optimal rhythm structure tree.
6. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 5, characterized in that: A CKY-style bottom-up dynamic programming algorithm is used to find the optimal rhythm structure tree, specifically including: The n-ary prosody structure tree is constructed by continuously performing binary differentiation; nodes that appear during the binary differentiation process but are not components of the prosody structure will be given an empty label And satisfy the following properties: All spans with empty labels have a score of zero; Definitions best (i, j) represents the best score of the prosody structure tree of span (i, j). For a general span, a label l∈L is independently specified each time and a best separation point k is selected, i<k<j, and then the above recursive process is repeated for the left and right subspans after separation. The formula is as follows: For a span of length 1, the best score depends only on the choice of label l: Calculate the best score s for each sentence of length n best (0,n), which is the score of the optimal tree for the entire sentence; By backtracking the entire process of finding the separation point and specifying the label, the corresponding rhythm structure tree is reconstructed; during the reconstruction process, the spans with the specified empty label are merged to obtain a complete n-ary tree; The complexity of the entire algorithm is O(n 3 +Dn 2 ), where D is the size of the prosodic structure label set L.
7. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 6, characterized in that: For a given truth tree T * , so that all possible predicted rhythm structure trees T satisfy the following interval constraints: s(T * )≥s(T)+Δ(T,T * ) The above constraints are implemented by minimizing the following chain losses: where Δ is the Hamming loss for the labeled spans.
8. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 7, characterized in that: The chain loss is achieved by making a minor modification to the dynamic programming algorithm in the decoding phase, namely, replacing s(i,j,l) with in, is the span span(i,j) in the truth tree T * The tags in .
9. The method for determining the prosodic structure of Chinese text when synthesizing speech as claimed in claim 8, characterized in that: The optimal prosody structure tree is converted into a final complete character sequence with a prosody structure tag through a rule-based format, including: S41: moving each rhythm structure label of the optimal rhythm structure tree to the back of the corresponding rhythm level component to obtain a sequence containing multiple layers of rhythm structure labels; S42: The multiple layers of prosodic tags behind each prosodic word are merged, and only the prosodic structure tag at the highest level is retained to obtain the final complete character sequence with the prosodic structure tag.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
