Speech synthesis method and device, and computer equipment

Abstract

The invention relates to a speech synthesis method and device, and computer equipment. The method comprises the following steps: determining a phoneme vector of a target text, and determining a word vector of the target text based on the phoneme vector; encoding the phoneme vector to obtain a first feature; encoding the word vector to obtain a second feature; and performing attention processing based on the first feature and the second feature to determine a third feature, and performing speech synthesis based on the third feature and the first feature to obtain a speech synthesis result of the target text. By adopting the method, the naturalness of speech synthesis can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, apparatus, and computer equipment. Background Art

[0002] With the development of artificial intelligence technology, the application of text to speech (TTS) is becoming more and more widespread. Speech synthesis is usually a data processing that converts text into anthropomorphic speech. The commonly used speech synthesis method is to pre-train the speech synthesis model, and then input the text into the speech synthesis model to output the speech. Summary of the invention

[0003] The present disclosure provides a speech synthesis method, apparatus, and computer equipment, which can effectively improve the naturalness of speech synthesis.

[0004] In a first aspect, the present disclosure provides a speech synthesis method, the method comprising:

[0005] Determine a phoneme vector of a target text, and determine a word vector of the target text based on the phoneme vector;

[0006] Encoding the phoneme vector to obtain a first feature; and encoding the word vector to obtain a second feature;

[0007] Performing attention processing based on the first feature and the second feature to determine a third feature;

[0008] Speech synthesis is performed based on the third feature and the first feature to obtain a speech synthesis result of the target text.

[0009] In a second aspect, the present disclosure further provides a speech synthesis device, the device comprising:

[0010] A phoneme processing module, used to determine a phoneme vector of a target text, and based on the phoneme vector, determine a word vector of the target text;

[0011] An encoding module, used for encoding the phoneme vector to obtain a first feature; and encoding the word vector to obtain a second feature;

[0012] an intermediate feature processing module, configured to perform attention processing based on the first feature and the second feature to determine a third feature;

[0013] An audio feature processing module is used to perform speech synthesis based on the third feature and the first feature to obtain a speech synthesis result of the target text.

[0014] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any method embodiment of the present disclosure are implemented.

[0015] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any method embodiment of the present disclosure are implemented.

[0016] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in any method embodiment of the present disclosure are implemented.

[0017] It can be seen that in the embodiment provided by the present disclosure, the processing of word features added to the speech synthesis process, since the word vector is determined based on the phoneme vector, the word vector in this solution is associated with the features of the phoneme vector. Since the third feature is obtained by performing attention processing based on the first feature obtained by processing the phoneme vector and the second feature obtained by processing the word vector, the third feature reflects the feature learning result between the phoneme-level feature and the word-level feature. Therefore, speech synthesis based on the third feature and the first feature can learn feature information at the word level that is higher than the phoneme level, so that the information content of the obtained speech features is richer and more accurate, thereby making the speech synthesis result of the text more natural, more consistent with the speech of natural people, and greatly improving the effect of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of an application environment of a speech synthesis method in an embodiment;

[0020] Figure 2 A schematic diagram of an implementation scenario of a speech synthesis method in an embodiment;

[0021] Figure 3 is a flowchart of a speech synthesis method in one embodiment;

[0022] Figure 4 A schematic diagram of the module structure of a word encoder provided by the present disclosure;

[0023] Figure 5 A schematic diagram of an implementation scenario of a speech synthesis method in another embodiment;

[0024] Figure 6 is a structural schematic diagram of a speech synthesis device in one embodiment;

[0025] Figure 7 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0028] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0029] The present disclosure provides a speech synthesis method which can be applied to Figure 1 As an implementation of the present disclosure, the speech synthesis method provided in the embodiment of the present disclosure can be applied to a variety of computer devices. Among them, the computer device can be such as Figure 1The server 110 shown in the figure can receive or generate text, and can process the text through the method of the embodiment of the present disclosure, and can directly generate speech or output feature data related to the generated speech. The computer device described in the embodiment of the present disclosure can be a single server or a server cluster composed of multiple servers, or a distributed subsystem, a cloud processing platform, a quantum computing platform, an intelligent processing platform combined with AI (Artificial Intelligence), a server including a blockchain node, and a combination of devices. The embodiments provided by the present disclosure can also be applied to devices such as Figure 1 The terminal device 120 performs speech synthesis data processing, and the terminal device 120 may include but is not limited to various personal computers, laptops, smart phones, tablet computers, wearable devices, vehicle-mounted devices, medical devices, etc. The embodiments of the present disclosure may also be applied to computer devices based on the terminal device 120, the server 110, and a data transmission network for providing a communication link between the server 110 and the terminal device 120.

[0030] The training data input into current speech synthesis models during training is phoneme-level data. However, in actual human speech scenarios, the speech of the speaker usually also includes emotional expressions such as emotion and tone, which have a crucial impact on the degree of fit between the synthesized speech and natural human speech. The current speech synthesis model based on phoneme-level data processing still has a large difference from natural human speech in terms of emotion, semantics and other expressions. The synthesized speech is less natural and has obvious traces of machine synthesis.

[0031] At present, there are many types of speech synthesis models used in the industry, among which FastSpeech and FastSpeech2 are attracting more and more attention due to their speech synthesis effects and response speed. Taking the commonly used FastSpeech2 as an example, its model, based on FastSpeech1, abandons the Teacher-Student knowledge distillation framework to reduce the complexity of training, directly uses real speech data as the training target to avoid information loss, and introduces more accurate duration information and other variable information in speech to improve the quality of speech synthesis. However, the input of the model is still a feature sequence at the phoneme level, and it becomes relatively difficult for the model to learn the corresponding speaker's emotions, tone and other information. Therefore, the speech synthesized based on this type of model is still a certain gap from the naturalness of human speech, and there is currently no good solution. The disclosed implementation scheme takes into account that words are closely related to the emotions, tone, etc. expressed by natural people when speaking, and emotions, tone, etc. have an important impact on the naturalness of speech synthesis. Therefore, some embodiments provided by the present disclosure can add relevant processing of words in the target text to the existing phoneme-level speech synthesis model, which not only incorporates the emotions and speech feature information expressed at the word level, but also allows the basic features of phonemes and the basic features of words to learn from each other, thereby obtaining richer and more accurate speech features, making the speech output by the model more natural.

[0032] The present disclosure uses FastSpeech2 as the implementation scenario of the speech synthesis model to illustrate the present disclosure. It should be noted that FastSpeech2 as the speech synthesis model is only one achievable implementation of the solution provided by the present disclosure, and does not exclude that it can also be implemented in the data processing of other speech synthesis models. At the same time, the implementation scheme provided by the present disclosure can be used in the model training stage of speech synthesis, and can also be used in the application stage of the speech synthesis model. In some embodiments disclosed, such as Figure 2 As shown, a word processing module that performs relevant processing on words in the target text can be added to the original speech synthesis model (such as FastSpeech2). The word module processing can be combined with one or more intermediate results of the original speech synthesis model, such as the phoneme vector obtained by the original speech model processing the phoneme. The word module converts the phoneme vector into a word vector, and then outputs new feature information based on the feature information of the phoneme vector and the word vector. After that, the new feature information obtained by the word processing module and the feature information of the original speech synthesis model can be used for subsequent speech synthesis. For example Figure 2As shown, the FastSpeech2 model input can be the phonemes of the target text, which can be processed by the embedding module (Phoneme Embedding) to obtain the phoneme vector, and the phoneme vector can be processed by the phoneme encoder (Encoder) to obtain the first feature. The phoneme vector and the first feature are output to the word processing module, which converts the phoneme vector into a word vector and performs feature learning at the phoneme level and word level. The output feature information is then fused with the first feature output by the phoneme encoder. The fused feature is input to the variable information adapter (Variance Adaptor) to obtain the mel spectrogram, which is then processed by the mel spectrogram decoder (mel spectrogram Decoder) to output the feature vector. The feature vector output by the mel spectrogram decoder can be used to generate speech through a vocoder, or it can be visualized by drawing software. Specifically, Figure 3 As shown, in one embodiment provided by the present disclosure, a speech synthesis method is provided, which can be executed in Figure 1 The server 110 may include the following steps:

[0033] S30: Determine a phoneme vector of a target text, and determine a word vector of the target text based on the phoneme vector.

[0034] The target text may include text data for synthesizing the corresponding speech in some embodiments of the present disclosure. The target text may be one or more sentences of text data, or one or more paragraphs of text data. The language of the target text is not limited, and may be Chinese or other language text. In some embodiments, the target text may be split and converted into phonemes. Phoneme is a concept in phonetics, usually referring to the smallest phoneme unit in a language, which is the smallest phonetic unit that can distinguish the meaning of a word. The number and characteristics of phonemes may be different in different languages. For example, in English, there are about 48 different phonemes. In the field of Chinese speech synthesis, phonemes are usually pinyin + rhythm (the length of the pause between words). The target text may include one or more words, and a word may be composed of one or more phonemes. For example, "I miss you" may include three words "I", "I want", and "you", and the phonemes of the corresponding words may be wo3, xiang3, and ni3, respectively, wherein the phonemes corresponding to the word "I" may be "w" and "o", and the number 3 represents the tone of pinyin. A phoneme may have a corresponding phoneme dictionary (or library), in which a unique identifier corresponding to the phoneme is usually provided, and the unique identifier of the phoneme is usually represented by a number. The identifier of a phoneme is a number or a string, which is easier to process and store than the phoneme data itself. In some embodiments of the speech synthesis processing disclosed in the present invention, the identifier of the phoneme of the target text can be obtained, and the computer can greatly reduce the resource consumption of data processing and reduce the storage space by processing the identifier of the phoneme compared with the direct processing of the phoneme.

[0035] After obtaining the phonemes of the target text in the disclosed embodiment, the phoneme identifiers can be determined through the phoneme dictionary, and then the phoneme identifiers are embedded (Embedding) to obtain the phoneme vector after the phoneme conversion. Embedding is usually a process of reducing high-dimensional data to low-dimensional vector data in natural language processing, while retaining important high-dimensional information to facilitate the model to better understand and process text data. For example, the phoneme identifiers of a section of the target text can be:

[0036] [1,23,4,15,16,17,35,6,3,11,12,18]

[0037] In some embodiments of the present disclosure, Phoneme Embedding of FastSpeech2 can be used to process phoneme identifiers. The processed phoneme vector can be expressed as {1, x, m}, where x represents the number of phoneme identifiers. For example, in this implementation example, if the number of phoneme identifiers is 12, then x is 12, and m represents the length of the phoneme vector, such as m is 128.

[0038] After obtaining the phoneme vector of the target text, the word vector of the target text can be determined based on the phoneme vector. Specifically, the conversion of the phoneme vector to the word vector can be achieved in a variety of ways, such as taking a fixed length of a phoneme as a word vector, or converting the phoneme vector of a word in the target text into a word vector of a word, or using some models or algorithms or custom regulations to convert the phoneme vector to a word vector. In one embodiment provided by the present disclosure, the determination of the word vector of the target text based on the phoneme vector may include:

[0039] According to the correspondence between the arrangement position of the words in the target text and the identification of the phonemes, the phoneme vector of the identification corresponding to the arrangement position of the words is calculated to obtain the word vector of the target text.

[0040] The target text may generally contain a number of words, each of which has a corresponding arrangement position in the target text. Among them, the specific words in the target text can be realized by word segmentation methods such as dictionaries, rules, statistics, and machine learning. For example, in some embodiments of the present disclosure, each group of words in the target text can be determined based on the maximum match of a dictionary (usually referring to a library or file containing words and their corresponding information). After determining each group of words in the target text, the phonemes corresponding to the words can be determined, and the identifiers of the phonemes corresponding to the words can also be determined.

[0041] In some embodiments of the present disclosure, the aforementioned obtained phoneme vector can be converted into a word vector of a word of the target text based on the corresponding relationship between the word arrangement position and the identifier. In the process of converting to a word vector, since the phoneme vector is generated based on the identifier of the phoneme, and each group of words has a corresponding arrangement position, the identifier of the phoneme corresponding to the word can be determined according to the corresponding relationship, so it is also possible to further determine the vector corresponding to each group of words in the phoneme vector. Therefore, a specific or customized algorithm can be used to calculate the phoneme vector corresponding to each group of words, and then the word vector of the target text is obtained. For example, the aforementioned phoneme vector can be expressed as {1, x, m}, and the word vector generated by the conversion can be expressed as x is expressed as {1, y, n}. Among them, y represents the number of words, and n represents the length of the word vector. Generally, in some embodiments of the present disclosure, in order to facilitate the processing of feature data and feature fusion, etc., the length of the phoneme vector can be set to be the same as the length of the word vector, that is, the values ​​of m and n in {1, x, m} and {1, y, n} are the same, such as both taking values ​​128.

[0042] In some embodiments of the present disclosure, the arrangement position of words can be predetermined. The arrangement position of words generally refers to the positional relationship of words relative to other words in a sentence or text. This positional relationship is used for semantic understanding of words and sentence structure analysis. The arrangement position described in some embodiments of the present disclosure may refer to the relative positional relationship of the position of words in the text relative to other words, such as whether the word is located at the beginning, middle or end of a sentence, and may refer to the absolute positional relationship of the specific position of the word in the sentence, such as the position of the word in the first word or sentence. There is a corresponding relationship between the arrangement position and the identifier of the phoneme, and the phoneme identifier corresponding to the word in the arrangement position can be found through the corresponding relationship, thereby establishing the relationship between the identifier of the phoneme and the arrangement position of the word. In this way, in the subsequent word-based feature information processing process, the feature information of the phoneme can be associated, and the mutual learning between the features at the phoneme level and the features at the word level can be realized, so that the speech features of the synthesized speech contain higher-dimensional and more feature information, thereby improving the naturalness of the synthesized speech.

[0043] The identification of the phoneme, the arrangement position of the word, and the corresponding relationship between the arrangement position and the identification of the phoneme can be predetermined. The data information of the arrangement position of the word can be determined by the word segmentation or corpus database or other customized rules. In general, in order to facilitate data processing, the length of the data information of the arrangement position can usually be set to be the same as the length of the phoneme identification. For example, in an implementation example, the target text is "I really miss you", and the set list of phonemes is:

[0044] [wo 3zh en 3d e 1h ao 3x i ang 3n i 3]

[0045] After querying the phoneme dictionary to obtain the identifier of the phoneme data, the corresponding phoneme identifier set list is:

[0046] [17,23,3,15,16,3,29,41,1,18,36,3,27,20,21,3,19,25,3]

[0047] The target text is divided into four groups of words, namely "I", "Really", "Good", and "Miss you". The order of the arrangement positions of words in the disclosed embodiment can be represented by numbers, for example, and the same number represents the same word. "I" is set as the first word in the arrangement position, represented by "0", and the phoneme corresponding to "I" is wo 3. The first three numbers [17, 23, 3] in the set list of corresponding phoneme identifiers are all 0. By analogy, the data set list of the arrangement position is: [0 0 0 11 1 1 1 1 2 2 2 3 3 33 3 3 3], where 000 represents wo (I). 1 1 1 1 1 1 represents zhende (really), 222 represents hao (good), and 3333333 represents xiangni (miss you). The correspondence between the arrangement position and the identifier can be seen in Table 1 below:

[0048] Schematic diagram of the corresponding relationship between arrangement position and identification table 1

[0049]

[0050] Therefore, in some embodiments provided by the present disclosure, the correspondence between the arrangement position of the words in the target text and the identification of the phonemes can be predetermined. Specifically, the method can also include:

[0051] S302: Determine the phonemes corresponding to each group of words in the target text;

[0052] S304: Determine the arrangement position of the phonemes corresponding to each group of words according to the arrangement position of each group of words;

[0053] S306: According to the arrangement positions of the phonemes corresponding to each group of words, the phonemes are converted into corresponding identifiers to obtain the corresponding relationship.

[0054] There is a corresponding relationship between the arrangement position and the phoneme identifier, and the phoneme identifier corresponding to the word at the arrangement position can be found through the corresponding relationship, thereby establishing a relationship between the phoneme identifier and the arrangement position of the word. In this way, in the subsequent word-based feature information processing process, the feature information of the phoneme can be associated, and the features at the phoneme level and the features at the word level can learn from each other, so that the speech features of the synthesized speech contain higher-dimensional and more feature information, thereby improving the naturalness of the synthesized speech.

[0055] In some embodiments of the present disclosure, the phoneme vector can be directly converted into a word vector as a whole, or each group of words can be converted into a vector in segments. In some embodiments provided by the present disclosure, the phoneme vector of the identifier corresponding to the arrangement position is calculated according to the correspondence between the arrangement position of the word in the target text and the identifier, and the word vector of the target text can be obtained by:

[0056] S308: Determine a target identifier of the phoneme corresponding to the word according to the corresponding relationship;

[0057] S310: Determine a target vector in the phoneme vector corresponding to the target identifier;

[0058] S312: Determine a word vector of the word based on the sum of the target vectors corresponding to the word and the number of phoneme identifiers corresponding to the word;

[0059] S314: Combining the word vectors of the words to obtain the word vector of the target text.

[0060] In one embodiment, in the process of converting the phoneme vector into the word vector, the target vector corresponding to each group of words in the phoneme vector can be determined based on the corresponding relationship, and then the target vector is processed to generate the word vector of the word. The word vectors of each group of words are combined in the order of the arrangement position of the words to obtain the word vector of the target text. There are many ways to implement the conversion of the target vector of a word into the corresponding vector of the word. In one method provided by the present disclosure, the average value of the target vector corresponding to the word can be taken. Specifically, the target vectors corresponding to a group of words can be added and then divided by the number of phoneme identifiers corresponding to the word to obtain the word vector corresponding to the word. The specific processing methods for the vector operation processing and finding the corresponding target vector of the word based on the corresponding relationship will not be elaborated here.

[0061] S32: Encode the phoneme vector to obtain a first feature; and encode the word vector to obtain a second feature.

[0062] After obtaining the phoneme vector, the phoneme vector can be encoded. In order to distinguish it from the encoding process of words, the encoding process of phoneme-related features here can be referred to as the first feature encoding. For example, the phoneme encoder (Encoder) of FastSpeech2 can be used for encoding and outputting the first feature. It should be understood that in the data processing scenario of speech synthesis in the disclosed embodiment, the encoding of the phoneme vector can usually be performed by an encoder (usually based on a hidden Markov model) to convert the input phoneme feature sequence (such as the phoneme vector obtained after embedding) into a phoneme hidden state sequence (a state description of a data sequence in a machine learning model, because the data sequence in the state is unobservable, usually corresponding to the observation sequence), which is convenient for subsequent other processing of audio feature data. In the encoding process, the phoneme vector is usually subjected to feature learning processing, such as learning the emotion, tone, intonation, rhythm and other features represented by the phoneme, and the output feature sequence can be referred to as the first feature.

[0063] After the phoneme vector is converted into a word vector, the word vector can be encoded. The process of encoding the word vector here can be called the second feature encoding. Referring to the aforementioned description of encoding the phoneme vector, the semantics of the word, local text features, etc. can also be learned during the encoding of the word vector. In the process of encoding the word vector, the embodiment of the present disclosure can also learn the feature information of the word based on the arrangement position of the word. For example, the pause duration, tone and other information of the same word at different arrangement positions may be different, and the emotional information expressed may also be different. In a specific implementation method, it can be processed by a word encoder (Word Encoder). Referring to the aforementioned description of encoding the phoneme vector by an Encoder, the device or model for encoding the word vector here can be called a word encoder.

[0064] The word encoder can learn the feature information of the arrangement position of words, can learn local feature information, and can learn global feature information. The duration, emotion and other information of the same word at different positions are usually different, so learning the feature information of the arrangement position can accurately express the meaning of the word. The local feature information of a word refers to the local feature or attribute information of the word. It can include the word's part of speech, morphological information, semantic relationship, etc., which are used to describe some local features of the word in the language. For example, for the word "cat", its local feature information may include its part of speech as a noun, its morphology as a singular, and its contrast relationship with "dog". These local feature information can help better understand and use words. The global feature information of a word refers to the universality and common features of a word in a language. These features include the lexical properties of the word (such as part of speech, word meaning, word frequency, etc.), grammatical properties (such as syntactic role, word order, etc.) and common features in linguistics (such as collocation rules, semantic relationships, etc.). Global feature information can help understand and describe the use and function of words in a language. In addition, it should be noted that the word encoder can be obtained by combining multiple different types of encoders and some neural network models. Figure 4 Schematic diagram of the module structure of a word encoder provided by the present disclosure. Figure 4As shown, the architecture of the word encoder may include positional encoding (Positional Encoding, which is usually used to add position information to each position in the input sequence in the Transformer model (a time series model based on a self-attention mechanism, which can effectively encode time series information in the encoder part)), multi-layer convolution blocks (ConvBlocks, which are usually used to extract features from data such as images and speech), and multi-layer encoder layers (Transformer Encoder Layers, a deep learning model for natural language processing based on an attention mechanism). The multi-layer convolution block may include multiple convolution blocks, each of which may be composed of a series of convolution layers, batch normalization layers, and activation function layers, which are used to extract features from the input data. ConvBlocks may contain multiple modules, such as Conv1D, LayerNorm, ReLU, Dropout, and Conv1D. Among them, Conv1D is a one-dimensional convolutional neural network (CNN) model. The Conv1D model extracts local features of the input data by performing a convolution operation on the input data using a one-dimensional convolution kernel (one-dimensional filter). LayerNorm is a normalization technology used in neural networks. Its full name is Layer Normalization. It plays a normalization role and can accelerate convergence. ReLU is a nonlinear activation function. Its full name is Rectified Linear Unit. ReLU can solve the problem of gradient disappearance. Dropout is a regularization technology used to prevent neural networks from overfitting. It can effectively reduce overfitting problems and improve the performance of neural networks.

[0065] Of course, the word encoder of other implementations of the present disclosure can have other module structures, such as a convolution block of multiple layers of convolution blocks.

[0066] By encoding the word vector, the feature information of each word can be obtained, and the meaning of the word in the entire sentence or all sentences in which it is currently located, as well as the emotion of the word can be obtained more accurately. For example, the meanings of apples in iPhones and apple cakes are different. The meaning of apple in iPhones is the brand name. The meaning of apple in apple cakes is fruit or taste. Therefore, this embodiment obtains the second feature by encoding the second feature of the word vector, such as using a word encoder. The second feature can more accurately understand and represent the feature information such as the meaning and emotion of the word in the target text.

[0067] S34: Perform attention processing based on the first feature and the second feature to determine a third feature.

[0068] The first feature and the second feature may be processed by an attention mechanism to obtain a third feature.

[0069] The attention processing generally refers to the feature output processing of the input through the attention mechanism model. For example, the model for attention processing in some embodiments of the present disclosure may be a multi-head attention mechanism. The multi-head attention mechanism may be Multi-Head Attention, which extends the attention mechanism to multiple attention heads, each with its own query, key, and value, and maps them to different spaces through independent linear transformations. By using multiple attention heads, Multi-Head Attention can capture different semantic information and relationships, thereby better representing the information of the input sequence. Finally, by splicing the outputs of multiple attention heads together and transforming them through another linear transformation, the output sequence after attention processing is obtained, which is the third feature of the present disclosure.

[0070] Specifically, since the phoneme vector obtained above is a feature at the phoneme level, and the word vector is a feature of the target text at the word level, some embodiments provided by the present disclosure can combine features at different levels, and the features learn and interact with each other, and can output a third feature that has been learned at a different level and contains more feature information such as emotion / tone, and the feature expression of emotion / tone is more accurate. The third feature obtained here can represent the relationship between phonemes and words.

[0071] S36: Perform speech synthesis based on the third feature and the first feature to obtain a speech synthesis result of the target text.

[0072] The third feature obtained above contains more and more accurate emotion / tone features, so the third feature can be merged with the first feature generated by processing the phoneme feature to obtain a speech feature that can be used for subsequent speech generation. Of course, the merging can have different processing methods in different models or scene processing requirements, which are specifically determined according to the processing requirements of speech synthesis. Generally, the data results of the processing of the first feature and the third feature are vectors, so the merging described in some embodiments can specifically add the first feature and the third feature to obtain the speech feature. The speech synthesis processing of the target text can be performed based on the speech feature, and the corresponding processing can be performed according to different processing scenarios and data processing stages, requirements, etc. For example, speech synthesis based on the third feature and the first feature can include obtaining speech features based on the third feature and the first feature, and the obtained speech features can be used as the input of the next processing module in the FastSpeech2 model, such as the input of the Variance Adaptor, the mel spectrogram output by the Variance Adaptor. The mel spectrum can be synthesized into speech through a vocoder. Of course, it can also be converted into a visual mel spectrum sequence through a mel spectrogram decoder.

[0073] In the speech synthesis method provided by the embodiment of the present disclosure, the processing of word features added in the speech synthesis process, since the determination of word vectors is based on the determination of phoneme vectors, the word vectors in this solution are associated with the features of phoneme vectors. Since the third feature is obtained by performing attention processing based on the first feature obtained by processing the phoneme vectors and the second feature obtained by processing the word vectors, the third feature reflects the feature learning result between the features at the phoneme level and the features at the word level, therefore, speech synthesis based on the third feature and the first feature can learn feature information at the word level at a higher level than the phoneme level, so that the information content of the obtained speech features is richer and more accurate, thereby making the speech synthesis result of the text more natural, more consistent with the speech of natural people, and greatly improving the effect of speech synthesis.

[0074] Some of the aforementioned implementations describe that a second feature of feature information at the word level can be obtained by encoding the word vector, which can be specifically encoded by a word encoder. The second feature and the first feature are processed by a multi-head attention mechanism to output a richer and more accurate third feature such as emotion and tone. In some embodiments of the present disclosure, during the encoding process, not only can the local feature information such as emotion and tone expressed by words in different positions be learned based on the arrangement position of the words, but also the global features of the target text can be learned to obtain a second feature including local features and global features. Specifically, in some implementations provided by the present disclosure, the encoding of the word vector to obtain the second feature includes:

[0075] Performing position encoding on the word vector based on the arrangement position of the words to obtain position features;

[0076] Performing convolution processing on the position feature to obtain a local feature in the second feature;

[0077] Encoding processing is performed based on the local features to obtain global features in the second features.

[0078] In the disclosed embodiment, the word vector can be positionally encoded based on the arrangement position of the word to obtain a position feature, and the position feature can be convolved to obtain a local feature containing the word meaning information of the word. Encoding processing is performed based on the local feature to obtain global features such as the emotional information and semantic information of the target text.

[0079] The word vector encoding process in this embodiment can learn the feature information of the arrangement position of the word. The duration, emotion and other information of the same word at different positions are usually different, so learning the feature information of the arrangement position can accurately express the meaning of the word. Not only can the feature information of different levels (local and global features) be learned, but also the feature information of multiple dimensions such as emotion, semantics, and word meaning can be learned, and the second feature such as emotion / tone and other feature information can be obtained, and the expression of emotion / tone and other features can be more accurate.

[0080] The word encoder used to encode word vectors in some embodiments of the present disclosure is implemented. The word encoder can be Transformer Encoder Layers or Conformer Encoder Layer. Transformer Encoder Layers is a component in the Transformer model for encoding input sequences. It consists of multiple identical layers, each of which contains two sublayers: a multi-head self-attention mechanism and a feed-forward neural network. The multi-head self-attention mechanism is used to capture contextual information in the input sequence, and the feed-forward neural network is used to map and transform the hidden state of each position. Conformer Encoder Layer is a neural network layer used for speech recognition and natural language processing tasks. It is an important component of the Conformer model and is used to encode the input sequence. Conformer Encoder Layer combines the self-attention mechanism and the convolutional neural network (CNN) to capture the global and local dependencies of the input sequence. It contains multiple sublayers, each of which has a multi-head self-attention mechanism and a feed-forward neural network. In the multi-head self-attention mechanism, the Conformer Encoder Layer captures global dependency information by calculating the correlation between different positions in the input sequence. Convolutional neural networks are used to capture local dependencies in input sequences and extract features by performing convolution operations on the input sequences.

[0081] Specifically, in some embodiments of the present disclosure, a word encoder including Transformer Encoder Layers or Conformer Encoder Layer can be used to process local features output by multiple layers of ConvBlocks to extract global features, such as feature information such as sentiment information and word meaning information.

[0082] In some other embodiments of the method provided by the present disclosure, performing attention processing based on the first feature and the second feature to determine the third feature may include:

[0083] The global feature and the local feature are used as the key vector and the value vector of the attention processing respectively, and the first feature is used as the query vector of the attention processing for feature learning to obtain the third feature, and the length of the third feature is the same as the length of the first feature.

[0084] The global and local features of the second feature of the word obtained by the second feature encoding process can be used as the key vector and value vector of the attention mechanism, and the first feature obtained by the phoneme vector encoding process can be used as the query vector of the attention mechanism, and the multi-head attention mechanism output is used to learn the third feature with richer feature information such as emotion / tone and more accurate expression of emotion / tone at different levels (phoneme and word level, local and global features). Speech synthesis processing is performed based on the third feature and the first feature. For example, the third feature can be used to combine with the first feature of the phoneme to obtain a speech feature, so that the speech synthesized based on the speech feature is more natural and more in line with the natural speech expression of the speaker, effectively improving the speech synthesis effect.

[0085] The speech synthesis method described above can be combined with a variety of speech synthesis models based on phoneme processing, so that the overall speech synthesis model can obtain richer and more accurate speech features such as emotion and tone, thereby making the synthesized speech more natural and more in line with the speaker's natural speech expression.

[0086] For example, in some implementation examples of the present disclosure, the speech synthesis model can be obtained by adjusting the FastSpeech2 network structure. Figure 5 As shown, a word processing module can be added to the FastSpeech2 network structure of the speech synthesis model, which can include a word vector processing module for converting phoneme vectors into word vectors, a word encoder and a multi-head attention network. The word processing module can convert the audio vector of the original phoneme processing model into a word vector, and then use the multi-head attention mechanism to process the second feature after the word vector feature encoding and the first feature after the phoneme vector feature encoding, and output a third feature containing richer and more accurate feature information such as emotion and tone. The third feature can then be combined with the original phoneme processing model to generate an audio feature for subsequent speech synthesis processing.

[0087] The speech synthesis method provided by the embodiments of the present disclosure can add a network structure for word processing while keeping the network structure of the original speech synthesis model based on phoneme processing (which can be called a phoneme processing model, such as FastSpeech2) unchanged (or with little intrusion / modification impact on the original speech synthesis model), thereby allowing the entire speech synthesis model to add word feature processing during the speech synthesis process, and to associate the original phoneme-based speech synthesis with word features, so that the feature information of the word can be learned. In some implementations, the multi-head attention mechanism can be used to achieve mutual learning between phoneme-level features and word-level features, so that the information content of the speech features output by the model is richer and more accurate, thereby making the speech synthesized based on speech features more natural and more consistent with the speech of natural people, greatly improving the effect of speech synthesis.

[0088] It is understandable that the various embodiments of the above method in this specification are described in a progressive manner, and the same / similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, refer to the description of other method embodiments.

[0089] It should be understood that, although the steps in the flowcharts or structure diagrams involved in the various embodiments described in the present disclosure are sequentially displayed or the organizational structure of the model or module according to the indication of the arrows, these steps or structures are not necessarily executed in sequence in the order indicated by the arrows or as shown in the accompanying drawings. Unless clearly stated in this document, the execution of these steps does not have a strict order restriction or structural connection relationship, and these steps can be executed in other orders or have more or less connection relationships. Moreover, at least a part of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps. Similarly, the structure diagram involved in the accompanying drawings is not limited to the structure of the required model or module or device or server, etc., and in some embodiments, more or less modules / units may be included, and their connection relationship is not limited to the connection shown in the accompanying drawings, and more or less or different connections may be included according to the functional implementation of the module.

[0090] Based on the same inventive concept of the above-mentioned speech synthesis method embodiment, the embodiment of the present disclosure also provides a speech synthesis device. The device may include a system (including a distributed system), software (application), module, component, server, client, electronic device, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided by the embodiment of the present disclosure is as described in the following embodiments. Since the implementation scheme of the device to solve the problem is similar to the method, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific definition in the one or more speech synthesis device embodiments provided below can refer to the definition of the speech synthesis method above, and will not be repeated here.

[0091] Based on the description of the foregoing related method embodiments, in one embodiment, Figure 6 As shown, a speech synthesis device 60 is provided, and the device may be the aforementioned computer device, or a module, component, device, unit, etc. integrated in the terminal computer device. In a specific implementation, the device may include:

[0092] A phoneme processing module 602, used to determine a phoneme vector of a target text, and based on the phoneme vector, determine a word vector of the target text;

[0093] An encoding module 604 is used to encode the phoneme vector to obtain a first feature; and encode the word vector to obtain a second feature;

[0094] An intermediate feature processing module 606, configured to perform attention processing based on the first feature and the second feature to determine a third feature;

[0095] The audio feature processing module 608 is used to perform speech synthesis based on the third feature and the first feature to obtain a speech synthesis result of the target text.

[0096] In some embodiments of the device, the encoding module 604 encodes the word vector to obtain the second feature including:

[0097] Performing position encoding on the word vector based on the arrangement position of the words to obtain position features;

[0098] Performing convolution processing on the position features to obtain local features containing semantic information of the words;

[0099] Encoding processing is performed based on the local features to obtain global features containing the emotional information of the target text.

[0100] In some embodiments of the apparatus, the intermediate feature processing module 606 performs attention processing based on the first feature and the second feature to determine a third feature, including:

[0101] The global feature and the local feature are used as the key vector and the value vector of the attention processing respectively, and the first feature is used as the query vector of the attention processing for feature learning to obtain the third feature, and the length of the third feature is the same as the length of the first feature.

[0102] In some embodiments of the device, the phoneme processing module 602 determines the word vector of the target text based on the phoneme vector, including:

[0103] According to the correspondence between the arrangement position of the words in the target text and the identification of the phonemes, the phoneme vector of the identification corresponding to the arrangement position of the words is calculated to obtain the word vector of the target text.

[0104] In some embodiments of the device, it also includes a correspondence determination module, which is used to determine the phonemes corresponding to each group of words in the target text, determine the arrangement positions of the phonemes corresponding to each group of words according to the arrangement positions of the phonemes corresponding to each group of words, and convert the phonemes into corresponding identifiers according to the arrangement positions of the phonemes corresponding to each group of words to obtain the correspondence.

[0105] In some embodiments of the device, the phoneme processing module 602 calculates the phoneme vector of the identifier corresponding to the arrangement position of the word according to the correspondence between the arrangement position of the word in the target text and the identifier of the phoneme, so as to obtain the word vector of the target text, including:

[0106] Determining, according to the corresponding relationship, a target identifier of the phoneme corresponding to the word;

[0107] Determine a target vector in the phoneme vector corresponding to the target identifier;

[0108] Determining a word vector of the word based on the sum of the target vectors corresponding to the word and the number of phoneme identifiers corresponding to the word;

[0109] The word vectors of each group of words are combined to obtain the word vector of the target text.

[0110] The method or device provided by the present disclosure can be implemented in combination with a variety of speech synthesis models based on phoneme processing. A word processing module structure for word processing can be added to the original phoneme processing model to achieve mutual learning and operation of feature information at the phoneme level and feature information at the word level, thereby making the audio features output by the entire model (the original phoneme processing model and the added word processing module can be regarded as a speech synthesis model together. Of course, the present disclosure does not exclude that the speech synthesis model can also include other structural parts) contain more dimensions, more information, and more accurate information, thereby improving the naturalness of the speech synthesized by the model.

[0111] Each module in the above-mentioned speech synthesis device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0112] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0113] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0114] The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device can be used to store target texts, phonemes, dictionaries / lexicons, phoneme identifiers, model information, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech synthesis method is implemented.

[0115] Those skilled in the art will understand that Figure 4-Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0117] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in any of the above method embodiments when executed by a processor.

[0118] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0119] It should be noted that the above-mentioned devices, equipment, etc. may also include other implementation methods corresponding to the method according to the description of the method embodiment. The specific implementation method can refer to the description of the relevant method embodiment, and in the description of the embodiments of other devices, computer equipment, models, etc. corresponding to the method, for example, the first module, the second module, etc. may be adaptively added according to the implementation of the corresponding method function to realize the data processing of the corresponding method function. At the same time, the new embodiments composed of the mutual combination of the features between the various methods and the device, equipment, and server embodiments still belong to the scope of implementation covered by this disclosure, and will not be described one by one here.

[0120] It should be noted that the target text, phonemes, identifiers, dictionaries / lexicons, encoders / decoders, models or algorithms and other data information involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with relevant laws, regulations and standards.

[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0122] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Determine a phoneme vector of a target text, and determine a word vector of the target text based on the phoneme vector; Encoding the phoneme vector to obtain a first feature; and encoding the word vector to obtain a second feature; Performing attention processing based on the first feature and the second feature to determine a third feature; Speech synthesis is performed based on the third feature and the first feature to obtain a speech synthesis result of the target text.

2. The method according to claim 1, characterized in that The encoding of the word vector to obtain a second feature includes: Performing position encoding on the word vector based on the arrangement position of the words to obtain position features; Performing convolution processing on the position feature to obtain a local feature in the second feature; Encoding processing is performed based on the local features to obtain global features in the second features.

3. The method according to claim 2, characterized in that The performing attention processing based on the first feature and the second feature to determine a third feature includes: The global feature and the local feature are used as the key vector and the value vector of the attention processing respectively, and the first feature is used as the query vector of the attention processing for feature learning to obtain the third feature, and the length of the third feature is the same as the length of the first feature.

4. The method according to claim 1, characterized in that The step of determining the word vector of the target text based on the phoneme vector includes: According to the correspondence between the arrangement position of the words in the target text and the identification of the phonemes, the phoneme vector of the identification corresponding to the arrangement position of the words is calculated to obtain the word vector of the target text.

5. The method according to claim 4, characterized in that The method further comprises: Determining the phonemes corresponding to each group of words in the target text; According to the arrangement position of each group of words, determining the arrangement position of the phonemes corresponding to each group of words; According to the arrangement positions of the phonemes corresponding to each group of words, the phonemes are converted into corresponding identifiers to obtain the corresponding relationship.

6. The method according to claim 4, characterized in that The step of calculating the phoneme vector of the identifier corresponding to the arrangement position of the word according to the correspondence between the arrangement position of the word in the target text and the identifier of the phoneme to obtain the word vector of the target text includes: Determining, according to the corresponding relationship, a target identifier of the phoneme corresponding to the word; Determine a target vector in the phoneme vector corresponding to the target identifier; Determining a word vector of the word based on a target vector corresponding to the word and the number of phoneme identifiers corresponding to the word; The word vectors of the words are combined to obtain the word vector of the target text.

7. A speech synthesis device, characterized in that: The device comprises: A phoneme processing module, used to determine a phoneme vector of a target text, and based on the phoneme vector, determine a word vector of the target text; An encoding module, used for encoding the phoneme vector to obtain a first feature; and encoding the word vector to obtain a second feature; an intermediate feature processing module, configured to perform attention processing based on the first feature and the second feature to determine a third feature; An audio feature processing module is used to perform speech synthesis based on the third feature and the first feature to obtain a speech synthesis result of the target text.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images